Skip to content

Resolve "Feedback in BLISS when ICAT fails"

Wout De Nolf requested to merge 2848-feedback-in-bliss-when-icat-fails-2 into master

Closes #2848 (closed)

Requires icat/icat-plus!379 (merged)

A dataset in Redis can be

  • closed: metadata is gathered and frozen (which means metadata cannot be changed anymore)
  • registered: metadata is registered with ICAT

A registered dataset is always closed but a closed dataset is not necessarily registered.

A dataset node (contains metadata) does not expire in Redis as long as it is not confirmed.

A dataset is confirmed when it is verified to have been stored in ICAT.

This verification is done when executing data policy commands like newdataset, newcollection, newproposal, ...

The SCAN_SAVING objects has new methods SCAN_SAVING.icat_* as listed below.

New features

  • Closed but unconfirmed datasets are shown when doing newdataset, newcollection, newproposal, ...

    DEMO_SESSION [4]: newdataset()
Dataset set to '0012'
Data path: /tmp/scans/inhouse/id002108/id00/sample/sample_0012
Number of unconfirmed ICAT dataset registrations: 4

  • Show the list of closed but unconfirmed datasets

    DEMO_SESSION [1]: SCAN_SAVING.icat_info()
ICAT proposal time slot:
 proposal    ID002108
 beamline    ID00
 startDate   2021-09-01T12:47:58.948+02:00
 id          *********
 title       ID002109
 url         https://data.esrf.fr/investigation/*********/datasets

Datasets: 3 unconfirmed, 110 confirmed

Unconfirmed datasets:
 Name          Time since end
 sample_0060   0:00:44.645211
 sample_0061   0:00:41.392353
 sample_0062   0:00:17.115062

  • Re-send all closed but unconfirmed datasets to ICAT

    DEMO_SESSION [9]: SCAN_SAVING.icat_register_datasets()
Dataset sample_0060 has been send to ICAT
Dataset sample_0061 has been send to ICAT
Dataset sample_0062 has been send to ICAT

  • Re-send a closed and unconfirmed dataset to ICAT

    DEMO_SESSION [9]: SCAN_SAVING.icat_register_dataset("sample_0060")
Dataset sample_0060 has been send to ICAT

  • ICAT dataset info is shown in the SCAN_SAVING info (last line)

    DEMO_SESSION [2]: SCAN_SAVING                                                                                                                                                                                                   
       Out [2]: Parameters (default) -                                                                                                                                                                                        
                 
                  .images_path_template                = 'scan{scan_number}'
                  .images_prefix                       = '{img_acq_device}_'
                  .date_format                         = '%Y%m%d'
                  .scan_number_format                  = '%04d'
                  .dataset_number_format               = '%04d'
                  .session                             = 'demo_session'
                  .date                                = '20210825'
                  .scan_name                           = '{scan_name}'
                  .scan_number                         = '{scan_number}'
                  .img_acq_device                      = '<images_* only> acquisition device name'
                  .writer                              = 'nexus'
                  .data_policy                         = 'ESRF'
                  .template                            = '{proposal_name}/{beamline}/{collection_name}/{collection_name}_{dataset_name}'
                  .beamline                            = 'id00'
                  .proposal_name                       = 'id002108'
                  .proposal_type                       = 'inhouse'
                  .base_path                           = '/tmp/scans/inhouse'
                  .collection_name                     = 'sample'
                  .dataset_name                        = '0012'
                  .data_filename                       = '{collection_name}_{dataset_name}'
                  .images_path_relative                = True
                  .mount_point                         = ''
                  .creation_date                       = '2021-08-25 16:27:54'
                  .last_accessed                       = '2021-08-25 17:10:42'
                --------------  ---------  ------------------------------------------------------------------
                does not exist  filename   /tmp/scans/inhouse/id002108/id00/sample/sample_0012/sample_0012.h5
                does not exist  directory  /tmp/scans/inhouse/id002108/id00/sample/sample_0012
                ICAT            url        https://data.esrf.fr/investigation/525483637/datasets
                ICAT            datasets   0 unconfirmed, 111 confirmed
                --------------  ---------  ------------------------------------------------------------------

  • Manually confirm dataset registrations in ICAT (called automatically in most of the commands above, normally you don't need to call this explicitly)

    DEMO_SESSION [2]: SCAN_SAVING.icat_confirm_datasets()

Beamline configuration

ICAT feedback requires direct communication with ICAT (the tango servers do not support it).

icat_servers:
    metadata_urls: [URL1, URL2]
    elogbook_url: URL3
    elogbook_token: elogbook-00000000-0000-0000-0000-000000000000
    elogbook_timeout: 0.1  # optional
    feedback_timeout: 2  # optional
    queue_timeout: 1  # optional
    disable: False # optional

When disable is True all e-logbook messages are lost but dataset metadata is kept in REDIS until enabled again or until switching to a different proposal.

The different timeouts are optional

  • elogbook_timeout: time to wait for elogbook message confirmation
  • feedback_timeout: time to wait for retrieving ICAT feedback on the current proposal
  • queue_timeout: time to wait for connection to metadata_urls
Edited by Wout De Nolf

Merge request reports