Draft: Resolve "Feedback in BLISS when ICAT fails"
Closes #2848 (closed)
Requires icat/icat-plus#255 (closed)
A dataset in Redis can be
- closed: metadata is gathered and frozen (which means metadata cannot be changed anymore)
- registered: metadata is registered with ICAT
A registered dataset is always closed but a closed dataset is not necessarily registered.
A dataset node (contains metadata) does not expire in Redis as long as it is not confirmed.
A dataset is confirmed when it is verified to have been stored in ICAT.
This verification is done when executing data policy commands like newdataset, newcollection, newproposal, ...
The SCAN_SAVING objects has new methods SCAN_SAVING.icat_* as listed below.
New features
-
Closed but unconfirmed datasets are shown when doing
newdataset,newcollection,newproposal, ...DEMO_SESSION [4]: newdataset() Dataset set to '0012' Data path: /tmp/scans/inhouse/id002108/id00/sample/sample_0012 Number of unconfirmed ICAT dataset registrations: 4 -
Show the list of closed but unconfirmed datasets
DEMO_SESSION [1]: SCAN_SAVING.icat_info() ICAT authentication: Authentication enabled User name myname ICAT session yes SSO token yes ICAT proposal time slot: proposal ID002108 beamline ID00 startDate 2021-09-01T12:47:58.948+02:00 id ********* title ID002109 url https://data.esrf.fr/investigation/*********/datasets Datasets: 3 unconfirmed, 110 confirmed Unconfirmed datasets: Name Time since end sample_0060 0:00:44.645211 sample_0061 0:00:41.392353 sample_0062 0:00:17.115062 -
Re-send all closed but unconfirmed datasets to ICAT
DEMO_SESSION [9]: SCAN_SAVING.icat_register_datasets() Dataset sample_0060 has been send to ICAT Dataset sample_0061 has been send to ICAT Dataset sample_0062 has been send to ICAT -
Re-send a closed and unconfirmed dataset to ICAT
DEMO_SESSION [9]: SCAN_SAVING.icat_register_dataset("sample_0060") Dataset sample_0060 has been send to ICAT -
ICAT dataset info is shown in the SCAN_SAVING info (last line)
DEMO_SESSION [2]: SCAN_SAVING Out [2]: Parameters (default) - .images_path_template = 'scan{scan_number}' .images_prefix = '{img_acq_device}_' .date_format = '%Y%m%d' .scan_number_format = '%04d' .dataset_number_format = '%04d' .session = 'demo_session' .date = '20210825' .scan_name = '{scan_name}' .scan_number = '{scan_number}' .img_acq_device = '<images_* only> acquisition device name' .writer = 'nexus' .data_policy = 'ESRF' .template = '{proposal_name}/{beamline}/{collection_name}/{collection_name}_{dataset_name}' .beamline = 'id00' .proposal_name = 'id002108' .proposal_type = 'inhouse' .base_path = '/tmp/scans/inhouse' .collection_name = 'sample' .dataset_name = '0012' .data_filename = '{collection_name}_{dataset_name}' .images_path_relative = True .mount_point = '' .creation_date = '2021-08-25 16:27:54' .last_accessed = '2021-08-25 17:10:42' -------------- --------- ------------------------------------------------------------------ does not exist filename /tmp/scans/inhouse/id002108/id00/sample/sample_0012/sample_0012.h5 does not exist directory /tmp/scans/inhouse/id002108/id00/sample/sample_0012 ICAT url https://data.esrf.fr/investigation/525483637/datasets ICAT user myname ICAT datasets 0 unconfirmed, 111 confirmed -------------- --------- ------------------------------------------------------------------ -
Manually confirm dataset registrations in ICAT (called automatically in most of the commands above, normally you don't need to call this explicitly)
DEMO_SESSION [2]: SCAN_SAVING.icat_confirm_datasets() -
Enable/disable SSO username/password prompt for the current process (e.g. when running long macro's that uses the
newdatasetcommand, you don't want it to prompt for anything)DEMO_SESSION [2]: SCAN_SAVING.icat_disable_prompt() DEMO_SESSION [2]: SCAN_SAVING.icat_enable_prompt() -
Manual login/logout from ICAT (called automatically in most of the commands above, normally you don't need to call this explicitly)
DEMO_SESSION [2]: SCAN_SAVING.icat_client.login() DEMO_SESSION [2]: SCAN_SAVING.icat_client.logout()
Beamline configuration
As we need to login to ICAT to retrieve the list of datasets, SSO support has been added and the sso_url needs to be provided (if you don't provide it, datasets will never be confirmed)
icat_servers:
metadata_urls: [URL1, URL2]
elogbook_url: URL3
elogbook_token: elogbook-00000000-0000-0000-0000-000000000000
sso_url: URL4 # only when ICAT feedback is requiredNote that this will prompt for username and password when needed (not too often thanks to SSO). The SSO token is lost when restarting Bliss so it will prompt again for username and password when the ICAT session has expired (after 12 hours).
SSO
The SSO client allows accessing private ICAT+ information
from bliss.icat.client.icatplus import IcatPlusSession
if __name__ == "__main__":
with IcatPlusSession(url=..., sso_url=...) as session:
investigation_id = session.investigation_id("id002109", "ID00")
print("investigation id:", investigation_id)
print("datasets:", session.dataset_redis_names(investigation_id))Note: when calling newproposal or endproposal it closes all SSO and ICAT connections, tokens and session id's. This also happens when you close the Bliss session as this information is too sensitive to store in Redis.