Skip to content

Resolve "Scan and dataset metadata"

Wout De Nolf requested to merge 2769-scan-and-dataset-metadata into master

Closes #2769 (closed)

Fix the scan metdata vs. dataset metadata mixup. To achieve this we had to make all metadata related things more uniform as there were lots of conflicts.

Scan metadata

Two scan metadata protocols for Bliss controllers

  1. HasMetadataForScan:
    • abstract method scan_metadata: returns controller metadata
    • property scan_metadata_name: by default equal to the controller name
    • enable/disable per controller: enable_scan_metadata, disable_scan_metadata
    • Redis destination of this metadata: scan_metadata_name key under the scan meta category "instrument" in scan_info
    • Metadata collection is done through a ScanMeta instance (see below)
  2. HasMetadataForScanExclusive: scan metadata will never be collected when not part of a scan
    • Derives from the HasMetadataForScan protocol: scan_metadata, enable_scan_metadata (still needs to be part of the scan), disable_scan_metadata
    • Redis destination of this metadata: Redis node of the controller + "devices" section of scan_info
    • Metadata collection is done through an AcquistionObject (see below)

The time aspect is not included in these protocols. The timing is introduced through Scan and AcquistionObject.

Examples:

  • controllers like lima, mca and all counters will use HasMetadataForScanExclusive because we only want their metadata when they are used in the scan.
  • controllers like multipositions, slits, machinfo and transfocators will use HasMetadataForScan because we always want their metadata for every scan.

Note: for MachInfo (HasMetadataForScan protocol) the controller for scanning is the counter controller which does not produce any metadata. So even if we wanted only the metadata when MachInfo is part of the scan by using HasMetadataForScanExclusive, you wouldn't see any metadata because MachInfo itself is not the controller in the acq. chain.

ScanMeta class

  1. We still have the global ScanMeta instance for user metadata as before. Only now it is really for user metadata (no "positioners" or metadata automatically generated from controllers).
  2. For each scan there is a separate ScanMeta instance which contains two metadata categories
    • "positioners" metadata category: positioners_start, positioners_end, ...
    • "instrument" metadata category: HasMetadataForScan.scan_metadata_name keys with HasMetadataForScan.scan_metadata values (excluding HasMetadataForScanExclusive which goes through AcquistionObject)

AcquisitionObject class

The methods fill_meta_at_scan_start and fill_meta_at_scan_end are replaced by a single get_acquisition_metadata with a META_TIMING parameter. The AcquisitionObject calls the HasMetadataForScan.scan_metadata of the underlying controllers.

Counter class

The Counter class is now derived from MetadataForScan so that get_metadata becomes scan_metadata like all other controllers.

Scan class

Filling of the local scan_info dictionary is no longer scattered all over the Scan class code. We have:

  1. _metadata_at_scan_instantiation:
    • time of local collection: when instantiating the Scan object
    • time of Redis publication: when creating the ScanNode
  2. _metadata_at_scan_start:
    • time of local collection: when the scan starts (Scan.run() called)
    • time of Redis publication: when creating the ScanNode
  3. _metadata_at_scan_prepared:
    • time of local collection: after all devices are prepared (this includes creating the Redis nodes)
    • time of Redis publication: by the call to ScanNode.prepared(...)
  4. _metadata_at_scan_end:
    • time of local collection: when the scan ends
    • time of Redis publication: by the call to ScanNode.end(...)

Apart from some custom scan metadata (session name, start time, end time, ...), there are three types of scan metadata:

  1. user scan metadata:
    • one of the ScanMeta instances (see above)
  2. controller scan metadata (HasMetadataForScan excluding HasMetadataForScanExclusive):
    • one of the ScanMeta instances (see above)
  3. acquisition controller scan metadata (HasMetadataForScan):
    • metadata collected through AcquisitionObject

Dataset metadata

One dataset metadata protocol for Bliss controllers

Controllers with the HasMetadataForDataset (with abstract method dataset_metadata) will be used by the Bliss sessions's ICATmeta controller to collect dataset metadata.

Yaml configuration

An overview of all metadata related configuration (not only related to this MR)

  • Specify the ICATmeta controller for a Bliss session (dataset metadata)
    - class: Session
      name: test_session
      setup-file: ./test_setup.py
      icat-mapping: icat  # 
  • Configuration for the sessions's ICATmeta controller (dataset metadata)
    - name: icat
      plugin: bliss
      class: ICATmeta
    
      objects:  # need to have the HasMetadataForDataset protocol
        primary_slit: $primary_slit
        attenuator01: $att1
    
      positioners:
        sample: $roby
        insertion_device_gap: [$roby,$robz]
    
      mapping:
        InstrumentVariables_name: [ $roby.name, $robz.name ]
        InstrumentVariables_value: [ $roby.position, $robz.position ]
  • The MultiplePositions class still has its metadata configured but now splits scan and dataset metadata (backward compatible with existing yaml files with FutureWarning)
    name: att1
    plugin: bliss
    class: MultiplePositions
    - label: Al40
      description: Attenuator in Al40 position
      scan_metadata:
        "@NX_class": NXattenuator
        type: Al
        thickness: 40
        thickness@unit: um
        status: in
      dataset_metadata:
        type: Al
        thickness: 40
        status: in
        Positioners_name: $att1z.name
        Positioners_value: $att1z.position
    The reason this exists is because MultiplePositions is a multi purpose class. Maybe this should be improved but not in this MR.

Other changes

  • Fix deepcopy problem of ScanMeta
  • Simplify TransfocatorMockup (no temporary object).
  • Add __close__ to Slits
  • Add more objects that generate metadata to the nexus_writer_session in the test suite
  • Add more scan metadata tests
Edited by Wout De Nolf

Merge request reports