Scan and dataset metadata

The difference between metadata in the scope of the scan (saved in the Nexus file) and metadata in the scope of the dataset (saved in ICAT, approx. 700 predefined field names) seems to have been lost somehow over time.

I tried to give an overview of the current situation for review purposes. (feel free to modify @valentin.valls @matias.guijarro)

Scan metadata

Gathering

This metadata is gathered at the start and end of each scan by scan.Scan._fill_meta wich will loop over the scan chain's acquisition objects and call fill_meta_at_scan_start or fill_meta_at_scan_end resp.

Implementation

• The HasMetadataForScan interface has the metadata_when_prepared method. Classes with this interface are
  • lima_base.Lima (lima controller class)
  • mca.base.BaseMCA (mca controller class)
• Every acquisition master or slave has 2 metadata gathering methods:

  class AcquisitionObject:
    def fill_meta_at_scan_start(self, scan_meta):
        device = self.device
        if isinstance(device, HasMetadataForScan):
            return device.metadata_when_prepared()
        return None
  
    def fill_meta_at_scan_end(self, scan_meta):
        return None

• Classes derived from AcquisitionObject that overwrite these methods are:
  • BaseCounterAcquisitionSlave.fill_meta_at_scan_start
  • LimaAcquisitionMaster.fill_meta_at_scan_end
• Every counter has a get_metadata method which is called by BaseCounterAcquisitionSlave.fill_meta_at_scan_start

  class Counter:
    def get_metadata(self):
       return {}

  This method is overwritten by lima.roi.RoiStatCounter, lima.roi.RoiProfileCounter and lima.roi.RoiCollectionCounter.
• MultiplePositions has a metadata field in its yaml configuration which is mixing scan and dataset metadata:

  class: MultiplePositions
  positions:
  - label: Al
    metadata:
      "@NX_class": NXattenuator  # scan metadata
      type: Al  # scan metadata
      status: in  # scan metadata
      Positioners_name: $att1z.name  # dataset metadata (ICAT database field name)
      Positioners_value: $att1z.position  # dataset metadata (ICAT database field name)

Dataset metadata

Gathering

This metadata is gathered when starting a new dataset by the session icat-mapping object if one is provided in the session's yaml configuration.

Implementation

• The IcatPublisher interface has the metadata method. Classes with this interface are
  • MachInfo
  • motors.slits.Slits (slits controller)
  • MultiplePositions (mp controller)
• Then there are several classes which have a _init_meta_data_publishing which insert metadata in the scan info through the user metadata mechanism. Sometimes this uses metadata from the yaml config. Clearly some ICAT metadata gets added here and it shouldn't.
• In the yaml files, you have the IcatMeta class which tells the Bliss session what dataset metadata to gather. It looks like this:

- name: icat
  plugin: bliss
  class: ICATmeta

  objects:
    primary_slit: $primary_slit
    attenuator01: $att1

  positioners:
    sample: $roby
    insertion_device_gap: [$roby,$robz]

  mapping:
    InstrumentVariables_name: [ $roby.name, $robz.name ]
    InstrumentVariables_value: [ $roby.position, $robz.position ]

TODO

• Review those _init_meta_data_publishing methods and the yaml metadata fields and their usage in the controller classes to make sure scan data does not end up in ICAT and dataset metadata are end up in HDF5.
• Review the method names fill_meta_at_scan_start, fill_meta_at_scan_end, get_metadata, metadata, metadata_when_prepared and _init_meta_data_publishing and the yaml metadata field of MultiplePositions and perhaps the IcatMeta class. Basically the name should make it very clear whether this is scan metadata or dataset metadata.