Scan and dataset metadata
The difference between metadata in the scope of the scan (saved in the Nexus file) and metadata in the scope of the dataset (saved in ICAT, approx. 700 predefined field names) seems to have been lost somehow over time.
I tried to give an overview of the current situation for review purposes. (feel free to modify @valentin.valls @matias.guijarro)
Scan metadata
Gathering
This metadata is gathered at the start and end of each scan by scan.Scan._fill_meta
wich will loop over the scan chain's acquisition objects and call fill_meta_at_scan_start
or fill_meta_at_scan_end
resp.
Implementation
- The
HasMetadataForScan
interface has themetadata_when_prepared
method. Classes with this interface are-
lima_base.Lima
(lima controller class) -
mca.base.BaseMCA
(mca controller class)
-
- Every acquisition master or slave has 2 metadata gathering methods:
class AcquisitionObject: def fill_meta_at_scan_start(self, scan_meta): device = self.device if isinstance(device, HasMetadataForScan): return device.metadata_when_prepared() return None def fill_meta_at_scan_end(self, scan_meta): return None
- Classes derived from
AcquisitionObject
that overwrite these methods are:BaseCounterAcquisitionSlave.fill_meta_at_scan_start
LimaAcquisitionMaster.fill_meta_at_scan_end
- Every counter has a
get_metadata
method which is called byBaseCounterAcquisitionSlave.fill_meta_at_scan_start
class Counter: def get_metadata(self): return {}
lima.roi.RoiStatCounter
,lima.roi.RoiProfileCounter
andlima.roi.RoiCollectionCounter
. -
MultiplePositions
has ametadata
field in its yaml configuration which is mixing scan and dataset metadata:class: MultiplePositions positions: - label: Al metadata: "@NX_class": NXattenuator # scan metadata type: Al # scan metadata status: in # scan metadata Positioners_name: $att1z.name # dataset metadata (ICAT database field name) Positioners_value: $att1z.position # dataset metadata (ICAT database field name)
Dataset metadata
Gathering
This metadata is gathered when starting a new dataset by the session icat-mapping object if one is provided in the session's yaml configuration.
Implementation
- The
IcatPublisher
interface has themetadata
method. Classes with this interface areMachInfo
-
motors.slits.Slits
(slits controller) -
MultiplePositions
(mp controller)
- Then there are several classes which have a
_init_meta_data_publishing
which insert metadata in the scan info through the user metadata mechanism. Sometimes this usesmetadata
from the yaml config. Clearly some ICAT metadata gets added here and it shouldn't. - In the yaml files, you have the
IcatMeta
class which tells the Bliss session what dataset metadata to gather. It looks like this:
- name: icat
plugin: bliss
class: ICATmeta
objects:
primary_slit: $primary_slit
attenuator01: $att1
positioners:
sample: $roby
insertion_device_gap: [$roby,$robz]
mapping:
InstrumentVariables_name: [ $roby.name, $robz.name ]
InstrumentVariables_value: [ $roby.position, $robz.position ]
TODO
- Review those
_init_meta_data_publishing
methods and the yamlmetadata
fields and their usage in the controller classes to make sure scan data does not end up in ICAT and dataset metadata are end up in HDF5. - Review the method names
fill_meta_at_scan_start
,fill_meta_at_scan_end
,get_metadata
,metadata
,metadata_when_prepared
and_init_meta_data_publishing
and the yamlmetadata
field ofMultiplePositions
and perhaps theIcatMeta
class. Basically the name should make it very clear whether this is scan metadata or dataset metadata.