Resolve "Scan and dataset metadata"
Closes #2769 (closed)
Fix the scan metdata vs. dataset metadata mixup. To achieve this we had to make all metadata related things more uniform as there were lots of conflicts.
Scan metadata
Two scan metadata protocols for Bliss controllers
-
HasMetadataForScan
:- abstract method
scan_metadata
: returns controller metadata - property
scan_metadata_name
: by default equal to the controller name - enable/disable per controller:
enable_scan_metadata
,disable_scan_metadata
- Redis destination of this metadata:
scan_metadata_name
key under the scan meta category "instrument" in scan_info - Metadata collection is done through a
ScanMeta
instance (see below)
- abstract method
-
HasMetadataForScanExclusive
: scan metadata will never be collected when not part of a scan- Derives from the
HasMetadataForScan
protocol:scan_metadata
,enable_scan_metadata
(still needs to be part of the scan),disable_scan_metadata
- Redis destination of this metadata: Redis node of the controller + "devices" section of scan_info
- Metadata collection is done through an
AcquistionObject
(see below)
- Derives from the
The time aspect is not included in these protocols. The timing is introduced through Scan
and AcquistionObject
.
Examples:
- controllers like lima, mca and all counters will use
HasMetadataForScanExclusive
because we only want their metadata when they are used in the scan. - controllers like multipositions, slits, machinfo and transfocators will use
HasMetadataForScan
because we always want their metadata for every scan.
Note: for MachInfo (HasMetadataForScan
protocol) the controller for scanning is the counter controller which does not produce any metadata. So even if we wanted only the metadata when MachInfo is part of the scan by using HasMetadataForScanExclusive
, you wouldn't see any metadata because MachInfo
itself is not the controller in the acq. chain.
ScanMeta class
- We still have the global
ScanMeta
instance for user metadata as before. Only now it is really for user metadata (no "positioners" or metadata automatically generated from controllers). - For each scan there is a separate
ScanMeta
instance which contains two metadata categories- "positioners" metadata category:
positioners_start
,positioners_end
, ... - "instrument" metadata category:
HasMetadataForScan.scan_metadata_name
keys withHasMetadataForScan.scan_metadata
values (excludingHasMetadataForScanExclusive
which goes throughAcquistionObject
)
- "positioners" metadata category:
AcquisitionObject class
The methods fill_meta_at_scan_start
and fill_meta_at_scan_end
are replaced by a single get_acquisition_metadata
with a META_TIMING
parameter. The AcquisitionObject
calls the HasMetadataForScan.scan_metadata
of the underlying controllers.
Counter class
The Counter
class is now derived from MetadataForScan
so that get_metadata
becomes scan_metadata
like all other controllers.
Scan class
Filling of the local scan_info
dictionary is no longer scattered all over the Scan
class code. We have:
-
_metadata_at_scan_instantiation
:- time of local collection: when instantiating the
Scan
object - time of Redis publication: when creating the
ScanNode
- time of local collection: when instantiating the
-
_metadata_at_scan_start
:- time of local collection: when the scan starts (
Scan.run()
called) - time of Redis publication: when creating the
ScanNode
- time of local collection: when the scan starts (
-
_metadata_at_scan_prepared
:- time of local collection: after all devices are prepared (this includes creating the Redis nodes)
- time of Redis publication: by the call to
ScanNode.prepared(...)
-
_metadata_at_scan_end
:- time of local collection: when the scan ends
- time of Redis publication: by the call to
ScanNode.end(...)
Apart from some custom scan metadata (session name, start time, end time, ...), there are three types of scan metadata:
- user scan metadata:
- one of the
ScanMeta
instances (see above)
- one of the
- controller scan metadata (
HasMetadataForScan
excludingHasMetadataForScanExclusive
):- one of the
ScanMeta
instances (see above)
- one of the
- acquisition controller scan metadata (
HasMetadataForScan
):- metadata collected through
AcquisitionObject
- metadata collected through
Dataset metadata
One dataset metadata protocol for Bliss controllers
Controllers with the HasMetadataForDataset
(with abstract method dataset_metadata
) will be used by the Bliss sessions's ICATmeta
controller to collect dataset metadata.
Yaml configuration
An overview of all metadata related configuration (not only related to this MR)
- Specify the
ICATmeta
controller for a Bliss session (dataset metadata)- class: Session name: test_session setup-file: ./test_setup.py icat-mapping: icat #
- Configuration for the sessions's
ICATmeta
controller (dataset metadata)- name: icat plugin: bliss class: ICATmeta objects: # need to have the HasMetadataForDataset protocol primary_slit: $primary_slit attenuator01: $att1 positioners: sample: $roby insertion_device_gap: [$roby,$robz] mapping: InstrumentVariables_name: [ $roby.name, $robz.name ] InstrumentVariables_value: [ $roby.position, $robz.position ]
- The
MultiplePositions
class still has its metadata configured but now splits scan and dataset metadata (backward compatible with existing yaml files with FutureWarning)name: att1 plugin: bliss class: MultiplePositions - label: Al40 description: Attenuator in Al40 position scan_metadata: "@NX_class": NXattenuator type: Al thickness: 40 thickness@unit: um status: in dataset_metadata: type: Al thickness: 40 status: in Positioners_name: $att1z.name Positioners_value: $att1z.position
MultiplePositions
is a multi purpose class. Maybe this should be improved but not in this MR.
Other changes
- Fix deepcopy problem of
ScanMeta
- Simplify
TransfocatorMockup
(no temporary object). - Add
__close__
toSlits
- Add more objects that generate metadata to the
nexus_writer_session
in the test suite - Add more scan metadata tests