Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • bliss bliss
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 538
    • Issues 538
    • List
    • Boards
    • Service Desk
    • Milestones
  • Jira
    • Jira
  • Merge requests 145
    • Merge requests 145
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • BlissBliss
  • blissbliss
  • Issues
  • #2769
Closed
Open
Issue created May 19, 2021 by Wout De Nolf@denolfMaintainer

Scan and dataset metadata

The difference between metadata in the scope of the scan (saved in the Nexus file) and metadata in the scope of the dataset (saved in ICAT, approx. 700 predefined field names) seems to have been lost somehow over time.

I tried to give an overview of the current situation for review purposes. (feel free to modify @valentin.valls @matias.guijarro)

Scan metadata

Gathering

This metadata is gathered at the start and end of each scan by scan.Scan._fill_meta wich will loop over the scan chain's acquisition objects and call fill_meta_at_scan_start or fill_meta_at_scan_end resp.

Implementation

  • The HasMetadataForScan interface has the metadata_when_prepared method. Classes with this interface are
    • lima_base.Lima (lima controller class)
    • mca.base.BaseMCA (mca controller class)
  • Every acquisition master or slave has 2 metadata gathering methods:
    class AcquisitionObject:
      def fill_meta_at_scan_start(self, scan_meta):
          device = self.device
          if isinstance(device, HasMetadataForScan):
              return device.metadata_when_prepared()
          return None
    
      def fill_meta_at_scan_end(self, scan_meta):
          return None
  • Classes derived from AcquisitionObject that overwrite these methods are:
    • BaseCounterAcquisitionSlave.fill_meta_at_scan_start
    • LimaAcquisitionMaster.fill_meta_at_scan_end
  • Every counter has a get_metadata method which is called by BaseCounterAcquisitionSlave.fill_meta_at_scan_start
    class Counter:
      def get_metadata(self):
         return {}
    This method is overwritten by lima.roi.RoiStatCounter, lima.roi.RoiProfileCounter and lima.roi.RoiCollectionCounter.
  • MultiplePositions has a metadata field in its yaml configuration which is mixing scan and dataset metadata:
    class: MultiplePositions
    positions:
    - label: Al
      metadata:
        "@NX_class": NXattenuator  # scan metadata
        type: Al  # scan metadata
        status: in  # scan metadata
        Positioners_name: $att1z.name  # dataset metadata (ICAT database field name)
        Positioners_value: $att1z.position  # dataset metadata (ICAT database field name)

Dataset metadata

Gathering

This metadata is gathered when starting a new dataset by the session icat-mapping object if one is provided in the session's yaml configuration.

Implementation

  • The IcatPublisher interface has the metadata method. Classes with this interface are
    • MachInfo
    • motors.slits.Slits (slits controller)
    • MultiplePositions (mp controller)
  • Then there are several classes which have a _init_meta_data_publishing which insert metadata in the scan info through the user metadata mechanism. Sometimes this uses metadata from the yaml config. Clearly some ICAT metadata gets added here and it shouldn't.
  • In the yaml files, you have the IcatMeta class which tells the Bliss session what dataset metadata to gather. It looks like this:
- name: icat
  plugin: bliss
  class: ICATmeta

  objects:
    primary_slit: $primary_slit
    attenuator01: $att1

  positioners:
    sample: $roby
    insertion_device_gap: [$roby,$robz]

  mapping:
    InstrumentVariables_name: [ $roby.name, $robz.name ]
    InstrumentVariables_value: [ $roby.position, $robz.position ]

TODO

  • Review those _init_meta_data_publishing methods and the yaml metadata fields and their usage in the controller classes to make sure scan data does not end up in ICAT and dataset metadata are end up in HDF5.
  • Review the method names fill_meta_at_scan_start, fill_meta_at_scan_end, get_metadata, metadata, metadata_when_prepared and _init_meta_data_publishing and the yaml metadata field of MultiplePositions and perhaps the IcatMeta class. Basically the name should make it very clear whether this is scan metadata or dataset metadata.
Edited May 23, 2021 by Wout De Nolf
Assignee
Assign to
Time tracking