Skip to content

Resolve "Protect metadata gathering from parallel scans"

Closes #2185 (closed)

Introduction of ICAT metadata and the Dataset in Redis caused 2 race conditions for parallel scans:

  • ESRFScanSaving.on_scan_run called in multiple greenlets on a clone of the same scan saving:
    • issue: it raised on exception that the metadata was already gathered
    • solution: dataset.gather_metadata(on_exists="skip") no longer raises an exception when the metadata is already gathered. Calling dataset.gather_metadata() still raises an exception when the metadata is already gathered (and it should).
  • get_node of a Dataset in Redis would sometimes return a DataNodeContainer:
    • issue: get_node would return a DataNode even when it is not fully instantiated yet (in this case the node_type is not set yet)
    • solution: the principal Redis key of the DataNode now has a version property (str) which is set after DataNode.__init__ is completed with create=True and before adding itself to its parent.
  • as a side effect: nodes with unsupported versions in Redis are ignored (by default) by the utility functions get_node, get_nodes, etc.

Note that the last change doesn't require a db=1 flush. get_node will return None if the node's version doesn't match the current implementation (before it returned None when the name property was None; which meant the key didn't exist or was in the process of being created). So Flint and the writer will not accept nodes that are not supported.

Edited by Wout De Nolf

Merge request reports