Resolve "Protect metadata gathering from parallel scans"
Closes #2185 (closed)
Introduction of ICAT metadata and the Dataset
in Redis caused 2 race conditions for parallel scans:
-
ESRFScanSaving.on_scan_run
called in multiple greenlets on a clone of the same scan saving:- issue: it raised on exception that the metadata was already gathered
- solution:
dataset.gather_metadata(on_exists="skip")
no longer raises an exception when the metadata is already gathered. Callingdataset.gather_metadata()
still raises an exception when the metadata is already gathered (and it should).
-
get_node
of aDataset
in Redis would sometimes return aDataNodeContainer
:- issue:
get_node
would return aDataNode
even when it is not fully instantiated yet (in this case thenode_type
is not set yet) - solution: the principal Redis key of the
DataNode
now has aversion
property (str
) which is set afterDataNode.__init__
is completed withcreate=True
and before adding itself to its parent.
- issue:
- as a side effect: nodes with unsupported versions in Redis are ignored (by default) by the utility functions
get_node
,get_nodes
, etc.
Note that the last change doesn't require a db=1 flush. get_node
will return None
if the node's version doesn't match the current implementation (before it returned None
when the name
property was None
; which meant the key didn't exist or was in the process of being created). So Flint and the writer will not accept nodes that are not supported.
Edited by Wout De Nolf