Data policy: scan file path
To summarize the discussion on where data needs to be saved according to the data policy (@matias.guijarro @demariaa @andy.gotz). Part of the larger task #592 (closed). I'll give a description and a proposal at the end. I think it is simpler than you might expect.
- Scan file template
{root}/{proposal}/{beamline}/{sample}/{sample}_{dataset}/{sample}_{dataset}.h5
In the current SCAN_SAVING
this corresponds to:
-
SCAN_SAVING.base_path
:{root}
-
SCAN_SAVING.template
:{proposal}/{beamline}/{sample}/{sample}_{dataset}
-
SCAN_SAVING.data_filename
:{sample}_{dataset}.h5
- Base path of experiments managed by ICAT (supports e-logbook and archieval for example)
-
Normal experiments (visitors and industrial):
root = /data/visitor
-
Inhouse and commissioning experiments:
/root = /data/{beamline}/{inhouse}
For example BLC-1234, IH-CH-1234, IH-MI-1234 , IH-ES-1234, ...
Caveat: some beamlines have multiple inhouse folders "inhouse1", "inhouse2", ... At ID21 we have just "inhouse".
-
Test experiments (used during MDT or testing):
root = /data/{beamline}/{inhouse}
The name of the proposal has a fixed template
{beamline}{year}{month}
. For example id191911 is november 2019 at ID19
-
Base path of experiment not managed by ICAT (so no e-logbook and archieving):
/data/{beamline}/tmp/<anyname>
(devices on other machines CAN reach this folder)/tmp/<anyname>
(devices on other machines CANNOT reach this folder)
Proposal
- The user can use these commands to specify the data path indirectly
newexperiment(proposal=..., sample=..., dataset=...)
newsample(sample=..., dataset=...)
newdataset(dataset=...)
So all arguments are optional. When not specified, this is their default:
- proposal default:
{beamline}{year}{month}
- sample default:
default
- dataset default: numbering
"0001", "0002", ...
(check directory for existing ones and increment)
The root is derived from the proposal name:
- BLC* and IH* :
root = /data/{beamline}/{inhouse}
- all other cases:
root = /data/visitor
- Then for the beamline staff and BCU we can have experiments not managed by ICAT (for testing that does not need a logbook, archieval, metadata)
newtmpexperiment("somename", root=..., sample=..., dataset=...)
When the root is not specified root = /data/{beamline}/tmp
so that devices on other machines can reach the folder.
When you come to the beamline in january, you can type newexperiment()
and you're good to go.