data_policy.md 7.16 KB
Newer Older
1
2
3
A data policy determines data structure (file format and directory structure)
and registeration of data collection with external services. BLISS comes with
two data policies
Wout De Nolf's avatar
Wout De Nolf committed
4

5
6
7
8
1. The [ESRF data policy](#esrf-data-policy) which allows users to access their
   data and electronic logbook at https://data.esrf.fr. The data is written in
   [Nexus compliant](https://www.nexusformat.org/) HDF5 files in a specific
   directory structure.
Wout De Nolf's avatar
Wout De Nolf committed
9

10
!!! note
11
12
    Read more about [configuring ESRF data
    policy](installation_esrf.md#ESRF data policy)
Wout De Nolf's avatar
Wout De Nolf committed
13

14
15
16
17
2. The [basic data policy](#basic-data-policy) does not impose a data directory
   structure or register data with any external service. Data can (but does not
   have to be) written in [Nexus compliant](https://www.nexusformat.org/) HDF5
   files. The basic data policy is the default policy for BLISS.
Wout De Nolf's avatar
Wout De Nolf committed
18

Cyril Guilloud's avatar
Cyril Guilloud committed
19
20
21
22

![Data diagram](img/data_ESRF_paths.svg)


Wout De Nolf's avatar
Wout De Nolf committed
23
24
## ESRF data policy

25
This data policy requires the user to specify *proposal*, *collection* and
26
*dataset*. This will completely define how data is organized.
Wout De Nolf's avatar
Wout De Nolf committed
27
28
29

### Change proposal

30
```python
Wout De Nolf's avatar
Wout De Nolf committed
31
DEMO  [1]: newproposal("blc123")
Matias Guijarro's avatar
Matias Guijarro committed
32
Proposal set to 'blc123'
Wout De Nolf's avatar
Wout De Nolf committed
33
34
Data path: /data/id00/inhouse/blc123/id00/sample/sample_0001
```
35
36
37
38

When no proposal name is given, the default proposal is inhouse proposal
`{beamline}{yymm}`. For example at ID21 in January 2020 the default proposal
name is `id212001`.
Wout De Nolf's avatar
Wout De Nolf committed
39
40
41
42
43
44
45
46

The data root directory is derived from the proposal name

* no name given: `/data/{beamline}/inhouse/`
* *ih** and *blc**: `/data/{beamline}/inhouse/`
* *test**, *tmp** or *temp**: `/data/{beamline}/tmp/`
* all other names: `/data/visitor/`

47
48
These root path can be [configured](installation_esrf.md#ESRF data policy) but
these are the defaults.
Wout De Nolf's avatar
Wout De Nolf committed
49

50
51
52
### Change collection

A collection is a group of datasets that share some characteristics
Wout De Nolf's avatar
Wout De Nolf committed
53

54
```python
55
56
DEMO  [2]: newcollection("sample1")
Dataset collection set to 'sample1'
Wout De Nolf's avatar
Wout De Nolf committed
57
58
59
Data path: /data/id00/inhouse/blc123/id00/sample1/sample1_0001
```

60
61
62
63
64
65
66
67
68
69
When no collection name is given, the default "sample" is used. Note
that you can always come back to an existing collection to add more dataset.

When the datasets in a collection share the same sample, you can use

```python
DEMO  [2]: newsample("sample1")
Dataset collection set to 'sample1'
Data path: /data/id00/inhouse/blc123/id00/sample1/sample1_0001
```
Wout De Nolf's avatar
Wout De Nolf committed
70
71
72
73
74

### Change dataset

#### Named datasets

75
```python
Wout De Nolf's avatar
Wout De Nolf committed
76
DEMO  [3]: newdataset("area1")
Matias Guijarro's avatar
Matias Guijarro committed
77
Dataset set to 'area1'
Wout De Nolf's avatar
Wout De Nolf committed
78
79
80
Data path: /data/id00/inhouse/blc123/id00/sample1/sample1_area1
```

81
82
83
When the dataset already exists the name will be automatically incremented
("area1_0002", "area1_0003", ...). Note that you can never come back to the same
dataset after you changed dataset.
Wout De Nolf's avatar
Wout De Nolf committed
84
85
86

#### Unnamed datasets

87
```python
Wout De Nolf's avatar
Wout De Nolf committed
88
DEMO  [4]: newdataset()
Matias Guijarro's avatar
Matias Guijarro committed
89
Dataset set to '0002'
Wout De Nolf's avatar
Wout De Nolf committed
90
91
92
Data path: /data/id00/inhouse/blc123/id00/sample1/sample1_0002
```

93
94
95
The dataset will be named automatically "0001", "0002", ... The dataset number
is independent for each sample. Note that you can never come back to the same
dataset after you changed dataset.
Wout De Nolf's avatar
Wout De Nolf committed
96
97
98
99
100

### Policy state

To get an overview of the current state of the data policy

101
```python
Wout De Nolf's avatar
Wout De Nolf committed
102
DEMO  [5]: SCAN_SAVING
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
  Out [5]: Parameters (default) -

     .user_name             = 'denolf'
     .images_path_template  = 'scan{scan_number}'
     .images_prefix         = '{img_acq_device}_'
     .date_format           = '%Y%m%d'
     .scan_number_format    = '%04d'
     .dataset_number_format = '%04d'
     .session               = 'demo'
     .date                  = '20200208'
     .scan_name             = '{scan_name}'
     .scan_number           = '{scan_number}'
     .img_acq_device        = '<images_* only> acquisition device name'
     .writer                = 'nexus'
     .data_policy           = 'ESRF'
118
     .template              = '{proposal_name}/{beamline}/{sample_name}/{sample_name}_{dataset_name}'
119
     .beamline              = 'id00'
120
     .proposal_name         = 'blc123'
121
122
     .proposal_type         = 'inhouse'
     .base_path             = '/data/id00/inhouse'
123
124
125
     .sample_name           = 'sample1'
     .dataset_name          = '0001'
     .data_filename         = '{sample_name}_{dataset_name}'
126
127
128
129
130
131
132
133
     .images_path_relative  = True
     .creation_date         = '2020-02-08-12:09'
     .last_accessed         = '2020-02-08-12:12'
   ---------  ---------  -------------------------------------------------------------------
   exists     filename   /data/id00/inhouse/blc123/id00/sample1/sample1_0001/sample1_0001.h5
   exists     directory  /data/id00/inhouse/blc123/id00/sample1/sample1_0001
   Metadata   RUNNING    Dataset is running
   ---------  ---------  -------------------------------------------------------------------
Wout De Nolf's avatar
Wout De Nolf committed
134
135
```

136
137
138
139
140
141
142
143
144
145
#### MetadataManager state

The state of the MetadataManager device can be

 * OFF: No experiment ongoing
 * STANDBY: Experiment started, sample or dataset not specified
 * ON: No dataset running
 * RUNNING: Dataset is running
 * FAULT: Device is not functioning correctly

146
147
148
Every time a scan is started, BLISS verifies that the dataset as specified in
the session's `SCAN_SAVING` object is *RUNNING*. If this is not the case, BLISS
will close the previous running dataset (if any) and start the new dataset.
149

Wout De Nolf's avatar
Wout De Nolf committed
150
151
## Basic data policy

152
153
154
155
This data policy requires the user to use the
[`SCAN_SAVING`](dev_data_policy_basic.md#scan_saving) object directly to define
where the data will be saved. The data location is completely determined by
specifying *base_path*, *template*, *data_filename* and *writer*
Wout De Nolf's avatar
Wout De Nolf committed
156

157
```python
Wout De Nolf's avatar
Wout De Nolf committed
158
159
160
161
162
163
164
165
166
167
DEMO  [1]: SCAN_SAVING.base_path = "/tmp/data"
DEMO  [2]: SCAN_SAVING.writer = "nexus"
DEMO  [3]: SCAN_SAVING.template = "{date}/{session}/{mysubdir}"
DEMO  [4]: SCAN_SAVING.date_format = "%y%b"
DEMO  [5]: SCAN_SAVING.add("mysubdir", "sample1")
DEMO  [6]: SCAN_SAVING.data_filename = "scan{scan_number}"
DEMO  [7]: SCAN_SAVING.filename
  Out [7]: '/tmp/data/20Feb/demo/sample1/scan{scan_number}.h5'
```

168
169
170
Note that each attribute can be a template string to be filled with other
attributes from the [`SCAN_SAVING`](dev_data_policy_basic.md#scan_saving)
object.
Wout De Nolf's avatar
Wout De Nolf committed
171
172
173
174
175

### Policy state

To get an overview of the current state of the data policy

176
```python
Wout De Nolf's avatar
Wout De Nolf committed
177
DEMO [8]: SCAN_SAVING
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
 Out [8]: Parameters (default) -

    .base_path            = '/tmp/data'
    .data_filename        = 'scan{scan_number}'
    .user_name            = 'denolf'
    .template             = '{date}/{session}/{mysubdir}'
    .images_path_relative = True
    .images_path_template = 'scan{scan_number}'
    .images_prefix        = '{img_acq_device}_'
    .date_format          = '%y%b'
    .scan_number_format   = '%04d'
    .mysubdir             = 'sample1'
    .session              = 'demo'
    .date                 = '20Feb'
    .scan_name            = '{scan_name}'
    .scan_number          = '{scan_number}'
    .img_acq_device       = '<images_* only> acquisition device name'
    .writer               = 'nexus'
    .data_policy          = 'None'
    .creation_date        = '2020-02-08-12:04'
    .last_accessed        = '2020-02-08-12:05'
    --------  ---------  -----------------------------------------------------------------
    exists    filename   /tmp/data/20Feb/demo/sample1/scan{scan_number}.h5
    exists    directory  /tmp/data/20Feb/demo/sample1
    --------  ---------  -----------------------------------------------------------------
Wout De Nolf's avatar
Wout De Nolf committed
203
```