installation_esrf.md 10.2 KB
Newer Older
1
2
3
# Installing BLISS at ESRF

At the ESRF, it is recommended to follow the Beamline Control Unit guidelines
4
5
6
for software installation. In the case of BLISS, a special [deployment procedure
using the Ansible tool](https://bliss.gitlab-pages.esrf.fr/ansible/index.html)
has been put in place in order to ease the work on beamlines.
7
8
9
10
11

## Updating BLISS installation

To update BLISS on an ESRF installation:

12
### release version (bliss)
13
14
15

For the "release" version in the `bliss` Conda environement, update the conda package:

16
17
18
19
20
21
22
```
conda update --channel esrf-bcu bliss
```
or
```
conda install bliss=X.Y.Z
```
23

24
### development version (bliss_dev)
25

26
For the development version, i.e in the `bliss_dev` Conda environement:
27

28
* update bliss repository:
29

30
31
32
    `cd local/bliss.git/`
    `git checkout master`
    `git pull`
33

34
* install up-to-date dependencies:
35

36
    `conda install --file ./requirements.txt`
37

38
* **Exit and re-enter** into the conda environment to ensure using up-to-date modules.
39

40
* Pip-install BLISS by creating a link in the conda environment directory pointing to the git repository:
41

42
    `pip install --no-deps -e .`
43
44
45
46
47
48
49

!!! note

    Make sure to keep the Conda channels up-to-date (using `conda info`) and correct, if
    needed:

    ```bash
50
    conda config --env --set channel_priority false
51
52
53
    conda config --env --add channels conda-forge
    conda config --env --append channels defaults
    conda config --env --append channels esrf-bcu
54
55
    conda config --env --append channels tango-controls
    ```
56
57
58
59
60
    NB:

    * `add` prepends
    * `append` moves to the bottom if already exists.

61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

### About BLISS version

At BLISS startup, its version is printed. This version's format depend on the
installation mode. If using a package-installed BLISS version, the package
number is printed:

```python
...
Welcome to BLISS version 1.1.0-359-gff1e64292 running on pcsht
Copyright (c) 2015-2019 Beamline Control Unit, ESRF
...
```

If using a `git`-installed BLISS, a cryptic `git` version number like
`1.1.0-359-gff1e64292` is printed. the three fields correspond to:

* `1.1.0`: last git tag number
* `359`: number of git commits since last tag
* `gff1e64292`: git hash: `ff1e64292`  (! without the `g`)


```python
...
Welcome to BLISS version 1.1.0-359-gff1e64292 running on pcsht
Copyright (c) 2015-2019 Beamline Control Unit, ESRF
...
```

The version can also be printed with:
```python
DEMO [1]: import bliss.release
DEMO [2]: bliss.release.version
 Out [2]: '1.1.0-359-gff1e64292'
```

## Post-installation configuration

### Instrument name

Cyril Guilloud's avatar
Cyril Guilloud committed
101
102
103
In order to properly fill information about the *instrument* on which data has
been collected in future data files, do not forget to set the `instrument`
field.
104

Cyril Guilloud's avatar
Cyril Guilloud committed
105
Format is free, but it is a good idea to put "ESRF-" followed by the
106
beamline or endstation name. In capital letters.
Cyril Guilloud's avatar
Cyril Guilloud committed
107

108
Example, in file:`__init__.yml` located at beamline configuration root, add:
109
110

```yaml
Cyril Guilloud's avatar
Cyril Guilloud committed
111
    ...
112
    instrument: ESRF-ID42A
Cyril Guilloud's avatar
Cyril Guilloud committed
113
    ...
114
115
116
117
```

### Nexus writer service

Cyril Guilloud's avatar
Cyril Guilloud committed
118
119
120
121
A TANGO device referred to as the *[Nexus writer](dev_data_nexus_server.md)*
saves all data produced by BLISS. It comes with any BLISS installation (no
additional package is required). Refer to the linked page to know about
installing this server.
122

Cyril Guilloud's avatar
Cyril Guilloud committed
123
124
125
!!! warning
    There must be **one Nexus writer device** per BLISS session. Do not
    forget to add a device when a new BLISS session is created.
126

127
128


129
130
### ESRF data policy

Cyril Guilloud's avatar
Cyril Guilloud committed
131
The ESRF data policy allows users to access their data and electronic logbook at
132
https://data.esrf.fr Data is registered with [ICAT](https://data.esrf.fr) and
Cyril Guilloud's avatar
Cyril Guilloud committed
133
134
135
136
137
138
139
140
141
142
the data written in [Nexus compliant](https://www.nexusformat.org/) HDF5 files
in a specific directory structure.

Two additional TANGO devices, installed automatically with BLISS (as
dependencies), need to be running and enabled for this. BLISS also needs to be
configured to use ESRF data policy.

!!! warning
    There must be **two ICAT server** per BLISS session. Do not
    forget to add the two devices when a new BLISS session is created.
143
144
145
146
147
148
149


#### ICAT Tango severs configuration

* `MetaExperiment` server handles the proposal and the sample
* `MetadataManager` server handles the dataset

Cyril Guilloud's avatar
Cyril Guilloud committed
150
151
These are referred to as the ICAT servers. They will inform the ICAT database
about the collected datasets during an experiment and they allow BLISS to
152
communicate with the **electronic logbook**.
Cyril Guilloud's avatar
Cyril Guilloud committed
153
154
155
156
157
158

On a beamline there can be multiple `MetadataManager` servers, each serving a
specific technique that needs a specific set of metadata parameters to be
registered with the ICAT database.

#### Using BEACON as the tango database
159

Cyril Guilloud's avatar
Cyril Guilloud committed
160
161
The registration can be done by defining servers and devices properties in the
beamline configuration.
162
163
164
165
166
167
168

```yaml
- class: MetaExperiment
  properties:
    queueName: "/queue/icatIngest"
    queueURLs:
        - bcu-mq-01.esrf.fr:61613
Cyril Guilloud's avatar
Cyril Guilloud committed
169
        - bcu-mq-02.esrf.fr:61613
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
- class: MetadataManager
  properties:
    queueName: "/queue/icatIngest"
    queueURLs:
        - bcu-mq-01.esrf.fr:61613
        - bcu-mq-02.esrf.fr:61613
    API_KEY: elogbook-be70ac55-fd08-4840-9b29-b73262958ca8
    icatplus_server: "https://icatplus.esrf.fr"
    server: "icat.esrf.fr"
    port: 443
    username: reader
- server: MetadataManager
  personal_name: icatservers
  device:
  - tango_name: id00/metadata/test
    class: MetadataManager
    properties:
      beamlineID: id00
      dataFolderPattern: "{dataRoot}"
      metaExperimentDevice: "id00/metaexp/test"
- server: MetaExperiment
  personal_name: icatservers
  device:
  - tango_name: id00/metaexp/test
    class: MetaExperiment
    properties:
      beamlineID: id00
```


Cyril Guilloud's avatar
Cyril Guilloud committed
200
201
#### Using the traditional tango database

202
The registration can be done by defining servers and devices properties in Jive.
Cyril Guilloud's avatar
Cyril Guilloud committed
203
204
205
206

A kind way to proceed is:

* to get templates files `manager.tango` and `experiment.tango`
207
208
209
* to adapt templates files:
    - replace `idXX` by bemaline name
    - replace `session_WWZ` by the session name
Cyril Guilloud's avatar
Cyril Guilloud committed
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
* to load templates in Jive
* to create device for other sessions, just copy devices using Jive:
    - in **Server** tab
    - unfold  `MetadataManager` ; `idxx` ; `MetadataManager` ; `idxx/metadata/sessionXXX`
    - right clic on existing device and then :  **rename**   or  **create device** / **copy**


`manager.tango`:
```
MetadataManager/idXX/DEVICE/MetadataManager: "idXX/metadata/session_WWZ"

idXX/metadata/session_WWZ->beamlineID: idXX
idXX/metadata/session_WWZ->dataFolderPattern: {dataRoot}
idXX/metadata/session_WWZ->metaExperimentDevice: "idXX/metaexp/idXX"

idXX/metadata/session_WWZ/dataFolder->__value: ""
idXX/metadata/session_WWZ/datasetName->__value: ""
idXX/metadata/session_WWZ/metadataFile->__value: ""

CLASS/MetadataManager->API_key: elogbook-be70ac55-fd08-4840-9b29-b73262958ca8
230
CLASS/MetadataManager->icatplus_server: "https://icatplus.esrf.fr"
Cyril Guilloud's avatar
Cyril Guilloud committed
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
CLASS/MetadataManager->queueName: "/queue/icatIngest"
CLASS/MetadataManager->queueURLs: bcu-mq-01.esrf.fr:61613,\
                                  bcu-mq-02.esrf.fr:61613
```


`experiment.tango`:
```
MetaExperiment/idXX/DEVICE/MetaExperiment: "idXX/metaexp/session_WWZ"

idXX/metaexp/session_WWZ->beamlineID: idXX

idXX/metaexp/session_WWZ/dataRoot->__value: ""
idXX/metaexp/session_WWZ/proposal->__value: ""
idXX/metaexp/session_WWZ/sample->__value: ""

CLASS/MetaExperiment->queueName: "/queue/icatIngest"
CLASS/MetaExperiment->queueURLs: bcu-mq-01.esrf.fr:61613,\
                                 bcu-mq-02.esrf.fr:61613
```




255
##### MetaExperiment server
256

Cyril Guilloud's avatar
Cyril Guilloud committed
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
Class properties:

![Meta experiment class properties](img/data_policy/jive_metaexp_classprops.jpg)

These properties are used to register [datasets](data_policy.md#change-dataset).

They are the same for all the beamlines.

* `queueName`: "/queue/icatIngest"
* `queueURLs`:
      - `bcu-mq-01.esrf.fr:61613`
      - `bcu-mq-02.esrf.fr:61613`



Devices:

![Meta experiment device properties](img/data_policy/jive_metaexp_props.jpg)

The devices have only one property to be adapted for the beamline:

* `beamlineID`: `id00`


281
##### MetadataManager server
Cyril Guilloud's avatar
Cyril Guilloud committed
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297


These properties are used to send messages to the [electronic
logbook](data_metadata.md#electronic-logbook).

They are the same for all the beamlines.

* `API_KEY`:`elogbook-be70ac55-fd08-4840-9b29-b73262958ca8`
    - security key to connect to elogbook
* `icatplus_server`: `https://icatplus.esrf.fr`
* `queueName`: "/queue/icatIngest"
* `queueURLs`:
    - `bcu-mq-01.esrf.fr:61613`
    - `bcu-mq-02.esrf.fr:61613`

```
298
Not present in jive ???
Cyril Guilloud's avatar
Cyril Guilloud committed
299
300
301
302
303
304
305
306
307
308
309
310
 server: "icat.esrf.fr"
 port: 443
 username: reader
```

Class properties:

![Metadata manager class properties](img/data_policy/jive_metadata_manager_classprops.jpg)

Devices:

![Metadata manager device properties](img/data_policy/jive_metadata_manager_props.jpg)
311
312

!!! note
313
     `MetaExperiment` must be started **before** `MetadataManager`.
314

315
316
#### Enable in BLISS

Cyril Guilloud's avatar
Cyril Guilloud committed
317
318
Finally, data policy must be enabled in BLISS. This is done by adding a
dedicated section in the BLISS configuration, either:
319

Cyril Guilloud's avatar
Cyril Guilloud committed
320
321
322
323
* In file: `__init__.yml` at beamline configuration root
* or together with a session configuration
    - this is particularly useful when the same Beacon configuration is used by
      multiple endstations
Matias Guijarro's avatar
Matias Guijarro committed
324
325

The section that has to be added is:
326
327

```yaml
328
329
330
331
332
333
334
335
...
scan_saving:
    class: ESRFScanSaving
    beamline: id00
    tmp_data_root: /data/{beamline}/tmp
    visitor_data_root: /data/visitor
    inhouse_data_root: /data/{beamline}/inhouse
...
Cyril Guilloud's avatar
Cyril Guilloud committed
336
```
Matias Guijarro's avatar
Matias Guijarro committed
337
338


Cyril Guilloud's avatar
Cyril Guilloud committed
339
340
341
342
343
344
345
346
!!! note
    The beamline name specified under `scan_saving:` will be used to find
    the metadata servers: `id00/metadata/<session_name>` and
    `id00/metaexp/<session_name>`. So it must correspond to the Tango
    device domain name.
    **There must be 2 metadata Tango devices running per BLISS session.**
    (+1 nexus writer) Do not forget to add them for each new BLISS session.

347
348
##### Multiple mount points

349
Multiple mount points can be configured for each proposal type (visitor, inhouse and temp) and optionally for the icat servers (`MetadataManager` and `MetaExperiment`)
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366

```yaml
...
scan_saving:
    inhouse_data_root:
        nfs: /data/{beamline}/inhouse
        lsb: /lsbram/{beamline}/inhouse
    icat_inhouse_data_root: /data/{beamline}/inhouse
...
```

The active mount points can be selected in BLISS

```python
DEMO [1]: SCAN_SAVING.mount_point = "lsb"
```

367
The default mount point is `SCAN_SAVING.mount_point == ""` which selects the first mount point in the configuration.