Commit 406c23dd authored by Matias Guijarro's avatar Matias Guijarro

rename file in developers' corner with dev_ prefix

parent 034c4e34
Pipeline #21288 passed with stages
in 5 minutes and 24 seconds
Detectors publish their own metadata by default. Here we describe how to add user metadata. A more flexible and presistent way to add metadata is described [here](data_metadata_dev.md).
Detectors publish their own metadata by default. Here we describe how to add user metadata. A more flexible and presistent way to add metadata is described [here](dev_data_metadata.md).
## Electronic logbook
Send user message to the [electronic logbook](https://data.esrf.fr)
......@@ -16,4 +16,4 @@ DEMO [2]: s.add_comment("This is a comment")
DEMO [3]: s.add_comment("This is another comment")
DEMO [4]: s.add_comment("And another one")
DEMO [4]: s.run()
```
\ No newline at end of file
```
Currently Bliss supports only one data format: [Nexus compliant](https://www.nexusformat.org/) HDF5 files written by the [Nexus writer](data_nexus_server.md). Here we describe the logic of this Nexus structure.
Currently Bliss supports only one data format: [Nexus compliant](https://www.nexusformat.org/) HDF5 files written by the [Nexus writer](dev_data_nexus_server.md). Here we describe the logic of this Nexus structure.
In the example below we show the file of [one dataset](data_policy.md) which contains the data of three scans:
......@@ -76,4 +76,4 @@ So each scan contains two groups (plots and application definitions are not show
* *positioners*: like start_positioners
* *measurement*: flat list of all NXpositioner and NXdetector data
Note that each *NXdetector* contains one primary value called *data* and each *NXpositioner* contains one primary value called *value*. Additional datasets and groups represent secondary detector/positioner data or metadata such as detector settings.
\ No newline at end of file
Note that each *NXdetector* contains one primary value called *data* and each *NXpositioner* contains one primary value called *value*. Additional datasets and groups represent secondary detector/positioner data or metadata such as detector settings.
......@@ -4,7 +4,7 @@ A data policy determines data structure (file format and directory structure) an
2. The [basic data policy](#basic-data-policy) does not impose a data directory structure or register data with any external service. Data can (but does not have to be) written in [Nexus compliant](https://www.nexusformat.org/) HDF5 files.
Installation and configuration of the [ESRF](data_policy_dev_esrf.md) and [basic](data_policy_dev_basic.md) data policy in a BLISS session can be found elsewhere as well as how to create [custom](data_policy_dev_custom.md) data policies. Below we describe how to use the data policies.
Installation and configuration of the [ESRF](dev_data_policy_esrf.md) and [basic](dev_data_policy_basic.md) data policy in a BLISS session can be found elsewhere as well as how to create [custom](dev_data_policy_custom.md) data policies. Below we describe how to use the data policies.
## ESRF data policy
......@@ -26,7 +26,7 @@ The data root directory is derived from the proposal name
* *test**, *tmp** or *temp**: `/data/{beamline}/tmp/`
* all other names: `/data/visitor/`
These root path can be [configured](data_policy_dev_esrf.md#configuration) but these are the defaults.
These root path can be [configured](dev_data_policy_esrf.md#configuration) but these are the defaults.
### Change sample
......@@ -102,7 +102,7 @@ DEMO [5]: SCAN_SAVING
## Basic data policy
This data policy requires the user to use the [`SCAN_SAVING`](data_policy_dev_basic.md#scan_saving) object directly to define where the data will be saved. The data location is completely determined by specifying *base_path*, *template*, *data_filename* and *writer*
This data policy requires the user to use the [`SCAN_SAVING`](dev_data_policy_basic.md#scan_saving) object directly to define where the data will be saved. The data location is completely determined by specifying *base_path*, *template*, *data_filename* and *writer*
```
DEMO [1]: SCAN_SAVING.base_path = "/tmp/data"
......@@ -115,7 +115,7 @@ DEMO [7]: SCAN_SAVING.filename
Out [7]: '/tmp/data/20Feb/demo/sample1/scan{scan_number}.h5'
```
Note that each attribute can be a template string to be filled with other attributes from the [`SCAN_SAVING`](data_policy_dev_basic.md#scan_saving) object.
Note that each attribute can be a template string to be filled with other attributes from the [`SCAN_SAVING`](dev_data_policy_basic.md#scan_saving) object.
### Policy state
......
Open the [Nexus compliant](https://www.nexusformat.org/) HDF5 file written by the [Nexus writer](data_nexus_server.md) with [pymca](http://pymca.sourceforge.net/)
Open the [Nexus compliant](https://www.nexusformat.org/) HDF5 file written by the [Nexus writer](dev_data_nexus_server.md) with [pymca](http://pymca.sourceforge.net/)
```bash
pymca /data/visitor/hg123/id21/sample/sample_0001/sample_0001.h5
......
Open the [Nexus compliant](https://www.nexusformat.org/) HDF5 file written by the [Nexus writer](data_nexus_server.md) with [silx](http://www.silx.org/)
Open the [Nexus compliant](https://www.nexusformat.org/) HDF5 file written by the [Nexus writer](dev_data_nexus_server.md) with [silx](http://www.silx.org/)
```bash
silx view /data/visitor/hg123/id21/sample/sample_0001/sample_0001.h5
......
If you want to look at the HDF5 file written by the [Nexus writer](data_nexus_server.md) during a scan, use [silx](data_vis_silx.md) or [pymca](data_vis_pymca.md). Do not use third-party tools or custom scripts that are not approved by the ESRF Data Analysis Unit. More details can be found [here](data_nexus_server.md#concurrent-reading).
If you want to look at the HDF5 file written by the [Nexus writer](dev_data_nexus_server.md) during a scan, use [silx](data_vis_silx.md) or [pymca](data_vis_pymca.md). Do not use third-party tools or custom scripts that are not approved by the ESRF Data Analysis Unit. More details can be found [here](dev_data_nexus_server.md#concurrent-reading).
!!! warning
A reader should never open the HDF5 file in append mode (which is the default in `h5py`). Even when only performing read operations, this will result in a corrupted file!
......
......@@ -6,8 +6,8 @@ storage). In Redis, data is stored for a limited period of time (1 day by defaul
Two primary [Redis](https://redis.io/) subscribers are provided by BLISS
1. The [Nexus writer](data_nexus_server.md) for writing [Nexus compliant](https://www.nexusformat.org/) HDF5 files.
1. The [Nexus writer](dev_data_nexus_server.md) for writing [Nexus compliant](https://www.nexusformat.org/) HDF5 files.
2. [Flint](flint_scan_plotting.md) for online data visualization
[Custom subscribers](data_subscribing.md) can be created for other types of data processing.
[Custom subscribers](dev_data_subscribing.md) can be created for other types of data processing.
......@@ -6,7 +6,7 @@ This policy is meant for testing only. It does not enforce data structure (file
To enable the basic data policy
1. Install and run the [Nexus writer](data_nexus_server.md) to write the data in Nexus format (optional)
1. Install and run the [Nexus writer](dev_data_nexus_server.md) to write the data in Nexus format (optional)
2. Specify file directory and name using the [SCAN_SAVING](#scan_saving) object in the BLISS session
......@@ -82,7 +82,7 @@ a dictionary, whose key `root_path` is the final path to scan files.
#### SCAN_SAVING writer
`.writer` is a special member of `SCAN_SAVING`; it indicates which
writer to use for saving data. BLISS supports `"hdf5"` (internal writer in BLISS), `"nexus"` (the [Nexus writer](data_nexus_server.md)) and `"null"` (writing disabled).
writer to use for saving data. BLISS supports `"hdf5"` (internal writer in BLISS), `"nexus"` (the [Nexus writer](dev_data_nexus_server.md)) and `"null"` (writing disabled).
### Configuration example
......
......@@ -6,11 +6,11 @@ The ESRF data policy allows users to access their data and electronic logbook at
To enable the ESRF data policy
1. Install and run the [Nexus writer](data_nexus_server.md) to write the data in Nexus format
1. Install and run the [Nexus writer](dev_data_nexus_server.md) to write the data in Nexus format
2. Install and run the [ICAT servers](data_policy_servers.md) to communicate with ICAT
2. Install and run the [ICAT servers](dev_data_policy_servers.md) to communicate with ICAT
3. Enable the ESRF data policy in the BLISS session to configure the data directory structure. This is done in the beamline configuration which will contain a mixture of [data policy configuration](#configuration) and [ICAT server configuration](data_policy_servers.md#enable-in-bliss):
3. Enable the ESRF data policy in the BLISS session to configure the data directory structure. This is done in the beamline configuration which will contain a mixture of [data policy configuration](#configuration) and [ICAT server configuration](dev_data_policy_servers.md#enable-in-bliss):
```yaml
scan_saving:
......
The [ESRF data policy](data_policy_dev_esrf.md) allows users to access their data and electronic logbook at https://data.esrf.fr. Two TANGO devices need to be installed, running and enabled for this.
The [ESRF data policy](dev_data_policy_esrf.md) allows users to access their data and electronic logbook at https://data.esrf.fr. Two TANGO devices need to be installed, running and enabled for this.
## Summary
......
......@@ -40,7 +40,7 @@ the session is created and the scan data is stored in this directory.
As two different samples will be scanned, one sub-directory per sample will be
created. To do that, the [SCAN_SAVING](data_policy_dev_basic.md#scan_saving) object has to
created. To do that, the [SCAN_SAVING](dev_data_policy_basic.md#scan_saving) object has to
be used. The data saving path is customized by adding a new parameter
'*s_name*' usable in the template of the PATH.
......
......@@ -4,7 +4,7 @@ BLISS offers the possibility to have a separate process (on the system level) fo
## Example 1: Save data in HDF5
!!! note
BLISS already comes with an [HDF5 writer](data_nexus_server.md). This is just an example.
BLISS already comes with an [HDF5 writer](dev_data_nexus_server.md). This is just an example.
The example script discussed here is provided in Bliss repository
at [scripts/external_saving_example/external_saving_example.py](https://gitlab.esrf.fr/bliss/bliss/blob/master/scripts/external_saving_example/external_saving_example.py).
To have a minimal working Bliss environment have a look at the [installation notes](installation.md#installation-outside-esrf)
......@@ -43,7 +43,7 @@ Using the `walk_on_new_events()` function with `filter="scan"`(limit walk to no
In the example script a new instance of the class `HDF5_Writer` is created per scan that is started. Following the
initialisation a _gevent greenlet_ is spawned to run the actual listener in a non blocking way. Inside `def run(self)`
a second iterator is started walking through all events emitted by the scan
[(see data structure section)](data_publishing.md#experiment-and-redis-data-structure):
[(see data structure section)](dev_data_publishing.md#experiment-and-redis-data-structure):
```python
for event_type, node in self.scan_node.iterator.walk_events():
......@@ -54,7 +54,7 @@ a second iterator is started walking through all events emitted by the scan
- [Overview scan engine](scan_engine.md)
- [Bliss data nodes](scan_data_node.md)
- [Data structure of published data](data_publishing.md)
- [Data structure of published data](dev_data_publishing.md)
Once an event is received it can be categorized by the event type:
......
......@@ -396,7 +396,7 @@ Out [1]: Parameters (default)
.writer = 'hdf5'
```
Find more info about how to use it in [SCAN_SAVING section](data_policy_dev_basic.md#scan_saving)
Find more info about how to use it in [SCAN_SAVING section](dev_data_policy_basic.md#scan_saving)
### Retrieving scan data
......
......@@ -162,19 +162,19 @@ nav:
- Speedgoat: config_speedgoat.md
- timedisplay: timedisplay.md
- Data:
- Architecture: data_architecture.md
- Publishing: data_publishing.md
- Subscribing: data_subscribing.md
- Architecture: dev_data_architecture.md
- Publishing: dev_data_publishing.md
- Subscribing: dev_data_subscribing.md
- Data policy:
- ESRF: data_policy_dev_esrf.md
- Basic: data_policy_dev_basic.md
- Custom: data_policy_dev_custom.md
- Nexus writer: data_nexus_server.md
- ESRF: dev_data_policy_esrf.md
- Basic: dev_data_policy_basic.md
- Custom: dev_data_policy_custom.md
- Nexus writer: dev_data_nexus_server.md
- Flint:
- Editing Lima ROI counters: flint_roi_counters.md
- For scan designers: flint_scan_info.md
- ICAT: data_policy_servers.md
- Metadata: data_metadata_dev.md
- ICAT: dev_data_policy_servers.md
- Metadata: dev_data_metadata.md
theme:
name: material
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment