Commit 3cf25bf5 authored by Matias Guijarro's avatar Matias Guijarro

Doc refactoring after 1st "training" session about Configuration

parent 65d6e179
Pipeline #8500 passed with stages
in 20 minutes and 34 seconds
# Introduction to Beacon
*Beacon* is the server process, that is at the corner stone of BLISS since it
provides services on which essential features of BLISS are built upon:
* **configuration database**
- text files in [*YAML*](https://yaml.org) format
- **complete description of an entire beamline
experimental setup at a centralized place**: devices, sessions, etc.
* **configuration editor** web application
* *settings*, i.e. **runtime configuration properties**
* *channels*, to **exchange data between different BLISS objects instantiated in different processes**
* **distributed lock management**, to be able to lock and release resources for a particular Beacon client
* **publishing of data produced during scans**, for online data analysis or visualisation
Beacon relies on [*Redis*](https://redis.io) for most of those services. Redis
is an open source, in-memory data structure store, used as a database, cache and
message broker.
In addition, Beacon can be used as a **TANGO database-compatible server**,
storing TANGO configuration information in Beacon YAML files.
## Beacon services overview
![Beacon services overview](img/beacon_services.png)
!!! note
Many BLISS features depend on Beacon. In practice it is not possible to use
BLISS without a Beacon server.
# Beacon channels
*Channels* use the built-in [publish/subscribe features of Redis](https://redis.io/topics/pubsub)
in order to provide a simple way to exchange data between different BLISS processes.
Contrary to Settings, channels data is **not persisted**. When the last
client holding the data disconnects, the channel data is cleared.
Use cases for channels are:
* to update state between processes
- for example, for `Axis` objects, `position` and `state` (`MOVING`, `READY`...) are shared between listeners
* caching for performance, e.g. to avoid reloading
- to memorize last loaded program in a MUSST or Opiom
- to memorize MCA parameters
* to prevent unwanted hardware initialization
- if an object is shared between multiple processes, is in use and is already initialized, there is no need (it can even be harmful) to re-initialize it
## `Channel` object usage
Process A:
```py
>>> from bliss.config.channels import Channel
>>> c = Channel("test", default_value=42)
>>> c.value
42
```
Process B:
```py
>>> from bliss.config.channels import Channel
>>> c = Channel("test")
>>> c.value
42
```
It is possible to register a callback to get notified of Channel value change events:
```py
# in process A
>>> def c_value_changed(new_value):
print(new_value)
>>> c.register_callback(c_value_changed)
```
```py
# in process B
>>> c.value = "something new"
# in process A, output:
something new
```
## `Cache` object
A Cache object is a helper to make a Channel attached to an object from the
configuration. It guarantees the name of the channel corresponds to the
object, by pre-pending the name of the object to the corresponding key in Redis.
```py
>>> from bliss.config.static import get_config
>>> config = get_config()
>>> obj = config.get("my_object")
>>> from bliss.config.channels import Cache
>>> cached_value = Cache(obj, "my_cached_value", default_value="X")
# the cached_value object is a Channel
>>> cached_value.value = "something"
```
The `clear_cache(*devices)` function from `bliss.config.channels` deletes all
cached values for the list of devices provided as function arguments.
!!! note
When the last client holding a channel disconnects, the channel is
removed. It is cleared from Redis. In case another channel with the same
name is created afterwards, reading it returns the default value.
The path specified with `--db_path` on the beacon server command line
points to the root directory of the **configuration database**.
Beacon uses plain text files in [YAML](http://yaml.org/) format (with `.yml`
extension) to describe the set of objects representing a BLISS system.
The idea is to have **a centralized configuration database per beamline**.
!!! info
YAML has been chosen because the standard native types (list,
dictionary, numbers, unicode strings...) can be easily mapped to Python
equivalents, it handles comments (contrary to JSON) and also because it is
optimized for human-readability (contrary to XML).
Beacon goes all over the configuration database directories, and builds
an internal representation of the objects that are defined in YAML
mapping nodes from the files. Ultimately this structure is flattened and
exposed as a Python dictionary with key-value pairs, keys being object names
and values being the corresponding configuration information.
## YAML items
A *YAML item* is a key-value pair, using `key: value` syntax. A *YAML mapping*
is a set of YAML items:
```yaml
# example YAML mapping node with 3 nodes
example_key: 1 #float value
example_key2: ['a', 'b', 'c'] # list value
example_key3: a string #string value
```
!!! info
An [online YAML parser](http://yaml-online-parser.appspot.com/) can help finding YAML syntax errors.
## Definition of a Beacon object
A Beacon object is simply defined by creating a `name` item within a YAML mapping:
```yaml
# example BLISS object
name: my_bliss_object
param1: 42
param2: example string
```
!!! warning
**name** items values must be unique for the whole configuration
Multiple objects can be defined in one file by using the YAML list notation `-`
to declare a sequence of mappings:
```yaml
# example of multiple objects
- name: object1
param1: ...
...
- name: object2
param1: ...
...
```
!!! note
**Files are *transparent***
- for each mapping defined in the configuration, **Beacon is only sensitive
to the presence of a `name` key**
- multiple files are just a convenient way of grouping related objects
within a container
## Tree structure
### Directories
Contrary to files, directories play a role in the organization of the
configuration database. Beacon's **internal tree structure** mirrors the
file-system directories.
The following database files:
beamline_configuration/
└── directory_1
└── file_1.yml
```YAML
# file_1.yml contents:
-
name: object_1
param: 42
-
name: object_2
param: 43
```
Produce the following internal representation:
```py
>>> from bliss.config.static import get_config
>>> config = get_config() # load YAML tree from Beacon server, and parse it to create a 'config' object
>>> config.pprint()
{ filename: None
directory_1:
[
{ filename: 'directory_1/file_1.yml'
name: object_1
param: 42
},
{ filename: 'directory_1/file_1.yml'
name: object_2
param: 43
}
]
}
>>> config.names_list # get existing names from config
['object_1', 'object_2']
```
Each YAML mapping is attached to a **parent node**. In this case, the parent
node of the two mappings defined in `file_1.yml` is `directory_1`.
### `__init__.yml` files
It is possible to associate mappings to a directory itself by adding a
`__init__.yml` file.
This can be used to define **global items**, with a scope limited to their
children:
beamline_configuration/
└── directory_1
└── file_1.yml
└── __init__.yml
```YAML
# __init__.yml file contents
my_global: hello
```
```py
>>> config.reload() # since the files changed, reload all config from server
>>> object_1 = config.get("object_1") #indexing by name (flattened dictionary)
>>> object_1.get_inherited("my_global")
hello
```
### Example configuration
Here is an excerpt of the ESRF ID31 beamline configuration, as an example of a
real-world files structure:
beamline_configuration/
|
├── EH
│   ├── detectors
│   │   ├── diodes.yml
│   │   ├── __init__.yml
│   │   ├── maxipix.yml
│   │   └── pilatus.yml
│   ├── motion
│   │   ├── aerotech.yml
│   │   ├── eh.yml
│   │   ├── energy.yml
│   │   ├── iceid314.yml
│   │   ├── iceid315.yml
│   │   └── __init__.yml
│   ├── samenv
│   │   ├── furnace.yml
│   │   ├── gasblower.yml
│   │   ├── gasrig.yml
│   │   ├── __init__.yml
│   │   ├── ls336.yml
│   │   └── pressure.yml
│   ├── wagos
│   │   ├── __init__.yml
│   │   ├── wcid31gasload1.yml
│   │   ├── wcid31l.yml
│   │   └── wcid31o.yml
│   └── __init__.yml
├── musst_prog #
│   ├── acc_step_scan.mprg # Any kind of file can be
│   ├── contscan.mprg # put in the configuration
│   └── laser.mprg #
├── OH1
│   ├── motion
│   │   ├── iceid311.yml
│   │   └── __init__.yml
│   ├── __init__.yml
│   ├── transfocator.yml
│   └── wbv1.yml
├── sessions
│   ├── scripts
│   │   ├── alignment.py
│   │   ├── gasblower.py
│   │   └── microstation.py
│   ├── __init__.yml
│   ├── sixc.py
│   └── sixc.yml
├── beacon.rdb # Redis database automatic dump
├── frontend.yml
├── __init__.yml
├── multiplexer.yml
├── musst.yml
├── p201.yml
├── safety_shutter.yml
└── undulator.yml
!!! note
Non-YAML files are ignored by Beacon for creating objects and mappings, but a remote client can still retrieve any file. This is part of
Beacon centralized file hosting.
#### Using Beacon API
##### Retrieving all objects names
>>> from bliss.config.conductor import client as beacon
>>> beacon.get_config_db_tree()
<returns
>>> file_like_object = beacon.remote_open(<filename>)
>>> file_contents = file_like_object.read()
# Running Beacon
The Beacon process can start up to 4 servers:
* Configuration database server
* Redis server
* (optional) configuration application web server
* (optional) TANGO database server
## Running Beacon at ESRF
At ESRF, the BLISS installation procedure automatically adds Beacon to the
daemons started by the system:
* The port number for the Beacon server is set to 25000
* The Redis server is started on port 25001
* The configuration files directory is set to `/users/blissadm/local/beamline_configuration`
* The configuration web application is available at `http://localhost:9030`
* The Beacon TANGO database service is disabled
!!! note
At ESRF there is at least one Beacon server per beamline.
## Custom startup
It is required to start Beacon server using `--db_path` to specify the path to the configuration files:
$ beacon-server --db_path=~/local/beamline_configuration
Beacon port number can be set manually (otherwise, by default, Beacon will just choose the first free port it finds):
$ beacon-server --db_path=~/local/beamline_configuration --port=25000
!!! note
Beacon implements a discovery protocol (in the same spirit as [SSDP](https://en.wikipedia.org/wiki/Simple_Service_Discovery_Protocol)).
Within the same sub-network clients will find a Beacon automatically.
But it is safer to specify where to connect manually, using the `BEACON_HOST`
environment variable to point to `<machine>:<port>` (example: `bibhelm:25000`).
The web configuration UI can be enabled, by specifying the web application port number using `--webapp_port`:
$ beacon-server --db_path=~/local/beamline_configuration --port=25000 --webapp_port=9030
BLISS Beacon server is also able to provide a full TANGO database server service that integrates nicely
with the BLISS configuration. To start this service it is just needed to provide the TANGO port that
you want the TANGO database server to serve:
$ beacon-server --db_path=~/local/beamline_configuration --port=25000 --webapp_port=9030 --tango_port=20000
# Beacon settings
*Beacon settings* are helper classes to store data in the Redis database at
runtime from Python code, using BLISS.
Use cases for settings:
* on top of static configuration parameters
- motor velocity, acceleration, limits
- auto-gain on a Keithley
* to keep (or share) information across executions (or across processes)
- selected counter for plot
- enabled loggers, log level
- scan saving path, file template...
## Settings classes
BLISS provides different classes for different kinds of settings.
### SimpleSetting
The `SimpleSetting` class is used to save a *scalar value*:
* int
* float
* string
* bool
Example:
```py
from bliss.config import settings
magicNumber = 63825363
iii = settings.SimpleSetting('myIntkey', default_value=magicNumber)
assert iii.get() == magicNumber
iii.set(42)
assert iii.get() == 42
assert isinstance(iii.get(), int)
```
The `default_value` is returned by the `SimpleSetting` object until it has been
set. Once the object has been set, the value is persisted in redis.
### HashSetting
The `HashSetting` class is used to represent a dictionary of scalar values:
```py
myDict = {"C1":"riri", "C2":"fifi"}
shs = settings.HashSetting('myHkey', default_values=myDict) # note the s :)
```
### QueueSetting
The `QueueSetting` class is used to represent a list of scalar values:
```py
>>> myList = ["a", "b", "c", "d"]
>>> sqs = settings.QueueSetting('myQkey')
>>> sqs.set(myList)
```
### Parameters
`Parameters` are more advanced objects -- it is used to group
simple settings, and to be able to switch from one set of values for
the parameters to another:
```py
>>> from bliss.config.settings import Parameters
>>> p = Parameters("my_params")
>>> p.add("test", 42)
>>> p
Parameters (default)
.test = 42
>>> p.test
42
>>> p.test=43
>>> p.configs
['default']
>>> p.switch("another_config")
>>> p.add("test2", 10)
>>> p
Parameters (another_config)
.test2 = 10
>>> p.switch("default")
>>> p
Parameters (default)
.test = 43
>>>
```
# Static configuration API
## Cheatsheet
```py
>>> from bliss.config.static import get_config
>>> #load YAML tree from Beacon server and parse it
>>> #into a Python 'Config' object
>>> config = get_config() # returns Config singleton
>>> #get all existing names in config
>>> config.names_list
['object_1', 'object_2']
>>> #reload config from server
>>> config.reload()
>>> #get 'object_1' configuration
>>> object_1_config = config.get_config('object_1')
>>> object_1_config
filename:<directory_1/file_1.yml>,plugin:None,{'name': 'object_1', 'param': 42}
>>> type(object_1_config)
<class 'bliss.config.static.Node'>
>>> #access with dict interface
>>> object_1_config['param']
42
>>> object_1_config['param'] = 0
>>> #send config to server
>>> object_1_config.save()
>>> #instantiate 'object_1' object
>>> object_1 = config.get('object_1')
>>> #getting any file from the server
>>> from bliss.config.conductor.client import remote_open
>>> # file-like object
>>> with remote_open("directory_1/file_1.yml") as f:
>>> print(f.read())
b'- !!omap\n - name: object_1\n - param: 0\n- !!omap\n - name: object_2\n - param: 43\n'
```
# Beacon and YAML
## Beacon
### plugin
Some Beacon plugins are provided to instantiate YAML configuration
tree into BLISS python objects usable in a session or a BLISS
sequence.
* `session`: to furnish configuration
* `emotion`: to instantiate BlissAxis objects (motors)
* `ct2`: for P201 counting cards
* `keithley`: for Keithley electrometers
* `temperature`: for temperature controllers
* `comm`: for standalone communication objects (serial lines, TCP, GPIB)
* `bliss`: a generic plugin to instantiate controllers
A new plugin can be easily created by a developer to fit new needs.
## YAML in brief
* .yml is the extention used for YAML formated files.
* YAML stands for Yet Another Markup Language
* `$` character indicates a **reference** to an existing axis
see also :
* http://yaml.org/
* https://fr.wikipedia.org/wiki/YAML
* in french:
* http://sweetohm.net/article/introduction-yaml.html
* https://learnxinyminutes.com/docs/fr-fr/yaml-fr/
* on-line parser: http://yaml-online-parser.appspot.com/
......@@ -6,7 +6,7 @@ For now is only tested on **Ensemble** controllers.
### Supported features
Encoder | Shutter | Trajectories
Encoder | Shutter | Trajectories
------- | ------- | ------------
YES | NO | NO
......@@ -49,5 +49,3 @@ in the YAML configuration file.
- name: rot_enc
aero_name: X
```
More information about **Encoder** objects [here](motion_encoder.md)
......@@ -4,8 +4,8 @@ This chapter explains how to configure an Galil motor controller.
### Supported features
Encoder | Shutter | Trajectories
------- | ------- | ------------
Encoder | Shutter | Trajectories
------- | ------- | ------------
NO | NO | NO
### Specific Galil controller parameters
......@@ -15,20 +15,20 @@ NO | NO | NO
### Specific IcePAP axis parameters
* **type**: is the motor type, default value is SERVO == 1. Values could be:
* 1 : Servo Motor
* -1 : Servo motor with reversed polarity
* -2 : Step motor with active high step pulses
* 2 : Step motor with active low step pulses
* -2.5: Step motor with reversed direction and active high step pulses
* 2.5: Step motor with reversed direction and active low step pulses
- `1`: Servo Motor
- `-1`: Servo motor with reversed polarity
- `-2`: Step motor with active high step pulses
- `2`: Step motor with active low step pulses
- `-2.5`: Step motor with reversed direction and active high step pulses
- `2.5`: Step motor with reversed direction and active low step pulses
* **vect_acceleration**: acceleration rate of the vector in a coordinated motion sequence. default value is 262144
* **vect_deceleration**: deceleration rate of the vector in a coordinated motion sequence. default value is 262144
* **vect_slewrate**: speed of the vector in a coordinated motion sequence. default value is 8192
* **encoder_type**: the quadrature type or the pulse and direction type. default value is QUADRA == 0. Values can be:
* 0: Normal quadrature
* 1: Normal pulse and direction
* 2: Reversed quadrature
* 3: Reversed pulse and direction
* **vect_slewrate**: speed of the vector in a coordinated motion sequence. Default value is 8192
* **encoder_type**: the quadrature type or the pulse and direction type. Default value is QUADRA == 0. Values can be:
- `0`: Normal quadrature
- `1`: Normal pulse and direction
- `2`: Reversed quadrature
- `3`: Reversed pulse and direction
* **kp**: Proportional Constant. default value is 1.0
* **ki**: Integrator. default value is 6.0
* **kd**: Derivative Constant. default value is 7.0
......@@ -55,4 +55,4 @@ volts. default value is set to the maximum.
velocity: 14.4
acceleration: 28.8
unit: deg
```
\ No newline at end of file
```
# MD2 Difractometer
# MD2 Difractometer
![MD2](img/md2.png)
......@@ -12,6 +12,12 @@ https://www.embl.fr/instrumentation/cipriani/downloads/md2_pdf.pdf
Some troubleshooting on usage:
https://www.esrf.eu/UsersAndScience/Experiments/MX/How_to_use_our_beamlines/Trouble_Shooting/id29-microdiffractometer-troubleshooting
### Supported features
Encoder | Shutter | Trajectories
------- | ------- | ------------
NO | NO | NO
## Underlining Control ##
MD2 uses the Exporter protocol.
......@@ -53,4 +59,4 @@ controller:
root_name: "AlignmentZ"
```
Usually the only parameter that should be changed is the `exporter_address`.
\ No newline at end of file
Usually the only parameter that should be changed is the `exporter_address`.
# Measurement groups
A measurement group is a container for counters. The measurement group helps to deal
with a coherent set of counters. For example, a measurement group can represent
counters related to a detector, a hutch or an experiment.
# BLISS measurement group
Measurement groups are loaded by the `session` Beacon plugin, thus it
is possible to configure those directly in a session YAML file:
```yaml
- class: MeasurementGroup
name: align_counters
counters: [simct1, simct2, simct3]
- class: MeasurementGroup
name: MG1
counters: [simct2, simct3]
This chapter explains how to create and to deal with *measurement
groups*.
A measurement group is an object to wrap counters in it. The measurement group helps to deal with a
coherent set of counters.
For example, a measurement group can represent the counters related to
a detector, a hutch or an experiment.
## Creation
A measurement group can be defined it in the YML file of a session:
- class: MeasurementGroup
name: align_counters
counters: [simct1, simct2, simct3]
- class: MeasurementGroup
name: MG1
counters: [simct2, simct3]
- class: MeasurementGroup
name: MG2
counters: [simct4, simct5]
- class: MeasurementGroup
name: MG2
counters: [simct4, simct5]
```
`counters` must be a list of names, corresponding to `Counter` objects.
[Read more about Counter objects](scan_ctmg.md)
## Usage
......@@ -38,7 +30,7 @@ Once a measurement group is created, it can be used in a BLISS session:
CYRIL [1]: align_counters
Out [1]: MeasurementGroup: align_counters (default)
Enabled Disabled
------- -------
simct1
......@@ -48,127 +40,109 @@ Once a measurement group is created, it can be used in a BLISS session:
One or many measurement group(s) can be passed as argument to a `scan`
or `ct` procedure to indicate which counters to use:
CYRIL [20]: print MG1.available, MG2.available # 4 counters defined
CYRIL [20]: print(MG1.available, MG2.available) # 4 counters defined
['simct2', 'simct3'] ['simct4', 'simct5']
CYRIL [21]: timescan(0.1, MG1, MG2, npoints=3)
Total 3 points, 0:00:00.300000 (motion: 0:00:00, count: 0:00:00.300000)
Scan 15 Wed Feb 21 16:31:48 2018 /tmp/scans/cyril/ cyril user = guilloud
timescan 0.1
# dt(s) simct2 simct3 simct4 simct5
0 0.0347409 0.50349 0.494272 0.501698 0.496145
1 0.13725 0.49622 0.503753 0.500348 0.500601
2 0.2391 0.502216 0.500213 0.494356 0.493359
Took 0:00:00.395435 (estimation was for 0:00:00.300000)
Took 0:00:00.395435 (estimation was for 0:00:00.300000)