Commit 7426120d authored by Mauro Rovezzi's avatar Mauro Rovezzi
Browse files

merge upstream

parents 9e713aa6 7fd531b0
Pipeline #49221 failed with stages
in 0 seconds
......@@ -45,7 +45,7 @@ check_style:
- master
- pip install -r requirements-dev.txt
- conda install --file requirements-dev.txt
# run black
- LC_ALL=C.UTF-8 black --check --safe $(git diff origin/$COMPARE_BRANCH_NAME... --diff-filter=ACMR --name-only | grep \\.py$)
......@@ -57,7 +57,7 @@ check_style_master:
- bliss_master
- pip install -r requirements-dev.txt
- conda install --file requirements-dev.txt
# run black
- LC_ALL=C.UTF-8 black --check --safe $(git diff origin/$COMPARE_BRANCH_NAME... --diff-filter=ACMR --name-only | grep \\.py$)
......@@ -65,7 +65,7 @@ check_lint:
stage: style
image: continuumio/miniconda3:latest
- pip install -r requirements-dev.txt
- conda install --file requirements-dev.txt
# run flake8 on diff between current branch and last common ancestor with master
- git diff -U0 origin/$COMPARE_BRANCH_NAME...$CI_COMMIT_SHA | flake8 --diff
# allow failure without impacting the rest of the CI (will show an orange warning in the UI)
......@@ -88,7 +88,7 @@ check_lint:
# install Xvfb and opengl libraries (needed for test_flint)
- apt-get update && apt-get -y install xvfb libxi6
# create test env and install BLISS
- conda create --quiet --name testenv --file requirements-conda.txt --file requirements-test-conda.txt
- conda create --quiet --name testenv --file requirements.txt --file requirements-test.txt
- source activate testenv
- conda install pytest-profiling --yes
- pip install . --no-deps
......@@ -211,7 +211,7 @@ package:
# install opengl libraries (needed to avoid problem with pyopengl dependency)
- apt-get update && apt-get -y install libgl1-mesa-glx
# create package env and install all requirements and conda-build, (gcc and g++ compiler required for flint)
- conda create --quiet --name buildenv --file requirements-conda.txt --file requirements-test-conda.txt conda-build
- conda create --quiet --name buildenv --file requirements.txt --file requirements-test.txt conda-build
- source activate buildenv
# create links to reach prefixed compilers of conda
- ln -s /opt/conda/envs/buildenv/bin/x86_64-conda_cos6-linux-gnu-gcc /opt/conda/envs/buildenv/bin/gcc
......@@ -285,7 +285,7 @@ create_reference_doc:
# install opengl libraries (needed to avoid problem with pyopengl dependency)
- apt-get update && apt-get -y install libgl1-mesa-glx
# create doc env and install all requirements
- conda create -q --yes --name docenv --file requirements-conda.txt --file requirements-doc-conda.txt
- conda create -q --yes --name docenv --file requirements.txt --file requirements-doc.txt
- source activate docenv
# build of documentation
- python build_sphinx
......@@ -302,7 +302,7 @@ create_user_doc:
# install opengl libraries (needed to avoid problem with pyopengl dependency)
- apt-get update && apt-get -y install libgl1-mesa-glx
# create doc env and install all requirements
- conda create -q --yes --name mkdocsenv --file requirements-conda.txt --file requirements-doc-conda.txt
- conda create -q --yes --name mkdocsenv --file requirements.txt --file requirements-doc.txt
- source activate mkdocsenv
# build of documentation (-s : strict : fail on warnings)
- cd doc && mkdocs build -s
......@@ -321,7 +321,7 @@ create_user_doc:
- mv bliss source # to avoid import errors (we want to test the packet, not local bliss folder)
- conda create -y --name testenv
- source activate testenv
- conda install bliss==$CI_COMMIT_TAG --file requirements-test-conda.txt --channel file://${CI_PROJECT_DIR}/conda-local-channel
- conda install bliss==$CI_COMMIT_TAG --file requirements-test.txt --channel file://${CI_PROJECT_DIR}/conda-local-channel
- echo ${PYTEST_ARGS}
- pytest ${PYTEST_ARGS}
......@@ -10,10 +10,94 @@ and this project adheres to [Semantic Versioning](
### Added
- Flint
- Added a `time-curve-plot` custom plot
- Added a new API from custom curve plots
- Rewritten custom plot user documentation including examples
- scan_info:
- Added description for curve plots
### Changed
- Flint
- Custom plot API
- `plot_image` now uses `ImageView` (which includes histogram)
- `plot_image_with_histogram` is deprecated (use `plot_inage`)
- `plot_scatter` signature was changed (x, y, value have to be individually
- `select_data`/`deselect_data`/`add_data` are deprecated
- The regulation plot now uses `time-curve-plot`
### Fixed
- Flint
- Fixed many inconsistencies in the custom plot API
### Removed
- Flint
- Custom plot API
- Removed method `add_single_data`
## [1.8.0 - 2021-05-28]
### Added
- new 'goto_min()' standard shell function
- diffractometer support
- calculations using libhkl from F.Picca (SOLEIL)
- new HKLTrajectoryMaster for scans
- 47 new spec-like commands: br, ubr, ca, ci, pa, setmode, ..., hklscan, etc.
- Axis
- user message when changing velocity or acceleration
- encoder steps per unit in info string
- Frontend shutter displays message on mode setting error
- ESRF CITY synchronization device support
- OPIOM output signals generator
- MOCO controller
- add set_default_config method which set parameters from yml file
- new oprange method
- check inbeam/outbeam methods
- take source=SOFT into account for 'inbeam' method
- allow 'undulator_prefix' in addition to 'undu_prefix' in Undulator yml configuration
- Writer
- prevents HDF5 file corruption in case of full disk
- print warning every 3 seconds if free disk space is below 1 GB
- FAULT state (will stop scan) if free space is below 200 MB
- add chunking and compression
- Display positions of real axes of calc motors during 'umv'
- MCS LA2000 Linear Servo Amplifier
- TwoMotorMaster: acquisition master for 2 independent motors
- additional check for good type of master and device in custom default chain
- ScanWagoHook: init, post_move, pre_move only called at first motion and last motion of a scan
- esrf hexapode reports metadata
- CT2
- Integrator functionality
- server: multi-board support for spec
- Lima
- NOSAVING mode handling
- ROI counters validity checks
- automatic adaptation of ROI after camera geometry changes
- improve error message on connection failure
- Shell
- new hardware initialization display during BLISS session initialization
- new 'tw' standard shell command
- display a graphical interface to move selected motors and count
- log message at BLISS startup
- session creation script displays Beacon server
- Flint
- Added a tool to reset a curve plot to the used plotselect
- Added dedicated widget for acqobj exposing 1D data
- Only 1D data from this acqobj is displayed
- Supports metadata from controllers or acqobj to custom the X-axis
- `xaxis_channel`, `xaxis_array`
- Added data display as index for curve plots and onedim plots
- Group MCA channels per detectors in the curve plot property tree
- Added histogram tool when displaying image in custom plots
- Added a content menu option to center profile ROIs in image/scatter
- Added curve stack as custom plot
- Added an overlay with size of Lima rect ROIs
- Added tools to duplicate/rename ROIs during ROI edition
- Better handling of timeout, and try not to have 30s
- Better handling of stucked state
- Added `restart_flint` command from `bliss.common.plot`
......@@ -25,19 +109,120 @@ and this project adheres to [Semantic Versioning](
- Added `--enable-watchdog` command line argument to log and kill Flint if
too much memory is used
- `scan_info["requests"]` is not anymore read (replaced by `channels`)
- Update to silx 0.15
- Update to silx 0.15.1
- Demo
- Added regulation mock to the demo session
- Scan publication
- Added device/channel metadata to the `scan_info`
- Added a `PREPARED` event with an updated scan_info
- `AcqObj.fill_meta_at_scan_start` is used to fill to `scan_info`
- Added metadata `type` for Lima detector and MCAs
- Added `ScansWatcher` and `ScansObserver` to replace `watch_session_scans`
### Changed
- Shell
- based on ptpython 3
- integration with asyncio with aiogevent
- starts in monochrome mode
- relative scans (dscans) translated to absolute scans in data saving
- ICEPAP motor controller: better error message if connection cannot be established
- raises CommunicationError exception
- Lima
- '.use_background_substraction' is replaced by 'use_background' True/False, and 'background_source'image/file
- progress bar refactoring
- new progress bar based on tqdm, avoid memory leak
- progress bar is no more intertwined with scan chain
- ct output: refactoring of formatting code
- Flint
- When Flint is not fast enough to reach data from Redis, NaN values
are used in order to keep the data alignment
- Improve initial curve plot selection to care about plotselect or user selection
The earlier one have the priority
- XIA mca
- Logger improved (can log at handel lib and BLISS levels)
- Run server according to config retrieved from beacon
- Client re-connection on server restart
- Improved management of current_configuration / default_configuration
- Project: Remove `-conda` suffix from requirement files
- PI E712 E753
- uniformization of communication and recorder.
- "wave" motion generator.
### Fixed
- Motion hook: post_scan is now executed after motor moved back to original position in case of 'dscan'
- better identification of communication errors
- when server is restarted, reconfigures device automatically
- block size adjusts automatically for Mercury and FalconX
- avoid collision between server and service methods
- disconnect callback
- reporting of BaseException raised in server to client
- reporting of Timeout exception
- Settings
- Struct: avoid too many calls to redis by using `__getattr__` instead of `__getattribute__`
- use redis to determine last access time
- CalcMotorController: avoid recursive calls in '_real_position_update'
- Display of Alias column in lscnt()
- Measurement Group
- fix counters returned in random order
- Flint
- Fix initial curve plot selection in order to properly reuse user selection
- Fixed slow rendering occurred on live curves and scatters with fast scans
- The video image is now also used for Lima EXTRERNAL_TRIGGER and EXTERNAL_GATE
- Fixed blinking of the regulation plot legend
- Fixed hidden ROIs during a scan. A tool is provided to display them
if not already selected.
- Fixed update of the property view after an update of the backend
- Fixed colormap LUT of the scatter plot when it is set with the style dialog
### Removed
- import of SpecClient_gevent in spec communication module
- as a consequence, "spec array" type is not supported
- .statistics for scans
- was source of a memory leak
- for now: replaced by debugging with yappi
- equivalent, graphical version based on work by Linus before he left will be proposed
## [1.7.4 - 2021-05-27]
### Added
- saving of Lima ROI collection
- regulation
- new Pace controller
- Eurotherm 2000 (old temperature framework Eurotherm is kept for compatibility)
- new Smaract MCS2 motor controller
- new "oscilloscope" module, new Lecroy oscilloscope
### Changed
- Bump silx version to 0.14.1
### Fixed
- Newport XPS: autoHome is now properly set to False by default, config checks for boolean value
- fix logging of unicode characters in Electronic Logbook
- CalcCounterController now works with aliases
- fixed bad sleep time in integrating counter reading loop
- removed 3-second hardcoded timeout waiting for STARTING state in scan group
- saving: ensure dataset is sent to ICAT even when missing Redis nodes (incomplete metadata)
- reset md5 cache on MUSST after RESET command is called
- scanning: ensure reading tasks are killed when acquisition stop fails
- Pilatus hardware ROIs
- Aerotech motor controller bug in "._cmd" method
- P201 (client side): clear buffer at the end of reading, in case acq. obj is reused
- Elmo controller, added retries and communication improvement
- Calc motor controller and dial_position with disabled caching
- Flint
- Default workspace per BLISS session
- Fixed slow rendering occurred on live curves and scatters with fast scans
- The video image is now also used for Lima EXTRERNAL_TRIGGER and EXTERNAL_GATE
- Fixed memory leak on OpenGL when a widget is not visible
- Fixed blinking of the regulation plot legend
### Removed
......@@ -57,7 +242,7 @@ and this project adheres to [Semantic Versioning](
- Wago modules catalogue: 750-342,352,363,515
- Writer
- explicit exception if parent node is missing
- OFF and RUNNING states, now means respecitively "not listening to events and resources released" and "writer alive"
- OFF and RUNNING states, now means respectively "not listening to events and resources released" and "writer alive"
### Changed
......@@ -97,6 +282,9 @@ and this project adheres to [Semantic Versioning](
### Added
- Flint
- Added a tool to display many scans in the curve widget
### Changed
### Fixed
......@@ -6,7 +6,7 @@ Bliss
The bliss control library.
Latest documentation from master can be found [here](
Check the [latest documentation from the master](
In short
......@@ -14,7 +14,7 @@ In short
To update BLISS from source:
conda install --file ./requirements-conda.txt
conda install --file ./requirements.txt
exit and re-enter into conda environment
......@@ -91,6 +91,7 @@ from bliss import global_map
from bliss.common.msgpack_ext import MsgpackContext
from bliss.comm.exceptions import CommunicationError
from bliss.common.greenlet_utils import Timeout as GreenletTimeoutError
MAX_MEMORY = min(psutil.virtual_memory().total, sys.maxsize)
......@@ -541,17 +542,36 @@ class RpcConnection:
# Disable Nagle and set TOS to low delay
self._socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
self._socket.setsockopt(socket.SOL_IP, socket.IP_TOS, 0x10)
except socket.gaierror as sockexc:
_msg = f"RPC socket.gaierror connecting to {}:{self.port} - ERR no {sockexc.errno} : {sockexc.strerror}"
raise CommunicationError(_msg) from sockexc
except socket.gaierror as sock_err:
# host does not exist ?
_msg = f"RPC - socket.gaierror connecting to {}:{self.port} - ERR no {sock_err.errno} : {sock_err.strerror}"
raise CommunicationError(_msg) from sock_err
except socket.timeout as sock_err:
# ?
_msg = (
f"RPC - socket.timeout error connecting to {}:{self.port}"
raise CommunicationError(_msg) from sock_err
except GreenletTimeoutError as grt_err:
# host is in DNS but OFF ?
_msg = (
f"RPC - GreenletTimeoutError connecting to {}:{self.port}"
raise CommunicationError(_msg) from grt_err
except ConnectionRefusedError as cnx_err:
_msg = f"RPC ConnectionRefusedError error connecting to {}:{self.port}"
# port not open ?
_msg = f"RPC - ConnectionRefusedError '{cnx_err}' to {}:{self.port}"
raise CommunicationError(_msg) from cnx_err
# ??? no host
# why no host ?
self._socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
self._reading_task = gevent.spawn(self._raw_read)
def get_class(self):
......@@ -593,6 +613,9 @@ class RpcConnection:
def _call__(self, code, args, kwargs, retry_on_disconnect=True):
log_debug(self, f"rpc client ({self._address}): '{code}' args={args}")
if self._socket is None:
# Check if already return a sub client
method_name = args[0] if args else ""
value = self._subclient.get((code, method_name))
......@@ -639,9 +662,15 @@ class RpcConnection:
def _raw_read(self):
unpacker = msgpack.Unpacker(raw=False, max_buffer_size=MAX_BUFFER_SIZE)
exception = None
while True:
msg = self._socket.recv(READ_BUFFER_SIZE)
msg = self._socket.recv(READ_BUFFER_SIZE)
except ConnectionResetError as cr_err:
_msg = f"connection reset by peer ({}:{self.port})"
raise CommunicationError(_msg) from cr_err
if not msg:
# set socket to None, so another connect() will make a new one;
# do not close here since we are in another greenlet
......@@ -649,7 +678,7 @@ class RpcConnection:
for m in unpacker:
call_id = m[0]
call_id = m[0] # can raise TypeError
if call_id < 0: # event:
value, signal = m[1]
louie.send(signal, self._proxy, value)
......@@ -658,12 +687,19 @@ class RpcConnection:
wq = self._queues.get(call_id)
if wq:
except Exception as e:
exception = e
except TypeError as terr:
# Exception: "TypeError: 'int' object is not subscriptable"
# raised from line: " call_id = m[0] ",
# can mean that connection is made to a port dedicated to
# another comm than RPC (ex: connection on SSH port 22).
# Or corrupted server ?
exception = terr
except Exception as err:
exception = err
for _, wq in self._queues.items():
wq.put(RpcConnectionError(f"Disconnected from {self._address}"))
if (exception is None or not self._queues) and callable(
......@@ -682,11 +718,11 @@ class RpcConnection:
class Client(proxy.Proxy):
def __init__(self, address, timeout=3., disconnect_callback=None):
def __init__(self, address, timeout=3.0, disconnect_callback=None):
rpc_connection = RpcConnection(address, disconnect_callback, timeout)
object.__setattr__(self, "_rpc_connection", rpc_connection)
object.__setattr__(self, "_Client__class", None)
super().__init__(lambda: rpc_connection._proxy)
super().__init__(lambda: rpc_connection._proxy, init_once=True)
def __class__(self):
......@@ -122,11 +122,8 @@ class _Base:
def wait_ready(self):
return self.device.wait_ready()
def fill_meta_at_scan_init(self, scan_meta):
return self.device.fill_meta_at_scan_init(scan_meta)
def fill_meta_at_scan_end(self, scan_meta):
return self.device.fill_meta_at_scan_end(scan_meta)
def get_acquisition_metadata(self, *args, **kw):
return self.device.get_acquisition_metadata(*args, **kw)
def _all_point_rx(self):
......@@ -637,10 +637,8 @@ def lazy_init(func):
class Axis(Scannable):
Bliss motor axis
Typical usage goes through the bliss configuration (see this module
documentation above for an example)
This class is typically used by motor controllers in bliss to export
axis with harmonised interface for users and configuration.
......@@ -716,12 +714,12 @@ class Axis(Scannable):
def unit(self):
"""Axis name"""
"""unit used for the Axis (mm, deg, um...)"""
return self._unit
def name(self):
"""Axis name"""
"""name of the axis"""
return self.__name
......@@ -735,7 +733,10 @@ class Axis(Scannable):
def controller(self):
"""Reference to :class:`~bliss.controllers.motor.Controller`"""
Motor controller of the axis
Reference to :class:`~bliss.controllers.motor.Controller`
return self.__controller
......@@ -951,9 +952,9 @@ class Axis(Scannable):
def measured_position(self):
Return the encoder value in user units.
Return measured position (ie: usually the encoder value).
float: encoder value in user units
return self.dial2user(self.dial_measured_position)
......@@ -962,10 +963,10 @@ class Axis(Scannable):
def dial_measured_position(self):
Return the dial encoder position.
Dial encoder position.
float: dial encoder position
float: Dial encoder position
if self.encoder is not None:
......@@ -1002,7 +1003,7 @@ class Axis(Scannable):
Return current dial position, or set dial
float: current dial position (dimensionless)
dial_pos = self.settings.get("dial_position")
......@@ -1050,10 +1051,21 @@ class Axis(Scannable):
def position(self):
Return current user position, or set new user position
Return current user position, or set new user position in user units.
float: current user position (user units)
new_pos : float
New position to set, in user units.
This update offset.
pos = self.settings.get("position")
if pos is None:
......@@ -1064,6 +1076,7 @@ class Axis(Scannable):
def position(self, new_pos):
""" see property getter """
log_debug(self, " : position(new_pos=%r)" % new_pos)
if self.is_moving:
raise RuntimeError(
......@@ -1110,9 +1123,6 @@ class Axis(Scannable):
Return the axis state
Keyword Args:
read_hw (bool): read from hardware [default: False]
AxisState: axis state
......@@ -1128,12 +1138,7 @@ class Axis(Scannable):
def hw_state(self):
Return the current hardware axis state
AxisState: axis state
""" Return the current hardware axis state (:obj:`AxisState`) """
return self.__controller.state(self)
......@@ -1253,10 +1258,12 @@ class Axis(Scannable):
def velocity(self):
Return the current velocity
Return or set the current velocity.
float: new_velocity in user unit/second
float: current velocity (user units/second)
float: current velocity in user unit/second
# Read -> Return velocity read from motor axis.
_user_vel = self.settings.get("velocity")
......@@ -1268,7 +1275,7 @@ class Axis(Scannable):
def velocity(self, new_velocity):
# Write -> Converts into motor units to change velocity of axis."
# Write -> Converts into motor units to change velocity of axis.
new_velocity = float(
) # accepts both float or numpy array of 1 element
......@@ -1296,6 +1303,9 @@ class Axis(Scannable):
f"Controller velocity ({_user_vel}) is different from set velocity ({new_velocity})",
curr_vel = self.settings.get("velocity")
if curr_vel != _user_vel:
user_print(f"'{}` velocity changed from {curr_vel} to {_user_vel}")
self.settings.set("velocity", _user_vel)
return _user_vel
......@@ -1465,7 +1475,7 @@ class Axis(Scannable):
Return the config jog velocity.
float: config jog velocity (user units/second)
float: config jog velocity (user_units/second)
return self.__config_jog_velocity
......@@ -1473,7 +1483,13 @@ class Axis(Scannable):
def acceleration(self, new_acc=None, from_config=False):
<new_acc> is given in user_units/s2.
new_acc: float
new acceleration that has to be provided in user_units/s2.
acceleration: float