Commit 2f471a26 authored by Mauro Rovezzi's avatar Mauro Rovezzi

Merge remote-tracking branch 'upstream/master'

parents 31520419 bc68b818
Pipeline #14053 failed with stages
......@@ -3,6 +3,7 @@
*.pyc
*.pyo
*~
*.py,cover
*.bck
build
*.swp
......@@ -26,3 +27,4 @@ htmlcov/
*.so
bliss/release.py
prof
.mypy_cache/
......@@ -2,6 +2,7 @@ before_script:
# set pip cache to the Docker volume
- echo ${CI_PROJECT_DIR}
- export PIP_CACHE_DIR="/opt/cache/pip"
- /opt/conda/bin/conda init && source /root/.bashrc
- conda config --append channels conda-forge
- conda config --add channels defaults
- conda config --add channels http://bcu-ci.esrf.fr/stable
......@@ -44,7 +45,7 @@ run_tests:source:
only:
changes: # skip tests for doc changes
- "bin/**/*"
- "bliss/**/*"
- "**/*.{py}"
- "conda_recipe/**/*"
- "extensions/**/*"
- "scripts/**/*"
......@@ -59,9 +60,12 @@ package:
stage: build
image: continuumio/miniconda3:latest
script:
# create package env and install all requirements and conda-build
# create package env and install all requirements and conda-build, (gcc and g++ compiler required for flint)
- conda create --quiet --name buildenv --file requirements-conda.txt --file requirements-test-conda.txt conda-build
- source activate buildenv
# create links to reach prefixed compilers of conda
- ln -s /opt/conda/envs/buildenv/bin/x86_64-conda_cos6-linux-gnu-gcc /opt/conda/envs/buildenv/bin/gcc
- ln -s /opt/conda/envs/buildenv/bin/x86_64-conda_cos6-linux-gnu-g++ /opt/conda/envs/buildenv/bin/g++
# triggering the creation of bliss/release.py file
- python -c "from setup import generate_release_file;generate_release_file()"
# creating the meta.yaml file for conda packet generation
......@@ -131,6 +135,8 @@ run_tests:package:
pages:
stage: deploy
before_script:
- ''
tags:
- conda
- builder
......@@ -156,6 +162,8 @@ pages:
deploy_bliss:
stage: deploy
before_script:
- ''
tags:
- conda
- builder
......
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.2.0] - 2019-09-23
### Added
- Bliss Shell Interface
- New Bliss interface for visualize Scan results on a different window (pressing F5).
It uses Tmux under the hood allowing a bunch of new capabilities like remote assistance (replacing the need of `screen`)
Launching Bliss will use Tmux as a default, use the `--no-tmux` option to start a session without tmux
- New UserDialogs available to interact with users (ask questions and display messages)
- Bliss Shell
- Exception Management
- more friendly exception management hiding exception details under `last_error` global variable (#402)
- Introduced global variable ERROR_REPORT.expert_mode to allow full traceback for expert users
- added `history` tooltip for viewing last commands (globals)
- add shell autocomplexion for dynamic attributes
- **info(obj)** function will standardize the representation of an object inside Bliss shell,
it uses underline `__info__` method if it exists and eventually falling back to `repr`
- New controllers support added:
- Aerotech Soloist
- Elmo whistle
- Lakeshore 331/332/335/336/340
- MultiplePositions
- Mythen/Mythen2
- Counters
- Soft Timer provides also `epoch` time (before was only delta)
- Added simulation_counter for user testing purposes
- Calculation counter can now be defined over other counters
- Tango Attribute counter can now use `unit` and `display unit` from Tango config
- Sampling counter modes statistics
- Now is possible to iterate Counters and Motors
- Sessions
- A Bliss Session will connect to an existing one if they share the same name and host.
In fact the same session will have only one instance at a time. This assumes underline use of Tmux
- Aliases
- shortcut for bliss objects (Axis, Counters, Channels) at a session level
- used in hdf5
- global ALIASES
- Global Map
- New Session map where controllers/connections/aliases/counters are registered when they are
loaded. Provides convenient runtime access to all instances and allows building on
top other services.
- Comes with visualization capabilities useful for debug or other purposes.
- Logging
- Based on Global Map it allows to log each individual instance of the same class of
controller/connection.
- Rich set of shell commands: lslog, log_debug, debugon, ...
- GitLab continuous integration
- test is skip if only docs are changed
- allow graphical tests
- new coverage report
- package building for tagged version (releases)
- Beacon Config
- Config
- order of element in file is now keeped while saving (instead of sorted)
- new BeaconObject that groups Static Configs and Settings, to be used when planning to
use both Config and Settings
- Scan
- New `pointscan` that performs scan on a list of positions
- New ScanPreset argument to perform operation on prepare/start/stop
- Alligment with multiple motors can be done propertly with (cen, com, peak, ...)
- New possibility to add further metadata when launching a Scan using scan_meta
- hdf5
- Values of underline Calculation Motors are exported
- Documentation about external data writing script
- SamplingModes refactor with new modes: STATS, SAMPLES, SINGLE, LAST, INTEGRATE_STATS
- flint new possibility to select any (X,Y) axis combination for plot
- IntegratingCounter: master_controller can now be None
### Changed
- controllers: py2to3 porting, improvements, refactor and bugfix
- Add commands for keithley 6514 and 2000
- Icepap
- Change of names
- API change in linked axis
- New `show` command to check enabled/disabled axis
- New check of trajectory number of points in regard to Icepap memory
- Euro2400
- Keithley 485
- Lakeshore
- Musst
- Md2
- pi_c663
- pi_e712
- pi_e727
- pi_e871
- pi_hexa
- Prologix
- Scpi
- Shexapod v1 and v2
- Web interface py2to3 port
- Parameters renamed in ParametersWardrobe with improved functionalities:
- allow export to YAML file and beacon
- allow property_attributes (read-only) and not-removable parameters
- add purge, remove, copy, freeze, show_table
- SamplingModes refactor with renaming of SIMPLE_AVERAGE to MEAN
### Fixed
- musst: python3 data reading, synchronization
- "Add WagoAirHook hook" (#772 #110)
- Static config: simplify reparent process (#495)
- fuelcell: "import of fuelcell (moved to id31 dir.) (#817)
- hdf5: "writer try to create an existing scan entry" (#620)
- shell:
- "Line numbering does not increase in bliss shell" (#610)
- "Exit shell with <CTRL-d> + return"
- "Deprecation warning coming from jinja2" (#688)
- ".counters namespace not accesible from command line" (#625)
- "kwargs in signature display in shell" (#798)
- "typinghelper check callable " (#746)
- gevent: "gevent timeout"
- gitlab CI:
- "do not copy htmlcov dir to public/"
- "added missing libxi6"
- gpib: "ibwrt python3 tango" (#574)
- scan:
- "scan preset" (#561)
- "Scan display: does not work when multiple points are received at the same time" (#743)
- "Scan listener missing output lines on fast scan" (#747)
- "real motors of nested calc axis not published" (#714)
- "scan.run should raise an exception if re-started" (#771)
- "empty .children() list of data note after NEW_CHILD" (#826)
- web interface: "web page interface" (#627)
- louie: Remove louie from StepScanDataWatchCallback (#645)
- rpc: "uds connection when the socket is removed, return AttributError"
- fix on node __internal_walk
- serial: "use url if port is not set"
- redis:
- "fullnames containing : and ." (#615)
- "exception in TTL setter" (#630)
- "really use alias name in session" (#700)
- flint/silx:
- "axes colors" (#717)
- "rulers are not updated" (#718)
- "error in Flint interaction" (#829)
- "Throwing an error in channel notification breaks object" (#719)
- "apply_config() fails if wrong settings have been set at first init" (#751)
- "double definition of close" (#587)
- "Load_scripts does not return an object or function" (#722)
- lima:
- "fix missing .fullname in Lima data node object"
- "fix of lima bpm simulator working again" (#664)
- "YAML 1.1 specification says ON, OFF, and other values should be converted to boolean" (#781)
- "BaseShutter `repr` bug" (#783)
- "motor is not initialised with MOVING state in axis settings"
- "comm.tcp.Command.connect : undefined variables" (#824)
- "prdef does not print correctly the inspected function" (#777)
- "Cannot set unit on tango_attr_as_counter" (#833)
- "Saving/Editing configuration implemented in a controller" (#835)
- "lima: fixed timescan" (#844)
- "manage limits always in DIAL. convert limits only on user interaction" (#854)
### Removed
## [0.1.3] - 2019-03-07
......@@ -18,7 +18,7 @@ import gevent
import json
from bliss.common.utils import OrderedDict
from collections import OrderedDict
try:
import sps
......
......@@ -7,11 +7,6 @@
"""Bliss main package
For your convenience, configuration motion and scan APIs have been made available
directly at this level.
Here are the main bliss sub-systems:
.. autosummary::
:toctree:
......@@ -25,7 +20,6 @@ Here are the main bliss sub-systems:
shell
tango
"""
from . import release
__version__ = release.version
......@@ -33,12 +27,46 @@ __author__ = release.author
__license__ = release.license
version_info = release.version_info
from gevent import monkey
from gevent import monkey as _monkey
_monkey.patch_all(thread=False)
from bliss.common.proxy import Proxy as _Proxy
def get_current_session():
from bliss.common import session
return session.get_current_session()
current_session = _Proxy(get_current_session)
from bliss.common.alias import MapWithAliases as _MapWithAliases
global_map = _MapWithAliases(current_session)
from bliss.common.logtools import Log as _Log
global_log = _Log(map=global_map)
monkey.patch_all(thread=False)
def logging_startup(
log_level="WARNING", fmt="%(levelname)s %(asctime)-15s %(name)s: %(message)s"
):
"""
Provides basicConfig functionality to bliss activating at proper level the root loggers
"""
import logging # this is not to pollute the global namespace
from redis import selector
# save log messages format
global_log.set_log_format(fmt)
global_log._LOG_DEFAULT_LEVEL = log_level # to restore level of non-BlissLoggers
selector._DEFAULT_SELECTOR = selector.SelectSelector
# setting startup level for session and bliss logger
logging.getLogger("session").setLevel(log_level)
logging.getLogger("bliss").setLevel(log_level)
from bliss.common import logtools
# install an additional handler, only for debug messages
# (debugon / debugoff)
global_log.set_debug_handler(logging.StreamHandler())
......@@ -6,10 +6,12 @@
# Distributed under the GNU LGPLv3. See LICENSE for more info.
from warnings import warn
import logging
from .embl import ExporterClient
from bliss.common.logtools import *
from bliss import global_map
import gevent
from gevent.queue import Queue
from .embl import ExporterClient
exporter_clients = {}
......@@ -57,6 +59,10 @@ class Exporter(ExporterClient.ExporterClient):
self.events_queue = Queue()
self.events_processing_task = None
global_map.register(
self, parents_list=["comms"], tag=f"exporter: {address}:{port}"
)
def start(self):
pass
# self.started=True
......@@ -139,8 +145,9 @@ class Exporter(ExporterClient.ExporterClient):
try:
cb(self._to_python_value(value))
except:
logging.exception(
"Exception while executing callback %s for event %s", cb, name
log_exception(
self,
f"Exception while executing callback {cb} for event {name}",
)
continue
......
......@@ -21,3 +21,5 @@ This module gathers different communication interfaces
tcp
util
"""
from bliss.comm.util import get_comm
This diff is collapsed.
......@@ -35,8 +35,8 @@
import socket, sys
from struct import *
from bliss.common import mapping
from bliss.common.logtools import LogMixin
from bliss.common.logtools import *
from bliss import global_map
# debug = ["io", "ignore_not_impl"] # "dummy_io", "rw"
debug = ["ignore_not_impl"]
......@@ -66,7 +66,7 @@ def _not_impl(name):
return _dbg(wrap, name)
class EnetSocket(LogMixin):
class EnetSocket:
def __init__(self, host, port=5000):
self._host = host
self._port = port
......@@ -75,12 +75,12 @@ class EnetSocket(LogMixin):
self.sta = self.err = self.cnt = 0
self.enet1000 = False
self._extra_socket = list()
mapping.register(self, parents_list=["comms"], tag=str(self))
global_map.register(self, parents_list=["comms"], tag=str(self))
def _open(self):
self._sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._sock.connect((self._host, self._port))
mapping.register(
global_map.register(
self, parents_list=["comms"], children_list=[self._sock], tag=str(self)
)
......
......@@ -7,6 +7,7 @@
import struct
import weakref
from functools import wraps
from gevent import socket, select
from gevent import lock
from gevent import queue
......@@ -18,8 +19,8 @@ from .exceptions import CommunicationError, CommunicationTimeout
from ..common.greenlet_utils import KillMask, protect_from_kill
from . import serial
from bliss.common import mapping
from bliss.common.logtools import LogMixin
from bliss.common.logtools import *
from bliss import global_map
class ModbusError(CommunicationError):
......@@ -89,12 +90,12 @@ class Modbus_ASCII:
return lrc & 0xff
class Modbus_RTU(LogMixin):
class Modbus_RTU:
def __init__(self, node, *args, **kwargs):
self._serial = serial.Serial(*args, **kwargs)
self.node = node
self._lock = lock.RLock()
mapping.register(self, children_list=[self._serial])
global_map.register(self, children_list=[self._serial])
def __del__(self):
self._serial.close()
......@@ -284,6 +285,7 @@ class Modbus_RTU(LogMixin):
def try_connect_modbustcp(fu):
@wraps(fu)
def rfunc(self, *args, **kwargs):
timeout = kwargs.get("timeout")
if not self._connected:
......
......@@ -71,40 +71,6 @@ import psutil
MAX_BUFFER_SIZE = int(psutil.virtual_memory().total * 0.8)
# msgpack patch for numpy and pickle
import pickle
# Patch msgpack
import msgpack_numpy
# Add fallback pickle packing
_msgpack_numpy_encode = msgpack_numpy.encode
def _pickle_fallback_encoding(obj, chain=None):
robj = _msgpack_numpy_encode(obj, chain=chain)
if robj is obj: # try to pickle
return {b"<pickled>": True, b"data": pickle.dumps(obj)}
else:
return robj
_msgpack_numpy_decode = msgpack_numpy.decode
def _pickle_fallback_decode(obj, chain=None):
if obj.get(b"<pickled>") is True:
return pickle.loads(obj[b"data"])
else:
return _msgpack_numpy_decode(obj, chain=chain)
# replace patched encode decode
msgpack_numpy.encode = _pickle_fallback_encoding
msgpack_numpy.decode = _pickle_fallback_decode
msgpack_numpy.patch()
# END patch
import os
import re
import inspect
......@@ -120,7 +86,14 @@ from gevent import socket
from bliss.common.greenlet_utils import KillMask
from bliss.common.utils import StripIt
import msgpack
from bliss.common.msgpack_ext import MsgpackContext
msgpack = MsgpackContext()
# Registration order matter
msgpack.register_numpy()
msgpack.register_tb_exception()
msgpack.register_pickle()
SPECIAL_METHODS = set(
......@@ -514,7 +487,13 @@ class _cnx(object):
self._socket.sendall(msg)
value = w.get()
if isinstance(value, Exception):
raise value
# FIXME: checking the traceback is an approximation
# It would be better to know it was a raised exception
# from the server msg
if value.__traceback__ is None:
return value
else:
raise value
elif isinstance(value, self.Retry):
self.try_connect()
continue
......
This diff is collapsed.
......@@ -13,14 +13,14 @@ import re
import struct
import weakref
import binascii
from functools import wraps
import gevent
from gevent import socket, select, lock, event
from ..common.greenlet_utils import KillMask
from bliss.common.cleanup import capture_exceptions
from bliss.common import mapping
from bliss.common.logtools import LogMixin
from bliss.common.logtools import *
from bliss import global_map
import serial
......@@ -77,6 +77,7 @@ class SerialTimeout(CommunicationTimeout):
def try_open(fu):
@wraps(fu)
def rfunc(self, *args, **kwarg):
try:
with KillMask():
......@@ -151,7 +152,7 @@ class _BaseSerial:
msg = self._data[:eol_pos]
self._data = self._data[eol_pos + len(eol) :]
self._cnt()._logger.debug_data("Rx:", msg)
log_debug_data(self._cnt(), "Rx:", msg)
return msg
def read(self, size, timeout):
......@@ -177,7 +178,7 @@ class _BaseSerial:
break
msg = self._data[:size]
self._data = self._data[size:]
self._cnt()._logger.debug_data("Rx:", msg)
log_debug_data(self._cnt(), "Rx:", msg)
return msg
def write(self, msg, timeout):
......@@ -185,7 +186,7 @@ class _BaseSerial:
return self._write(msg)
def _write(self, msg):
self._cnt()._logger.debug_data("Tx:", msg)
log_debug_data(self._cnt(), "Tx:", msg)
while msg:
_, ready, _ = select.select([], [self.fd], [])
size_send = os.write(self.fd, msg)
......@@ -206,7 +207,7 @@ class _BaseSerial:
msg = self._data
self._data = b""
self._cnt()._logger.debug_data("Rx:", msg)
log_debug_data(self._cnt(), "Rx:", msg)
return msg
@staticmethod
......@@ -709,7 +710,7 @@ class TangoSerial(_BaseSerial):
self._device.DevSerFlush(self.FLUSH_INPUT)
class Serial(LogMixin):
class Serial:
LOCAL, RFC2217, SER2NET, TANGO = list(range(4))
def __init__(
......@@ -746,7 +747,7 @@ class Serial(LogMixin):
self._timeout = timeout
self._raw_handler = None
self._lock = lock.RLock()
mapping.register(self, parents_list=["comms"], tag=str(self))
global_map.register(self, parents_list=["comms"], tag=str(self))
def __del__(self):
self.close()
......@@ -761,8 +762,8 @@ class Serial(LogMixin):
def open(self):
if self._raw_handler is None:
serial_type = self._check_type()
self._logger.debug("open - serial_type=%s" % serial_type)