Access data from memory for slow Lima scans
Online data processing of a slow scan can be problematic if you cannot wait for the lima files to be completed. For example because you need to see the results during the scan.
That's why we currently save Lima data 1 image per file at the moment.
A better solution is to read data from a memory, but also be able to process from file as well (in case of re-processing).
This is an example of a task that needs:
- scan metadata (e.g. static motor positions)
- scan counters (e.g. diodes)
- lima images
from typing import Optional
from ewokscore import Task
from blissdata.h5api import dynamic_hdf5
from ewoksdata.data.bliss import iter_bliss_scan_data
from ewoksdata.data.blissdata import iter_bliss_scan_data_from_memory
class SlowLimaTask(
Task,
input_names=[
"filename",
"scan_number",
"lima_name",
"scalar_counters",
"motor_names",
],
optional_input_names=[
"scan_key",
"retry_timeout",
"retry_period",
],
):
def run(self):
retry_options = {
"retry_timeout": self.get_input_value(
"retry_timeout", None
),
"retry_period": self.get_input_value(
"retry_period", None
),
}
# Get static motor positioner are any other metadata.
with dynamic_hdf5.File(
self.inputs.filename, **retry_options
) as h5:
start_positioners_path = f"/{self.inputs.scan_number}.1/instrument/positioners_start"
start_positioners = h5[start_positioners_path]
start_positioners = {
motor: start_positioners[motor][()]
for motor in self.inputs.motor_names
}
print(start_positioners)
# Iterate over the data until finished.
for scan_point_data_dict in _iter_data(
self.inputs.filename,
self.inputs.scan_number,
self.get_input_value("scan_key", None),
lima_names=[self.inputs.lima_name],
counter_names=self.inputs.scalar_counters,
**retry_options,
):
print(scan_point_data_dict)
def _iter_data(
filename: str, scan_nr: int, scan_key: Optional[str], **kwargs
):
# Helper to iterate over memory or HDF5 file.
if scan_key:
print("Iterate from memory")
yield from iter_bliss_scan_data_from_memory(
scan_key, **kwargs
)
else:
print("Iterate from file")
yield from iter_bliss_scan_data(filename, scan_nr, **kwargs)Example for the Bliss demo session
{
"nodes": [
{
"task_type": "class",
"task_identifier": "results.slow_lima_task.SlowLimaTask",
"default_inputs": [
{
"name": "filename",
"value": "RAW_DATA/sample/sample_0001/sample_0001.h5"
},
{
"name": "scan_number",
"value": 1
},
{
"name": "lima_name",
"value": "difflab6"
},
{
"name": "motor_names",
"value": ["robx"]
},
{
"name": "scalar_counters",
"value": ["diode1"]
}
],
"id": "read_data"
}
]
}Process from memory
export BEACON_HOST="localhost:10001"
ewoks execute results/workflow4.json --outputs=all --log info -p retry_timeout=5 \
-p filename="${rootdir}/inhouse/id002506/id00/20250601/RAW_DATA/sample/sample_0001/sample_0001.h5" \
-p scan_number=1 \
-p scan_key='esrf:scan:01JY21ZYHASBX7VVPPQC8M3X6X'Process from file
ewoks execute results/workflow4.json --outputs=all --log info -p retry_timeout=5 \
-p filename="${rootdir}/inhouse/id002506/id00/20250601/RAW_DATA/sample/sample_0001/sample_0001.h5" \
-p scan_number=1 \
-p scan_key=nullThe output looks like this
###################################
# Execute workflow 'results/workflow4.json'
###################################
2025-06-18 19:43:54,866 - ewokscore.events.global_state - INFO - [job 'f88c4229-b14e-4cfd-8bb1-543ec1f2dfa3'] job started
2025-06-18 19:43:54,866 - ewokscore.graph.serialize - WARNING - Graph has no "id": use "notspecified"
2025-06-18 19:43:54,867 - ewokscore.graph.schema - INFO - Graph has no "schema_version": assume version "1.1"
2025-06-18 19:43:54,868 - ewokscore.events.global_state - INFO - [job 'f88c4229-b14e-4cfd-8bb1-543ec1f2dfa3'] [workflow 'notspecified'] workflow started
2025-06-18 19:43:55,221 - ewokscore.events.global_state - INFO - [job 'f88c4229-b14e-4cfd-8bb1-543ec1f2dfa3'] [workflow 'notspecified'] [node 'read_data'] [task 'results.slow_lima_task.SlowLimaTask'] task started
{'robx': 0.0}
Iterate from memory
{'difflab6': array([[ 5, 6, 6, ..., 0, 0, 0],
[ 7, 5, 6, ..., 0, 0, 0],
[ 7, 5, 5, ..., 0, 0, 0],
...,
[ 0, 0, 0, ..., 5, 5, 9],
[ 0, 0, 0, ..., 6, 8, 11],
[ 1, 0, 0, ..., 5, 10, 15]], dtype=uint32), 'diode1': 49.86666666666666}
{'difflab6': array([[ 6, 9, 5, ..., 1, 0, 0],
[ 8, 4, 6, ..., 0, 0, 0],
[ 4, 5, 3, ..., 0, 0, 0],
...,
[ 0, 0, 0, ..., 2, 5, 11],
[ 0, 0, 0, ..., 8, 8, 11],
[ 0, 0, 0, ..., 9, 11, 12]], dtype=uint32), 'diode1': 49.31999999999999}
{'difflab6': array([[ 6, 6, 6, ..., 0, 0, 0],
[ 5, 5, 4, ..., 0, 0, 0],
[ 6, 4, 6, ..., 0, 0, 0],
...,
[ 0, 0, 0, ..., 4, 6, 10],
[ 0, 0, 0, ..., 5, 10, 11],
[ 0, 0, 0, ..., 7, 11, 15]], dtype=uint32), 'diode1': 52.14333333333333}
{'difflab6': array([[ 8, 6, 6, ..., 0, 1, 0],
[ 8, 6, 5, ..., 0, 0, 0],
[ 6, 4, 3, ..., 0, 0, 0],
...,
[ 0, 0, 0, ..., 4, 4, 10],
[ 0, 0, 0, ..., 3, 11, 13],
[ 1, 0, 0, ..., 7, 9, 11]], dtype=uint32), 'diode1': 52.55444444444444}
{'difflab6': array([[ 7, 6, 5, ..., 0, 0, 1],
[ 6, 6, 4, ..., 0, 1, 0],
[ 4, 6, 5, ..., 1, 0, 1],
...,
[ 0, 0, 0, ..., 4, 7, 10],
[ 0, 0, 0, ..., 6, 7, 12],
[ 0, 0, 0, ..., 5, 11, 13]], dtype=uint32), 'diode1': 53.849999999999994}
{'difflab6': array([[6, 5, 4, ..., 0, 0, 0],
[3, 5, 3, ..., 0, 0, 0],
[4, 4, 3, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 3, 5, 6],
[0, 0, 0, ..., 3, 6, 9],
[0, 0, 0, ..., 4, 6, 8]], dtype=uint32), 'diode1': 46.34777777777778}
{'difflab6': array([[ 4, 5, 5, ..., 0, 0, 0],
[ 5, 4, 5, ..., 0, 0, 0],
[ 3, 4, 4, ..., 0, 0, 0],
...,
[ 0, 0, 1, ..., 4, 5, 7],
[ 0, 0, 0, ..., 4, 6, 7],
[ 0, 0, 0, ..., 6, 7, 12]], dtype=uint32), 'diode1': 52.43444444444444}
{'difflab6': array([[ 4, 5, 6, ..., 0, 0, 0],
[ 6, 4, 4, ..., 0, 0, 0],
[ 6, 5, 4, ..., 1, 1, 0],
...,
[ 0, 0, 0, ..., 3, 4, 7],
[ 0, 0, 0, ..., 3, 5, 10],
[ 0, 0, 0, ..., 5, 7, 10]], dtype=uint32), 'diode1': 53.495}
{'difflab6': array([[ 4, 5, 4, ..., 0, 0, 0],
[ 6, 4, 4, ..., 0, 0, 0],
[ 4, 5, 2, ..., 0, 0, 0],
...,
[ 0, 0, 0, ..., 2, 5, 7],
[ 0, 0, 0, ..., 3, 5, 9],
[ 0, 0, 0, ..., 6, 8, 12]], dtype=uint32), 'diode1': 47.04875}
{'difflab6': array([[ 5, 5, 5, ..., 0, 0, 0],
[ 5, 4, 4, ..., 0, 0, 0],
[ 3, 4, 4, ..., 1, 0, 0],
...,
[ 0, 0, 0, ..., 3, 5, 7],
[ 0, 0, 0, ..., 3, 6, 10],
[ 0, 0, 0, ..., 6, 7, 10]], dtype=uint32), 'diode1': 44.78333333333333}
2025-06-18 19:43:55,539 - ewokscore.events.global_state - INFO - [job 'f88c4229-b14e-4cfd-8bb1-543ec1f2dfa3'] [workflow 'notspecified'] [node 'read_data'] [task 'results.slow_lima_task.SlowLimaTask'] task finished
2025-06-18 19:43:55,539 - ewokscore.events.global_state - INFO - [job 'f88c4229-b14e-4cfd-8bb1-543ec1f2dfa3'] [workflow 'notspecified'] workflow finished
2025-06-18 19:43:55,540 - ewokscore.events.global_state - INFO - [job 'f88c4229-b14e-4cfd-8bb1-543ec1f2dfa3'] job finishedEdited by Wout De Nolf