Skip to content

ewoksify dimension

payno requested to merge refactor_dimension into main

details

This PR intend to ewoksify the DimensionWidgetOW widget.

note to reviewer(s): as this is a pretty heavy refactoring I tried to keep commit 'grouped'. So the better to review is probably to do commit by commit review.

current behavior

Today users interaction are:

  • provide a tolerance
  • find dimensions -> using the tolerance to get the different values. It calls Dataset.find_dimension and define self.dataset.dims + update GUI
  • fit dimensions -> dimensions must already be found. If dims is already computed then define dataset dimension (add_dim) and update the GUI.
  • validate -> take the currently processed dataset and move to the next widget.

issues

issue with ewoks orange ?

processing propagation also met some troubles with widget like the z-sum:

image

It is related to workflow/ewoks/ewoksorange!169 So it must be merge first.

issue with settings (1).

When we want to save the following workflow:

image

We can end up with the following error:

 HDF5_SWMR_VERSION = calc_hexversion(*h5py.get_config().swmr_min_hdf5_version[:3])
QCssParser::parseColorValue: Specified color without alpha value but alpha given: 'rgb 0, 0, 0, 100'
----------------------------- TypeError Exception -----------------------------
Traceback (most recent call last):
  File "/home/payno/dev/esrf/darkfield/venv/lib/python3.11/site-packages/orangecanvas/application/canvasmain.py", line 1617, in save_swp
    self.save_swp_to(swpname)
  File "/home/payno/dev/esrf/darkfield/venv/lib/python3.11/site-packages/orangecanvas/application/canvasmain.py", line 1626, in save_swp_to
    propertiesDiff = document.uncleanProperties()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/payno/dev/esrf/darkfield/venv/lib/python3.11/site-packages/orangecanvas/document/schemeedit.py", line 698, in uncleanProperties
    return list(dictdiffer.diff(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/payno/dev/esrf/darkfield/venv/lib/python3.11/site-packages/dictdiffer/__init__.py", line 215, in _diff_recursive
    for diffed in recurred:
  File "/home/payno/dev/esrf/darkfield/venv/lib/python3.11/site-packages/dictdiffer/__init__.py", line 253, in _diff_recursive
    yield ADD, dotted_node, [
                            ^
  File "/home/payno/dev/esrf/darkfield/venv/lib/python3.11/site-packages/dictdiffer/__init__.py", line 256, in <listcomp>
    (key, deepcopy(_second[key])) for key in addition]
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle 'DataSelectionWidgetOW' object
-------------------------------------------------------------------------------
QLayout: Attempting to add QLayout "" to ErrorReporting "", which already has a layout
QLayout: Attempting to add QLayout "" to ErrorReporting "", which already has a layout
2024-05-30 11:01:32,663:ERROR:orangecanvas.scheme.readwrite: Error serializing properties for node 'dimension definition'
Traceback (most recent call last):
  File "/home/payno/dev/esrf/darkfield/venv/lib/python3.11/site-packages/orangecanvas/scheme/readwrite.py", line 662, in scheme_to_etree
    data, format = dumps(node.properties, format=data_format,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/payno/dev/esrf/darkfield/venv/lib/python3.11/site-packages/orangecanvas/scheme/readwrite.py", line 792, in dumps
    return base64.encodebytes(pickle.dumps(obj, protocol=PICKLE_PROTOCOL)). \
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot pickle 'DataSelectionWidgetOW' object

The error come from having the 'DataSelectionWidgetOW' providing a OWDataset that has itself ('DataSelectionWidgetOW') as parent. So it will be fix with the 'DataSelectionWidgetOW' refactoring.

warning: move to ewoks force us to define the "dataset" as an input. A consequence is that .ows will move from having some Settings 'dump as literal' in the .ows like

<node_properties>
   <properties node_id="0" format="literal">{'tolerance': 1e-8 'metadata_type': 1,>

to 'pickle' format like

<properties node_id="2" format="pickle">gASVMQEAAAAAAAB9lCiMFV9ld29rc19kZWZhdWx0X2lucHV0c5R9lCiMCXRvbGVyYW5jZZR

(as the Dataset class cannot be dump as litteral. This also bring an issue with the .ows file size as we can end up having the full file saved there.

We need to add an option to avoid saving some input in the .ows file

issue with settings

'dataset' input is saved: it can be huge and note dump as literal. See workflow/ewoks/ewoksorange#40 (closed)

issue with settings (3) and CLI execution

For processing from CLI (ewoks-execute) we must take this into account. Only implementing the current fit will work. As it compute dims if they don't exist. But today the dims will be stored with the workflow. Which can be an issue when processing another worflow. We don't expect the dimensions to be constant from one dataset to another...

This PR will not modify the current behavior. #129 has been created about this issue.

Edited by payno

Merge request reports