est issueshttps://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues2024-03-08T16:41:34+01:00https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/84Support bi-directional multi XAS scans2024-03-08T16:41:34+01:00Wout De NolfSupport bi-directional multi XAS scansLike fullfield but then flat and the energy direction is flipped in each consecutive run.
https://jira.esrf.fr/browse/DPDEV-207
[BM23_exafs_scans.h5](/uploads/d13a13e365ce959719c3415504e4e2d6/BM23_exafs_scans.h5)Like fullfield but then flat and the energy direction is flipped in each consecutive run.
https://jira.esrf.fr/browse/DPDEV-207
[BM23_exafs_scans.h5](/uploads/d13a13e365ce959719c3415504e4e2d6/BM23_exafs_scans.h5)https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/83Input widget: selecting the monitor does not affect the plot or table2024-02-11T08:26:56+01:00Wout De NolfInput widget: selecting the monitor does not affect the plot or table![image](/uploads/44521d4c7a39b2f1a622ad31ddc9af58/image.png)![image](/uploads/44521d4c7a39b2f1a622ad31ddc9af58/image.png)https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/82Input task needs to be simplified: all formats give the same data2024-02-08T02:02:25+01:00Wout De NolfInput task needs to be simplified: all formats give the same data* All data consists of several channels with random names.
* Every channel inside each file will have a URL.
* Every channel has either the shape `(Nchan,)` or the shape `(..., Nchan)`
HDF5, DAT, CSV, XMU: it is all the same, except for...* All data consists of several channels with random names.
* Every channel inside each file will have a URL.
* Every channel has either the shape `(Nchan,)` or the shape `(..., Nchan)`
HDF5, DAT, CSV, XMU: it is all the same, except for the URLs and the data reading.
Essentially you have energy, mu, i0, i1, i2, ... and each has `shape[-1]==Nchan`.
Note that the energy could have shape `(Nfiles, Nchan)` for .dat files but `est` only allows selecting 1 .dat file.
[exafs_data.zip](/uploads/b02eee33594a13c2fd6e756c979b5c66/exafs_data.zip)
```
ewoks-canvas ~/exafs_example/exafs.ows
ewoks execute ~/exafs_example/exafs.ows --outputs=end -p input_information='{"spectra_url": "silx:///home/denolf/exafs_example/cu_rt01.h5?/1.1/measurement/mu", "channel_url": "silx:///home/denolf/exafs_example/cu_rt01.h5?/1.1/measurement/energy", "dimensions": [2, 1, 0], "energy_unit": "electron_volt"}'
ewoks execute ~/exafs_example/exafs.ows --outputs=end -p input_information='{"spectra_url": "spec:///home/denolf/exafs_example/EXAFS_Cu.dat?1 cu.dat 1.1 Column 2/Column 2", "channel_url": "spec:///home/denolf/exafs_example/EXAFS_Cu.dat?1 cu.dat 1.1 Column 2/Column 1", "energy_unit": "electron_volt"}'
ewoks execute ~/exafs_example/exafs.ows --outputs=end -p input_information='{"spectra_url": "larch:///home/denolf/exafs_example/cu_rt01.xmu", "channel_url": "larch:///home/denolf/exafs_example/cu_rt01.xmu", "energy_unit": "electron_volt"}'
ewoks execute ~/exafs_example/exafs.ows --outputs=end -p input_information='{"spectra_url": "spec:///home/denolf/exafs_example/EXAFS_Ge.csv?mu", "channel_url": "spec:///home/denolf/exafs_example/EXAFS_Ge.dat?energy", "energy_unit": "electron_volt"}'
```https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/80Add tests for the example workflows2024-02-07T19:38:48+01:00Wout De NolfAdd tests for the example workflowsSimilar to https://gitlab.esrf.fr/workflow/ewoksapps/ewoksfluo/-/blob/main/src/ewoksfluo/tests/tutorials/test_single_scan_single_detector.py?ref_type=headsSimilar to https://gitlab.esrf.fr/workflow/ewoksapps/ewoksfluo/-/blob/main/src/ewoksfluo/tests/tutorials/test_single_scan_single_detector.py?ref_type=headshttps://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/77Handle multiple EXAFS scans2023-10-02T19:12:26+02:00Wout De NolfHandle multiple EXAFS scansAt ID24 they have two types of multi-EXAFS scans:
1. Repeats: repeat a single scan N times
2. Concatenate: concatenate N energy sweeps in a single scan
est already support multi-scans so probably we need a task that creates a reformed ...At ID24 they have two types of multi-EXAFS scans:
1. Repeats: repeat a single scan N times
2. Concatenate: concatenate N energy sweeps in a single scan
est already support multi-scans so probably we need a task that creates a reformed scan file using VDS.https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/75larch tests fail2022-10-14T15:25:10+02:00Wout De Nolflarch tests failhttps://gitlab.esrf.fr/workflow/ewoksapps/est/-/jobs/462588
```
spectrum_cu_from_larch = <est.core.types.spectrum.Spectrum object at 0x7f315be26510>
@pytest.mark.skipif(larch is None, reason="xraylarch not installed")
def test_s...https://gitlab.esrf.fr/workflow/ewoksapps/est/-/jobs/462588
```
spectrum_cu_from_larch = <est.core.types.spectrum.Spectrum object at 0x7f315be26510>
@pytest.mark.skipif(larch is None, reason="xraylarch not installed")
def test_single_spectrum(spectrum_cu_from_larch):
"""Make sure computation on one spectrum is valid"""
configuration = {
"window": "hanning",
"kweight": 2,
"kmin": 3,
"kmax": 13,
"dk": 1,
}
# for xftf we need to compute pre edge before
> process_spectr_autobk(spectrum_cu_from_larch, configuration={}, overwrite=True)
src/est/tests/larch/test_xftf.py:24:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/est/core/process/larch/autobk.py:83: in process_spectr_autobk
**opts
/usr/local/lib/python3.7/site-packages/larch/larchlib.py:532: in wrapper
result = fcn(*args, **kwargs)
/usr/local/lib/python3.7/site-packages/larch/xafs/autobk.py:119: in autobk
pre_edge(energy, mu, group=group, _larch=_larch, **pre_kws)
/usr/local/lib/python3.7/site-packages/larch/larchlib.py:532: in wrapper
result = fcn(*args, **kwargs)
/usr/local/lib/python3.7/site-packages/larch/xafs/pre_edge.py:334: in pre_edge
_atsym, _edge = guess_edge(group.e0)
/usr/local/lib/python3.7/site-packages/xraydb/xray.py:665: in guess_edge
xdb = get_xraydb()
/usr/local/lib/python3.7/site-packages/xraydb/xray.py:156: in get_xraydb
_xraydb = XrayDB()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <xraydb.xraydb.XrayDB object at 0x7f315be1cc50>
dbname = '/usr/local/lib/python3.7/site-packages/xraydb/xraydb.sqlite'
read_only = True
def __init__(self, dbname='xraydb.sqlite', read_only=True):
"connect to an existing database"
if not os.path.exists(dbname):
parent, _ = os.path.split(__file__)
dbname = os.path.join(parent, dbname)
if not os.path.exists(dbname):
raise IOError("Database '%s' not found!" % dbname)
if not isxrayDB(dbname):
> raise ValueError("'%s' is not a valid X-ray Database file!" % dbname)
E ValueError: '/usr/local/lib/python3.7/site-packages/xraydb/xraydb.sqlite' is not a valid X-ray Database file!
/usr/local/lib/python3.7/site-packages/xraydb/xraydb.py:84: ValueError
```https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/63BM23 pyqt-conda versions2022-07-04T17:12:03+02:00Wout De NolfBM23 pyqt-conda versions```
anyqt 0.1.1 pyh6c4a22f_0 conda-forge
pyqt 5.12.3 py37h89c1867_8 conda-forge
pyqt-impl 5.12.3 py37hac37412_8 conda-forge
pyqt5-qt5 ...```
anyqt 0.1.1 pyh6c4a22f_0 conda-forge
pyqt 5.12.3 py37h89c1867_8 conda-forge
pyqt-impl 5.12.3 py37hac37412_8 conda-forge
pyqt5-qt5 5.15.2 pypi_0 pypi
pyqt5-sip 12.11.0 pypi_0 pypi
pyqtchart 5.12 py37he336c9b_8 conda-forge
pyqtgraph 0.12.4 pyhd8ed1ab_0 conda-forge
pyqtwebengine 5.12.1 py37he336c9b_8 conda-forge
qt 5.12.9 hda022c4_4 conda-forge
qtconsole 5.3.1 pyhd8ed1ab_0 conda-forge
qtconsole-base 5.3.1 pyha770c72_0 conda-forge
qtpy 2.1.0 pyhd8ed1ab_0 conda-forge
```https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/62Why does the Spectrum class exist?2022-06-28T18:53:41+02:00Wout De NolfWhy does the Spectrum class exist?After https://gitlab.esrf.fr/workflow/ewoksapps/est/-/merge_requests/77 it seems `Spectrum` is basically a dictionary. So it should in fact *be* a dictionary. Why not?After https://gitlab.esrf.fr/workflow/ewoksapps/est/-/merge_requests/77 it seems `Spectrum` is basically a dictionary. So it should in fact *be* a dictionary. Why not?https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/60xftf widget2022-06-29T08:46:46+02:00paynoxftf widgetWhen phase is request from the interface I guess we would like to plot the result of it. This is not done at the moment.
![image](/uploads/94233a4be407934b10108fd6378c7277/image.png)When phase is request from the interface I guess we would like to plot the result of it. This is not done at the moment.
![image](/uploads/94233a4be407934b10108fd6378c7277/image.png)https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/51Task+widget that returns test data2022-06-19T15:56:09+02:00Wout De NolfTask+widget that returns test dataImplement the following task (+widget)
```
class GenerateExampleData(Task, optional_input_names=["spectrum", "shape", "dimensions"], output_names=["xas_obj"]):
def run(self):
...
```
Use `est.tests.data.example_spectrum` an...Implement the following task (+widget)
```
class GenerateExampleData(Task, optional_input_names=["spectrum", "shape", "dimensions"], output_names=["xas_obj"]):
def run(self):
...
```
Use `est.tests.data.example_spectrum` and `est.tests.data.example_spectra` to generate the data in memory.
Note that `spectrum` can be
* a string, `"exafs/EXAFS_Ge.dat"`
* a tuple, `(5000, 5500, 300)`
* or `None`
`dimensions` is a tuple, `(1,2,0)`. Best to wait until https://gitlab.esrf.fr/workflow/ewoksapps/est/-/merge_requests/57 is merged.https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/41integrate / integrate nxtoh52021-02-24T10:18:51+01:00paynointegrate / integrate nxtoh5https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/40personnalized plot2022-06-20T08:56:05+02:00paynopersonnalized plotIt would be interesting to allow the user to define what he wants to display. Like letting him define x and y. Even from a formula. Can be complicated but would be create and bring a lot of flexibility.It would be interesting to allow the user to define what he wants to display. Like letting him define x and y. Even from a formula. Can be complicated but would be create and bring a lot of flexibility.https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/39better maangement for "_NexusDatasetDef"2022-06-20T09:00:17+02:00paynobetter maangement for "_NexusDatasetDef"the `_NexusDatasetDef` class is used to define information to be store.
For now we redefine those each time with dimensions...
It would be safer to create a database with all those defined.the `_NexusDatasetDef` class is used to define information to be store.
For now we redefine those each time with dimensions...
It would be safer to create a database with all those defined.https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/38Output. saving results2021-10-05T11:11:39+02:00paynoOutput. saving resultsMaybe we could make the distinction when saving results if we are only saving results of one spectrum or several.
It would simplify the output.
But has we are "duplicating" data contained in the .h5 used also for the processing it might ...Maybe we could make the distinction when saving results if we are only saving results of one spectrum or several.
It would simplify the output.
But has we are "duplicating" data contained in the .h5 used also for the processing it might be cumbersome.
Another good point could be to invert the axis for processing. Spectrum are saved in Z when they should be in X.0.5https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/33roi curve - pymca2020-03-16T13:13:39+01:00paynoroi curve - pymcaWe should be able to use ROI curve to select edge, energy min and max...
As est is using the pymca widgets it is probably more interesting to add this in the pymca roi toll.We should be able to use ROI curve to select edge, energy min and max...
As est is using the pymca widgets it is probably more interesting to add this in the pymca roi toll.https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/25what to come & optimization ?2021-10-05T11:11:55+02:00paynowhat to come & optimization ?The next release should give an idea about the processing and the 'target'. But there is still a lot of limitation on several points that should be discussed.
# internal design - performances
There is several limitation regarding the i...The next release should give an idea about the processing and the 'target'. But there is still a lot of limitation on several points that should be discussed.
# internal design - performances
There is several limitation regarding the internal architecture of Spectrum and Spectra. See #24
# process file
an issue regarding it has been created: #21
# beta test and communication
Is it enough to create a pre-release / release and get some beta tester
# orange add-on
Initially (when processing was made on one spectrum) each 'orange widget' was making a copy
of the XASObject to insure the possibility to reprocess only within the upstream process (with a 'freeze' state).
But with the latest test using thousands of spectrum this approach seems to memory consuming.
So this copy was removed. Widget now have 'an update manually' button to avoid relaunching each computation as soon a parameter change.
If we continue on this way (no copy of the XASObject) we should insure:
- explain clearly the limitation and the reason of it
- show clearly the user that no reprocessing is possible (because the XASObject has changed since it has been received by the widget)
- the creation of a widget to create a copy in the flow (saving point)
# memory usage
When try to process the entire dataset (B33_xxxx) fail because of a too large memory consumption.
Some numbers:
- for a spectra (dim1: 400, dim2: 200, energy: 211):
| processing | memory used |
| ------------------- | ----------- |
| after loading | 0.421 GB |
| after normalization | 1.493 GB |
| after exafs | 3.898 GB |
| after k weight | 4.067 GB |
| after ft | 9.145 GB |
| after saving | 9.141 GB |
# some spectrum are not processed (especially by ft)
make sure this is expected.
should we prevent / avoid to continue the processing on those spectrum ? For me no because they might be some 'unrelated' treatments and Spectrum can fail with some processing and not on others.0.5https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/22XAS_Object & XASObjectViewer optimizations2019-12-10T14:56:02+01:00paynoXAS_Object & XASObjectViewer optimizations- they should be able to manage offsets if any
- spectrum display could be associated to a map to allow to pick a specific pixel / roi and to display the associated spectrum of mean spectrum...- they should be able to manage offsets if any
- spectrum display could be associated to a map to allow to pick a specific pixel / roi and to display the associated spectrum of mean spectrum...https://gitlab.esrf.fr/workflow/ewoksapps/est/-/issues/21Output > spectra representation2021-10-05T11:12:00+02:00paynoOutput > spectra representationFor now we are saving data as represented:
![Screenshot_from_2019-12-16_13-09-32](/uploads/01ff4dbc8a08f6420794af42860ba8f1/Screenshot_from_2019-12-16_13-09-32.png)
This way of storing data is also closely related to the `from_dict` an...For now we are saving data as represented:
![Screenshot_from_2019-12-16_13-09-32](/uploads/01ff4dbc8a08f6420794af42860ba8f1/Screenshot_from_2019-12-16_13-09-32.png)
This way of storing data is also closely related to the `from_dict` and `to_dict` functions.
This present several limitation:
- for now we have to define after each process which values (keys) of the spectra we want to save.
Those keys has to be check because they might be some information missing. If we could find a process to configure this it might be usefull for the user.
- for ft (only at the moment) we save the result of ft as a dictionary instead of a numpy array because several ft process cannot be executed (not enought valid 'EXAFSKValues' values for running). I guess it will be disturbing for the user. Maybe we could provide an of numpy.nan
- the output might be different from the internal representation which is not an issue but won't help for any external coder.
Maybe it would interesting to store all the properties of the spectra when saving it at the end ?0.5