nxtomomill merge requestshttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests2023-07-27T14:34:28+02:00https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/171case of estimated cor equal to None2023-07-27T14:34:28+02:00myroncase of estimated cor equal to Nonepossible bug correctionpossible bug correction0.13myronmyronhttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/169Add transformations2023-08-07T11:55:35+02:00paynoAdd transformations# TODO
* add `NXtransformations`
* [x] add NXTransformations class and utils
* [x] replace `flip_lr` and `flip_ud` dataset by standard NXTranslations -\> for the large part of this pr remove it but it would be better if we can keep ...# TODO
* add `NXtransformations`
* [x] add NXTransformations class and utils
* [x] replace `flip_lr` and `flip_ud` dataset by standard NXTranslations -\> for the large part of this pr remove it but it would be better if we can keep some compatibility for some time
* [x] ~~update rotation_angle, x_translation, y_translation, z_translation ?~~ -\> lets go one step at the time
* [x] add an example on how to add Transformation to an NXdetector from a script (and limitations)
* [x] when dump to disk; for the Transformation without dependencies add dependence on gravity
* [x] ~~integrate to nabu~~ -\> https://gitlab.esrf.fr/tomotools/nabu/-/issues/418
* [x] ~~integrate to tomwer~~ -\> https://gitlab.esrf.fr/tomotools/tomwer/-/issues/967
# `TRANSFORMATIONS` and `NXsample`
Today NXtomo documentation does not present any link to `NXtransformations` when `NXsample` does and recommand to use them. Which is confusing for the less.
Anyway for tomography `{x,y,z}_translation` and `rotation_angle` are convenient to have and we don't want to remove them. `rotation_angle` is even mandatory.
So what could be done is to generate `NXtransformations` group but based from other dataset values. This would prevent any conflict and make it more 'nexus' compliant. Else we keep this to the NXdetector which is necessary to store 'flip' l/r and u/d information
# related to
* https://gitlab.esrf.fr/tomotools/tomoscan/-/merge_requests/145
* https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/316
# screenshots
## results on a real use case output of h52nx (getting one 'x flip (lr_flip)')
![image](/uploads/e435d706482e81b46af160bfc841d017/image.png) ![image](/uploads/b92d2e24f951a0147eca6fdfb84c56a8/image.png)
## wout explanation of NXtransformations
![explaination of the NXtransformations by wout](/uploads/2f6d3d2d67f35ee69a33df935fbc5f4f/IMG_20230704_094828.jpg)1.0paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/168get scan type from technique sub group2023-07-05T16:31:57+02:00paynoget scan type from technique sub groupclose #127close #1270.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/167deprecate `from_dx_to_nx` and create from_dx_config_to_nx2023-07-27T14:34:31+02:00paynodeprecate `from_dx_to_nx` and create from_dx_config_to_nxclose #141close #1410.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/166rename `h5-quick-start` to `h5-config` and `edf-quick-start` to `edf-config`2023-06-21T17:02:07+02:00paynorename `h5-quick-start` to `h5-config` and `edf-quick-start` to `edf-config`simpler for users, more coherent with nabu
close #139
Signed-off-by: payno <henri.payno@esrf.fr>simpler for users, more coherent with nabu
close #139
Signed-off-by: payno <henri.payno@esrf.fr>0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/165h52nx: improve robustness against cancelled scans2024-01-08T14:37:33+01:00paynoh52nx: improve robustness against cancelled scans* test `/data/bm18/inhouse/testzhelicaljibril/` datasets and make sure all data that are relevant can be inserted into an NXtomo
* test YYY (to be provided by Jibril) and make sure the output are coherent with the latest version and that...* test `/data/bm18/inhouse/testzhelicaljibril/` datasets and make sure all data that are relevant can be inserted into an NXtomo
* test YYY (to be provided by Jibril) and make sure the output are coherent with the latest version and that helical scans are still properly concatenated / handled0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/164edf: add a new command to test the edf conversion on post processing2023-06-21T16:34:52+02:00paynoedf: add a new command to test the edf conversion on post processing* test on a real life use case
* test old command
* test new check command
* add test for the new use case (convert first then check conversion)
close #146* test on a real life use case
* test old command
* test new check command
* add test for the new use case (convert first then check conversion)
close #1460.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/163TomoHDF5Config.from_dict: Support lower case section keys2023-07-27T14:34:33+02:00Thomas VincentTomoHDF5Config.from_dict: Support lower case section keysThis MR makes TomoHDF5Config.from_dict supports case insensitive section keysThis MR makes TomoHDF5Config.from_dict supports case insensitive section keys0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/162Fix dangling relative path in h52nx converter2023-07-27T14:47:42+02:00myronFix dangling relative path in h52nx converter- [x] Fix the problem
close #145- [x] Fix the problem
close #1450.13myronmyronhttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/161Handle estimated cor for all scan type2023-07-27T14:34:36+02:00paynoHandle estimated cor for all scan type0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/160h52nx: fix for pcotomo: if the acquisition is a pcotomo create several nxtomo...2024-01-08T14:37:33+01:00paynoh52nx: fix for pcotomo: if the acquisition is a pcotomo create several nxtomo...h52nx: fix for pcotomo: if the acquisition is a pcotomo create several nxtomo file as the acquisition will be splitted later even if has only a single file and a single acquisition.h52nx: fix for pcotomo: if the acquisition is a pcotomo create several nxtomo file as the acquisition will be splitted later even if has only a single file and a single acquisition.0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/159h52nx: fix getting values if incoherent number vs number of frames (hard fix)...2024-01-08T14:37:33+01:00paynoh52nx: fix getting values if incoherent number vs number of frames (hard fix),...h52nx: fix getting values if incoherent number vs number of frames (hard fix), if a scan is cancelled can bring troublesh52nx: fix getting values if incoherent number vs number of frames (hard fix), if a scan is cancelled can bring troubles0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/158improve handling of cancel scan2024-01-08T14:37:32+01:00paynoimprove handling of cancel scan# TODO
when some bliss scan are canceled we should try to create as much as possible NXtomo as possible. Up to mocking some frame (and tag them as invalid of course)
- [x] add feature
- [x] add test
- [x] test on real data
# extra
cl...# TODO
when some bliss scan are canceled we should try to create as much as possible NXtomo as possible. Up to mocking some frame (and tag them as invalid of course)
- [x] add feature
- [x] add test
- [x] test on real data
# extra
close #1420.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/157CI: move test to pythons 3.8, 3.10 and 3.112023-04-18T09:18:04+02:00paynoCI: move test to pythons 3.8, 3.10 and 3.11# TODO
- [x] update CI
- [x] check logs for any deprecation warning
# warnings
## load_module
```
nxtomomill/test/test_plugins.py::test_get_plugins_instances
<frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() me...# TODO
- [x] update CI
- [x] check logs for any deprecation warning
# warnings
## load_module
```
nxtomomill/test/test_plugins.py::test_get_plugins_instances
<frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead
```0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/156h52nx: remove real pixel size and magnifications2023-06-21T16:36:52+02:00paynoh52nx: remove real pixel size and magnifications0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/155edf2nx: Add output check and delete source files.2023-06-21T16:34:51+02:00paynoedf2nx: Add output check and delete source files.# TODO
- [x] add an option to remove EDF source file
- [x] add a method to check the output volume. for this we have several ways to handle it
- get a boolean `check_output_volume` -> for me this does not bring enough option to fine...# TODO
- [x] add an option to remove EDF source file
- [x] add a method to check the output volume. for this we have several ways to handle it
- get a boolean `check_output_volume` -> for me this does not bring enough option to fine tune. Indeed I think in most of the cases we want to do a quick check (checking Virtual dataset) as in the `tomoscan validators` but sometimes (when removing source files - ID16A) we probably want to do more advanced stuff like comparing frame by frame the final vs raw volumes.
- get an integer to define a level (0: no check, 1: check for broken/invalid links on the final volume or frame at zero, 2: compare frame by frame)
- get a list of possible checks like -> this is the option selected
- empty: no check done
- `check_vds_links`: check for broken virtual source
- `compare_volumes`: compare input volume and output volume
- `check_vds_links, compare_volumes`: do both
- [x] add unit test
- [x] test on a real dataset
- [x] add CLI option for checking volume and removing input because I think ID16a script are using directly the CLI
- [x] dedicated handling for multi edf files
# limitations
* the .info will not be removed. This can be added but not so sure about the interest (especially in the case this is not the default one ("{basename}.info") but another one. This could lead to important technical mishap from users and with no ways to go back.
* For now the two options to access it are only available on 'advanced' `edf2nx` configuration file. But looks like to be the usage. If not we can still add it later to the CLI directly.
* I am not sure this test is enough ? But the core of the processing from my point if view is more on the algorithm behind `edf2nx`.
# usage
## from the configuration file
this PR adds two parameters to the `edf2nx` configuration file under the `general section`:
* `delete_edf_source_file`: as a boolean. False is the default, if True then the .edf used during the conversion will be removed.
* `output_checks`: as a tuple of str. For now the only str available is `compare-output-volume` which test frame by frame the url used and the final HDF5 volume
![image](/uploads/08e7af0fa18dc0527db9d517334193b0/image.png)
## from the command line
``` bash
nxtomomill edf2nx {} --output-checks compare-output-volume --delete-edf
```0.13paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/154replace string.format by f-strings when possible2023-03-10T09:59:51+01:00paynoreplace string.format by f-strings when possiblesame as for tomoscan, let's try to be modern (enough - python 3.6 :laughing: )same as for tomoscan, let's try to be modern (enough - python 3.6 :laughing: )0.13paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/153corrected a possible error message f-string2023-02-23T13:40:58+01:00myroncorrected a possible error message f-stringthe variable used in the f-string was wrongthe variable used in the f-string was wrong0.12paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/152fixed value method requested on bool2023-02-23T13:40:53+01:00myronfixed value method requested on boolfixfix0.12paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/151remove distutils2023-02-23T13:40:47+01:00paynoremove distutils0.12