nxtomomill merge requestshttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests2023-07-27T14:47:42+02:00https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/162Fix dangling relative path in h52nx converter2023-07-27T14:47:42+02:00myronFix dangling relative path in h52nx converter- [x] Fix the problem
close #145- [x] Fix the problem
close #1450.13myronmyronhttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/161Handle estimated cor for all scan type2023-07-27T14:34:36+02:00paynoHandle estimated cor for all scan type0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/160h52nx: fix for pcotomo: if the acquisition is a pcotomo create several nxtomo...2024-01-08T14:37:33+01:00paynoh52nx: fix for pcotomo: if the acquisition is a pcotomo create several nxtomo...h52nx: fix for pcotomo: if the acquisition is a pcotomo create several nxtomo file as the acquisition will be splitted later even if has only a single file and a single acquisition.h52nx: fix for pcotomo: if the acquisition is a pcotomo create several nxtomo file as the acquisition will be splitted later even if has only a single file and a single acquisition.0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/159h52nx: fix getting values if incoherent number vs number of frames (hard fix)...2024-01-08T14:37:33+01:00paynoh52nx: fix getting values if incoherent number vs number of frames (hard fix),...h52nx: fix getting values if incoherent number vs number of frames (hard fix), if a scan is cancelled can bring troublesh52nx: fix getting values if incoherent number vs number of frames (hard fix), if a scan is cancelled can bring troubles0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/158improve handling of cancel scan2024-01-08T14:37:32+01:00paynoimprove handling of cancel scan# TODO
when some bliss scan are canceled we should try to create as much as possible NXtomo as possible. Up to mocking some frame (and tag them as invalid of course)
- [x] add feature
- [x] add test
- [x] test on real data
# extra
cl...# TODO
when some bliss scan are canceled we should try to create as much as possible NXtomo as possible. Up to mocking some frame (and tag them as invalid of course)
- [x] add feature
- [x] add test
- [x] test on real data
# extra
close #1420.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/157CI: move test to pythons 3.8, 3.10 and 3.112023-04-18T09:18:04+02:00paynoCI: move test to pythons 3.8, 3.10 and 3.11# TODO
- [x] update CI
- [x] check logs for any deprecation warning
# warnings
## load_module
```
nxtomomill/test/test_plugins.py::test_get_plugins_instances
<frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() me...# TODO
- [x] update CI
- [x] check logs for any deprecation warning
# warnings
## load_module
```
nxtomomill/test/test_plugins.py::test_get_plugins_instances
<frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead
```0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/156h52nx: remove real pixel size and magnifications2023-06-21T16:36:52+02:00paynoh52nx: remove real pixel size and magnifications0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/155edf2nx: Add output check and delete source files.2023-06-21T16:34:51+02:00paynoedf2nx: Add output check and delete source files.# TODO
- [x] add an option to remove EDF source file
- [x] add a method to check the output volume. for this we have several ways to handle it
- get a boolean `check_output_volume` -> for me this does not bring enough option to fine...# TODO
- [x] add an option to remove EDF source file
- [x] add a method to check the output volume. for this we have several ways to handle it
- get a boolean `check_output_volume` -> for me this does not bring enough option to fine tune. Indeed I think in most of the cases we want to do a quick check (checking Virtual dataset) as in the `tomoscan validators` but sometimes (when removing source files - ID16A) we probably want to do more advanced stuff like comparing frame by frame the final vs raw volumes.
- get an integer to define a level (0: no check, 1: check for broken/invalid links on the final volume or frame at zero, 2: compare frame by frame)
- get a list of possible checks like -> this is the option selected
- empty: no check done
- `check_vds_links`: check for broken virtual source
- `compare_volumes`: compare input volume and output volume
- `check_vds_links, compare_volumes`: do both
- [x] add unit test
- [x] test on a real dataset
- [x] add CLI option for checking volume and removing input because I think ID16a script are using directly the CLI
- [x] dedicated handling for multi edf files
# limitations
* the .info will not be removed. This can be added but not so sure about the interest (especially in the case this is not the default one ("{basename}.info") but another one. This could lead to important technical mishap from users and with no ways to go back.
* For now the two options to access it are only available on 'advanced' `edf2nx` configuration file. But looks like to be the usage. If not we can still add it later to the CLI directly.
* I am not sure this test is enough ? But the core of the processing from my point if view is more on the algorithm behind `edf2nx`.
# usage
## from the configuration file
this PR adds two parameters to the `edf2nx` configuration file under the `general section`:
* `delete_edf_source_file`: as a boolean. False is the default, if True then the .edf used during the conversion will be removed.
* `output_checks`: as a tuple of str. For now the only str available is `compare-output-volume` which test frame by frame the url used and the final HDF5 volume
![image](/uploads/08e7af0fa18dc0527db9d517334193b0/image.png)
## from the command line
``` bash
nxtomomill edf2nx {} --output-checks compare-output-volume --delete-edf
```0.13paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/154replace string.format by f-strings when possible2023-03-10T09:59:51+01:00paynoreplace string.format by f-strings when possiblesame as for tomoscan, let's try to be modern (enough - python 3.6 :laughing: )same as for tomoscan, let's try to be modern (enough - python 3.6 :laughing: )0.13paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/153corrected a possible error message f-string2023-02-23T13:40:58+01:00myroncorrected a possible error message f-stringthe variable used in the f-string was wrongthe variable used in the f-string was wrong0.12paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/152fixed value method requested on bool2023-02-23T13:40:53+01:00myronfixed value method requested on boolfixfix0.12paynopaynohttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/151remove distutils2023-02-23T13:40:47+01:00paynoremove distutils0.12https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/150edfconverter: add unit for machine electric current.2023-02-14T14:30:55+01:00paynoedfconverter: add unit for machine electric current.handle unit for the machine electric current.
close #120handle unit for the machine electric current.
close #1200.12https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/149h52nx: improve electric current calculation.2023-01-27T14:56:08+01:00paynoh52nx: improve electric current calculation.# TODO
- [x] add an option to skip this part
- [x] test it on real dataset
- [x] speed up the interpolation
- [x] move calculation to multi-thread
- parallelize research of time stamp and `machine electrical current` interpol...# TODO
- [x] add an option to skip this part
- [x] test it on real dataset
- [x] speed up the interpolation
- [x] move calculation to multi-thread
- parallelize research of time stamp and `machine electrical current` interpolation which are the part taking most of the time.
- [x] reduce research time
- both timestamp and know_timestamp are ordered: only look forward (add start_index).
- force to filter research timestamp before but algorithmic gain is substantial (and remove workaround we did to speed up)
- filter know_timestamp (remove energies surround by the same energy ...) -> seems to reduce the `known_electric_current` about a factor of 4 to 5. Still good to take (speed up time stamp research)
- [x] test it on real dataset
# dummy figures
on the datasets (180k projections) I tested (from ludo / ID19) I had to stop this processing because it was taking way to much time (>10 minutes). I think it was way more. Anyway such amount of time is unacceptable for moving to a NXtomo.
With this improvement it takes ~~ ~ 10s on the `gpu` partition with (cpu_count() == 64). (note: 'only' half of the available cpu are used).~~ 4s in on gpu since using numpy.searchsorted thanks to @mirone
# screenshot
result on the dataset processed
![image](/uploads/4e081c15713b06b3a61c06d6c31892cf/image.png)
![image](/uploads/91252405f6aa8bbea83c1258fa966916/image.png)
on this last one the lost of the beam was 'real' and expected...0.12https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/147Handle (new) metadata2023-04-06T08:56:16+02:00paynoHandle (new) metadata# GOAL
several information like motor names used... must be available in the bliss `ebstomo` file from https://gitlab.esrf.fr/tomo/ebs-tomo/-/merge_requests/147.
This PR intend to benefit from it.
# new behavior
try to reach `instrume...# GOAL
several information like motor names used... must be available in the bliss `ebstomo` file from https://gitlab.esrf.fr/tomo/ebs-tomo/-/merge_requests/147.
This PR intend to benefit from it.
# new behavior
try to reach `instrument/tomo_config` group. If find it: use it, If not and if version > X: warn the user. In both case roll back to using the motor provided in the configuration file and default paths to search.
# TODO
- [x] add some preprocessing to get the motor position when possible
- add an option to ignore using 'tomoconfig' section (insure backward compatibility and allow user to still redefine motor names...)
- [x] from python API
- [x] from configuration file
- [x] add unit tests
- [x] test on real dataset0.13https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/146Fix flipping2022-12-15T09:43:26+01:00paynoFix flipping0.11https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/145Flatreducer2023-02-23T14:26:21+01:00myronFlatreducerFlat reduction from projections of atwo auxiliary scans ( B as before and E as End) targeting a scan with the sampleFlat reduction from projections of atwo auxiliary scans ( B as before and E as End) targeting a scan with the sample0.12myronmyronhttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/144NXtomo.concatenate: better handling of the Virtual Source relative path2022-12-07T16:30:08+01:00paynoNXtomo.concatenate: better handling of the Virtual Source relative path0.11https://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/143nexus: NXtomo.concatenate: fix corner case where no start_time and / or stop_...2022-12-07T13:50:10+01:00paynonexus: NXtomo.concatenate: fix corner case where no start_time and / or stop_timehttps://gitlab.esrf.fr/tomotools/nxtomomill/-/merge_requests/141Add pylint2022-11-23T14:53:55+01:00paynoAdd pylintclose #123close #1230.11