Nabu issueshttps://gitlab.esrf.fr/tomotools/nabu/-/issues2024-03-28T13:22:00+01:00https://gitlab.esrf.fr/tomotools/nabu/-/issues/496Sinogram rings removal issues when using non-contiguous arrays2024-03-28T13:22:00+01:00Pierre PaleoSinogram rings removal issues when using non-contiguous arrays - `munch`: when `padding != False`, the result is weird. This comes from DWT being applied on non-contiguous data. A temporary buffer has to be created.
- Same goes for `mean-subtraction` and `filter_cutoff = None` - `munch`: when `padding != False`, the result is weird. This comes from DWT being applied on non-contiguous data. A temporary buffer has to be created.
- Same goes for `mean-subtraction` and `filter_cutoff = None`https://gitlab.esrf.fr/tomotools/nabu/-/issues/495Get rid of Cuda texture warning2024-03-28T09:21:43+01:00Pierre PaleoGet rid of Cuda texture warningWhen JIT compiling the backprojection kernel with Cuda >= 11.7, the following warnings appear:
```
/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_27/lib/python3.11/site-packages/nabu/cuda/kernel.py:49: UserWarning:...When JIT compiling the backprojection kernel with Cuda >= 11.7, the following warnings appear:
```
/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_27/lib/python3.11/site-packages/nabu/cuda/kernel.py:49: UserWarning: The CUDA compiler succeeded, but said the following:
kernel.cu(228): warning #177-D: variable "x_stop_other" was declared but never referenced
kernel.cu(230): warning #177-D: variable "y_stop_other" was declared but never referenced
self.module = SourceModule(self.src, **self.sourcemodule_kwargs)
/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_27/lib/python3.11/site-packages/nabu/cuda/kernel.py:49: UserWarning: The CUDA compiler succeeded, but said the following:
kernel.cu(228): warning #177-D: variable "x_stop_other" was declared but never referenced
kernel.cu(230): warning #177-D: variable "y_stop_other" was declared but never referenced
self.module = SourceModule(self.src, **self.sourcemodule_kwargs)
/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_27/lib/python3.11/site-packages/nabu/cuda/kernel.py:49: UserWarning: The CUDA compiler succeeded, but said the following:
kernel.cu(131): warning #1215-D: function "tex2D(texture<T, 2, cudaReadModeElementType>, float, float) [with T=float]"
/cvmfs/hpc.esrf.fr/software/packages/ubuntu20.04/x86_64/cuda/11.7.1/bin/../targets/x86_64-linux/include/texture_fetch_functions.h(198): here was declared deprecated
kernel.cu(132): warning #1215-D: function "tex2D(texture<T, 2, cudaReadModeElementType>, float, float) [with T=float]"
/cvmfs/hpc.esrf.fr/software/packages/ubuntu20.04/x86_64/cuda/11.7.1/bin/../targets/x86_64-linux/include/texture_fetch_functions.h(198): here was declared deprecated
kernel.cu(133): warning #1215-D: function "tex2D(texture<T, 2, cudaReadModeElementType>, float, float) [with T=float]"
/cvmfs/hpc.esrf.fr/software/packages/ubuntu20.04/x86_64/cuda/11.7.1/bin/../targets/x86_64-linux/include/texture_fetch_functions.h(198): here was declared deprecated
kernel.cu(134): warning #1215-D: function "tex2D(texture<T, 2, cudaReadModeElementType>, float, float) [with T=float]"
/cvmfs/hpc.esrf.fr/software/packages/ubuntu20.04/x86_64/cuda/11.7.1/bin/../targets/x86_64-linux/include/texture_fetch_functions.h(198): here was declared deprecated
kernel.cu(21): warning #177-D: function "linear_interpolation" was declared but never referenced
```
This list of warnings is scary for users.
[Backprojector is ready for cuda 12](https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/337) so this warning can be removed.https://gitlab.esrf.fr/tomotools/nabu/-/issues/494`sliding-window` CoR estimation crashes on `bamboo_hercules.nx`2024-03-21T10:15:02+01:00Pierre Paleo`sliding-window` CoR estimation crashes on `bamboo_hercules.nx`When running non-regression tests on `bamboo_hercules.nx`, the CoR estimation crashes:
```
paleo@gpu4-01:~$ nabu /data/scisofttmp/paleo/nabu_end2end_tests/bamboo_hercules/nabu_2024.1.1.conf --slice middle
/cvmfs/tomo.esrf.fr/software/...When running non-regression tests on `bamboo_hercules.nx`, the CoR estimation crashes:
```
paleo@gpu4-01:~$ nabu /data/scisofttmp/paleo/nabu_end2end_tests/bamboo_hercules/nabu_2024.1.1.conf --slice middle
/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/cupyx/jit/_interface.py:173: FutureWarning: cupyx.jit.rawkernel is experimental. The interface can change in the future.
cupy._util.experimental('cupyx.jit.rawkernel')
Loaded darks from /scisoft/tomo_data/bamboo_hercules/bambou_hercules_0001_darks.hdf5
Loaded flats from /scisoft/tomo_data/bamboo_hercules/bambou_hercules_0001_flats.hdf5
Estimating center of rotation
Traceback (most recent call last):
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/bin/nabu", line 8, in <module>
sys.exit(main())
^^^^^^
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/nabu/app/reconstruct.py", line 111, in main
R = get_reconstructor(args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/nabu/app/reconstruct.py", line 86, in get_reconstructor
proc = ProcessConfig(conf_dict=conf_dict, create_logger=logfile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/nabu/pipeline/processconfig.py", line 64, in __init__
self._dataset_estimations()
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/nabu/pipeline/processconfig.py", line 164, in _dataset_estimations
self._get_cor()
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/nabu/pipeline/fullfield/processconfig.py", line 184, in _get_cor
cor = estimate_cor(
^^^^^^^^^^^^^
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/nabu/pipeline/estimators.py", line 67, in estimate_cor
estimated_cor = cor_finder.find_cor()
^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/nabu/pipeline/estimators.py", line 310, in find_cor
shift = self.cor_finder.find_shift(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/tomo.esrf.fr/software/packages/linux/x86_64/tomotools/2024_03_11/lib/python3.11/site-packages/nabu/estimation/cor.py", line 315, in find_shift
img_diff = img_1[:, win_1_start:win_1_end] - img_2[:, win_2_start:win_2_end]
```
It's probably because of a wrong `estimated_cor_from_motor` field in the `.nx` (actually `new_bamboo_hercules.nx`) should be used.
Still, the application should not crash like this.https://gitlab.esrf.fr/tomotools/nabu/-/issues/493Output three orthogonal slices during full-volume reconstruction2024-03-14T10:11:36+01:00Pierre PaleoOutput three orthogonal slices during full-volume reconstructionA common need (at least for BM05 and ID16A) is to extract slices in the three orthogonal planes, for quick preview (and upload in data portal).
This can be done in at least 3 ways:
1. Modify `chunked.py` so that, during full-volume re...A common need (at least for BM05 and ID16A) is to extract slices in the three orthogonal planes, for quick preview (and upload in data portal).
This can be done in at least 3 ways:
1. Modify `chunked.py` so that, during full-volume reconstruction, the relevant voxels are saved in dedicated (and separate) files
2. Reconstruct the full volume with `save_steps = sinogram`. Then, reconstruct the slices in 3 ortho planes resuming from the sinogram step.
3. AFTER full-volume reconstruction, read the reconstructed volume and extract the slices
Solution (1) should be the way to go, although tricky to implement, and I don't like the idea of having many outputs for the same set of parameters.
Solution (2) can be put in place almost immediately, possibly with a dedicated app. However it requires 3 additional steps.
Solution (3) is simple, but inefficient and heavy in I/O. However it will probably have to be done regardless, as user will sometimes forget to output orthogonal slices during reconstruction.https://gitlab.esrf.fr/tomotools/nabu/-/issues/491Phase Retrieval: support non-square pixel size2024-03-11T13:32:47+01:00Pierre PaleoPhase Retrieval: support non-square pixel sizeFor now `PaganinPhaseRetrieval` does
```python
fy = np.fft.fftfreq(nyp, d=self.pixel_size_micron)
fx = fftfreq(nxp, d=self.pixel_size_micron)
```
We should be available to provide a tuple for pixel size.
The same goes ...For now `PaganinPhaseRetrieval` does
```python
fy = np.fft.fftfreq(nyp, d=self.pixel_size_micron)
fx = fftfreq(nxp, d=self.pixel_size_micron)
```
We should be available to provide a tuple for pixel size.
The same goes for `CTFPhaseRetrieval`:
```python
fsample_vh = np.array(
[
self.geo_pars.length_scale / self.geo_pars.pix_size_rec,
self.geo_pars.length_scale / self.geo_pars.pix_size_rec,
]
)
```https://gitlab.esrf.fr/tomotools/nabu/-/issues/490multi-cor: create a master file ?2024-03-11T13:31:36+01:00paynomulti-cor: create a master file ?For now multi-cor creates one file per COR. There is no 'master-file'.
One reason is that the reconstructed slices have different shape (according to the cor value).
If we think this is necessary a Virtual dataset could be created with ...For now multi-cor creates one file per COR. There is no 'master-file'.
One reason is that the reconstructed slices have different shape (according to the cor value).
If we think this is necessary a Virtual dataset could be created with paddinghttps://gitlab.esrf.fr/tomotools/nabu/-/issues/489Detector tilt correction: use CoR by default (instead of middle of detector)2024-03-11T13:33:27+01:00paynoDetector tilt correction: use CoR by default (instead of middle of detector)By default `tilt_center = (middle_x/2, middle_z/2)`, configurable by user.
It would make more sense to have `tilt_center = (CoR/2, middle_z/2)`By default `tilt_center = (middle_x/2, middle_z/2)`, configurable by user.
It would make more sense to have `tilt_center = (CoR/2, middle_z/2)`R 2024.2.0https://gitlab.esrf.fr/tomotools/nabu/-/issues/488Radios rotation fails when reconstructing one slice2024-03-11T13:33:27+01:00Pierre PaleoRadios rotation fails when reconstructing one sliceRadios rotation (using `tilt_correction = 0.3`) works when reconstructing the full volume, but fails (incorrect result) when reconstructing a single slice (eg. `start_z = 305` and `end_z = 305`).
Using `--phase_margin 100000` gives a g...Radios rotation (using `tilt_correction = 0.3`) works when reconstructing the full volume, but fails (incorrect result) when reconstructing a single slice (eg. `start_z = 305` and `end_z = 305`).
Using `--phase_margin 100000` gives a good result, which suggests that partial rotation fails (i.e rotation applied to a subset of image instead of whole image).R 2024.2.0https://gitlab.esrf.fr/tomotools/nabu/-/issues/487nabu volume stitching: make sure axis stay consistant2024-03-07T14:15:48+01:00paynonabu volume stitching: make sure axis stay consistantA user reported that once reconstructed volume are stitched axis change. To be checked and fixed if neededA user reported that once reconstructed volume are stitched axis change. To be checked and fixed if neededhttps://gitlab.esrf.fr/tomotools/nabu/-/issues/486Make COR conversion from rel to abs consistent throughout the software2024-03-11T13:33:27+01:00Jerome LesaintMake COR conversion from rel to abs consistent throughout the softwareThe conversion from relative COR (output of `CORFinder.find_cor()`) to absolute position in not consistent depending of the COR estimator.
In `CORFinder` `SinoCORFinder`, the conversion is `abs = rel + detector_width // 2`.
In `Composite...The conversion from relative COR (output of `CORFinder.find_cor()`) to absolute position in not consistent depending of the COR estimator.
In `CORFinder` `SinoCORFinder`, the conversion is `abs = rel + detector_width // 2`.
In `CompositeCORFinder`, the conversion is `abs = rel + (detector_width - 1)// 2`.
After discussion with @paleo, it seems that the second one is the right one (to be confirmed).
Note that `tomwer` converts `CompositeCORFinder` does (see [#1136](https://gitlab.esrf.fr/tomotools/tomwer/-/issues/1136)).
Consistency should be enforced through the tomotools.R 2024.2.0https://gitlab.esrf.fr/tomotools/nabu/-/issues/484Doc: installation: warning about only linux supported2024-03-01T08:20:46+01:00paynoDoc: installation: warning about only linux supportedhttps://gitlab.esrf.fr/tomotools/nabu/-/issues/483Discrepancy in half-tomo size calculations, get_extended_sinogram_width() vs ...2024-03-11T13:33:27+01:00Vincent Favre-NicolinDiscrepancy in half-tomo size calculations, get_extended_sinogram_width() vs convert_halftomo()It seems there's an internal discrepancy (tested with 2024.1.1) in nabu when computing the extended sino width for half-tomo:
``` python
import numpy as np
from nabu.reconstruction.sinogram import get_extended_sinogram_width, SinoBuilde...It seems there's an internal discrepancy (tested with 2024.1.1) in nabu when computing the extended sino width for half-tomo:
``` python
import numpy as np
from nabu.reconstruction.sinogram import get_extended_sinogram_width, SinoBuilder
nproj = 540
nx = 640
cor = 144.89 # Works with 144.49 or 145
sino =np.zeros((nproj,1,nx), dtype=np.float32)
sino_builder = SinoBuilder(radios_shape=(nproj, 1, nx), rot_center=cor, halftomo=True)
sino_builder.get_sinos(sino)
```
results in:
```
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[1], line 12
8 sino =np.zeros((nproj,1,nx), dtype=np.float32)
10 sino_builder = SinoBuilder(radios_shape=(nproj, 1, nx), rot_center=cor, halftomo=True)
---> 12 sino_builder.get_sinos(sino)
File ~/.conda/envs/pynx-devel-py311-x86/lib/python3.11/site-packages/nabu/reconstruction/sinogram.py:193, in SinoBuilder.get_sinos(self, radios, output)
191 def get_sinos(self, radios, output=None):
192 if self.halftomo:
--> 193 return self._get_sinos_halftomo(radios, output=output)
194 else:
195 return self._get_sinos_simple(radios, output=output)
File ~/.conda/envs/pynx-devel-py311-x86/lib/python3.11/site-packages/nabu/reconstruction/sinogram.py:188, in SinoBuilder._get_sinos_halftomo(self, radios, output)
186 for i in range(n_z):
187 sino = self._get_sino_simple(radios, i)
--> 188 output[i] = self._get_sino_halftomo(sino)
189 return output
ValueError: could not broadcast input array from shape (270,990) into shape (270,991)
```
Issue seems to be that `get_extended_sinogram_width()` can compute an odd width, whereas `convert_halftomo()` will use `extended_width // 2 *2` for the shape.
This makes my half-tomo reconstructions fail when the fractional value of the CoR is within [0.5;1[R 2024.2.0https://gitlab.esrf.fr/tomotools/nabu/-/issues/482Cast_volume: enrich output_data_type to properly handle np.uint16 e.g.2024-02-27T13:50:14+01:00Jerome LesaintCast_volume: enrich output_data_type to properly handle np.uint16 e.g.For now, if `np.uint16` is passed, fails with `TypeError: output_data_type is expected to be a <class 'numpy.dtype'>. <class 'type'> provided`.
Passing `np.dtype(np.uint16)` works fine. Cast the type inside the function.For now, if `np.uint16` is passed, fails with `TypeError: output_data_type is expected to be a <class 'numpy.dtype'>. <class 'type'> provided`.
Passing `np.dtype(np.uint16)` works fine. Cast the type inside the function.https://gitlab.esrf.fr/tomotools/nabu/-/issues/480nabu-config: add an `--overwrite` option2024-02-26T10:43:30+01:00paynonabu-config: add an `--overwrite` optionWhen we know we want to overwrite a configuration file it would be convenientWhen we know we want to overwrite a configuration file it would be convenienthttps://gitlab.esrf.fr/tomotools/nabu/-/issues/477Extraneous folders created in "build"2024-02-13T15:27:53+01:00Pierre PaleoExtraneous folders created in "build"When installing nabu (eg. with `pip install .`), extraneous folders are created in `build`.
After several installations, we end up with something like `build/lib/build/lib/build/lib/....`
Worse, the installation `site-packages` is als...When installing nabu (eg. with `pip install .`), extraneous folders are created in `build`.
After several installations, we end up with something like `build/lib/build/lib/build/lib/....`
Worse, the installation `site-packages` is also affected: `venv/lib/python3.8/site-packages/build/lib/build/lib/....`
Probably an issue with `pyproject.toml`, to be investigated.https://gitlab.esrf.fr/tomotools/nabu/-/issues/476Fix cone-beam reconstruction2024-02-13T13:35:06+01:00Pierre PaleoFix cone-beam reconstructionThe cone-beam pipeline is probably broken as `SinoBuilder` is not used anymore.The cone-beam pipeline is probably broken as `SinoBuilder` is not used anymore.https://gitlab.esrf.fr/tomotools/nabu/-/issues/475Crashes with silx.io.get_data(), again2024-03-11T13:38:32+01:00Pierre PaleoCrashes with silx.io.get_data(), againThere still seems to be problems when using `get_data` from silx.
- Apparently the [SWMR mode was set to False by default](https://github.com/silx-kit/silx/pull/3939) when using `get_data()`
- On the other hand, nabu keeps open HDF5...There still seems to be problems when using `get_data` from silx.
- Apparently the [SWMR mode was set to False by default](https://github.com/silx-kit/silx/pull/3939) when using `get_data()`
- On the other hand, nabu keeps open HDF5 file descriptors in `H5Reader` (for performance - might be questionable), which is then used by `ChunkReader`, and incidentally `FlatFieldDataUrls` and so on
A workaround is to set the environment variable `HDF5_USE_FILE_LOCKING=1` if lock issues occur.
Consider the following code:
```python
radio_cor_finder = CORFinder("centered", dataset_info, do_flatfield=True, cor_options=cor_options)
sino_cor_finder = SinoCORFinder("sino-sliding-window", dataset_info, slice_idx="middle", do_flatfield=True, cor_options=cor_options)
composite_cor_finder = CompositeCORFinder(dataset_info, cor_options=cor_options)
```
From silx 2.0, these three estimators cannot be used simultaneously:
- `radio_cor_finder` has a `self.flatfield` which is a `FlatFieldDataUrls` instance, which keeps open HDF5 files descriptors
- `sino_cor_finder` has a `data_reader` which is a `H5Reader`, having the same issue
- `composite_cor_finder` has a `get_data()` which now uses `swmr=None` (was `True` in `silx < 2.0`)
An immediate but very brittle solution is do to
```python
del radio_cor_finder.flatfield
del sino_cor_finder.data_reader
```
once the data and flats are loaded by these instances.R 2024.2.0https://gitlab.esrf.fr/tomotools/nabu/-/issues/473nabu: allow reconstruction with several output format2024-02-07T15:15:15+01:00paynonabu: allow reconstruction with several output formatsometimes users would like to reconstruct in .hdf5 and .tiff for example or something else. Is it a good idea ? (data duplication) ?sometimes users would like to reconstruct in .hdf5 and .tiff for example or something else. Is it a good idea ? (data duplication) ?https://gitlab.esrf.fr/tomotools/nabu/-/issues/472stitching doc: inversion of the z2024-02-06T14:49:36+01:00paynostitching doc: inversion of the zOn the application by default everything is z down. when in the documentation the drawing are z up.On the application by default everything is z down. when in the documentation the drawing are z up.https://gitlab.esrf.fr/tomotools/nabu/-/issues/470Stitching: improve stitching log2024-02-01T16:36:36+01:00paynoStitching: improve stitching logWhen doing stitching get the following message:
```
0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;...When doing stitching get the following message:
```
0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001;0001 will be picked
```
Only providing entries is not very helpful. We should provide the identifier or the `short_description` of the object that would make more sense.