Nabu merge requestshttps://gitlab.esrf.fr/tomotools/nabu/-/merge_requests2024-03-05T10:40:20+01:00https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/306Half tomography reconstruction without sinogram stitching2024-03-05T10:40:20+01:00Pierre PaleoHalf tomography reconstruction without sinogram stitching## About
Follow-up of #378.
Building the extended sinogram when using half-tomography (HA) mode was appealing for some purposes (ex. debug), but it has numerous issues:
- When there are intensities fluctuations, stitching the two pa...## About
Follow-up of #378.
Building the extended sinogram when using half-tomography (HA) mode was appealing for some purposes (ex. debug), but it has numerous issues:
- When there are intensities fluctuations, stitching the two parts will give artefacts (#378)
- Cannot use horizontal translations #411
- Eventually more complex design
From this MR, HA-reconstruction will be done without building the half-tomography sinogram.
Nabu still may provide some script to visualize the extended sinogram.
On a ID16B dataset, the only working approach is the following:
- Filter the sinogram
- Extend using the algotom approach in `[Vo21]`
- Use plain backprojetion
## To do
- [x] `SinoMult` class and cuda backend
- [x] Include halftomo handling directly in FBP
- [x] Update halftomo unit tests
- [x] Update pipeline
- [x] End-to-end reconstruction test
## Notes
The sinogram extension used is the one from algotom: https://github.com/algotom/algotom/blob/master/algotom/prep/conversion.py#L394
The sub-pixel shift is probably overkill, though.
```python
# Extend the 360deg sinogram, then FBP
si_extended, nc = extend_sinogram(si, r, apply_log=False)
B_e = Backprojector(si_extended.shape, padding_mode="edges", rot_center=nc, extra_options={"centered_axis": True}, angles=np.linspace(0, 2*np.pi, si_p.shape[0]))
rec = B_e.fbp(si_extended) # not so good
# Filter the 360deg sinogram
B0 = Backprojector(si.shape, padding_mode="edges", rot_center=r)
si_f = B0.sino_filter(si)
# Extend this filtered sinogram, then plain backproj
si_extended2, nc = extend_sinogram(si_f, r, apply_log=False)
rec2 = B_e.backproj(si_extended2) # good
```
## References
`[Vo21] Nghia T. Vo, Robert C. Atwood, Michael Drakopoulos, and Thomas Connolley, "Data processing methods and data acquisition for samples larger than the field of view in parallel-beam tomography," Opt. Express 29, 17849-17874 (2021) - https://opg.optica.org/oe/fulltext.cfm?uri=oe-29-12-17849&id=451366`R 2023.2.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/298Add bliss-cor-8 estimation2024-01-16T13:32:39+01:00Jerome LesaintAdd bliss-cor-8 estimation## About
This update includes:
* \[\] The handling of the Nexus `estimated_cor_frm_motor` in `fourier-angles` and `sliding-window` COR estimators.
* \[\] A new `octave-accurate` COR estimator, which implements the older Octave/accurate...## About
This update includes:
* \[\] The handling of the Nexus `estimated_cor_frm_motor` in `fourier-angles` and `sliding-window` COR estimators.
* \[\] A new `octave-accurate` COR estimator, which implements the older Octave/accurate algo.
* \[\] The `fourier-angles` COR estimator, which is the origin of this MR has not changed, except for the API, which is now aligned with other estimators.
* \[\] The `near_pos` option can take on values which are either a relative COR position, `ignore`, `from_file`. In case of none of the previous values, set to \`ignore'.
* `ignore`: disregard the value in Nexus file and acts as before.
* `from_file`: tries to pick the value in Nexus file. Switch to `ignore` if fails.
* Number: the user-provided COR value overrides the value in the Nexus file.
Tests were added to check this behavior.
Add CoR estimation algorithm submitted by V. Valls (BCU), based on FT of angular pixel signals.R 2024.1.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/370Projections subsampling: reconstruct from even or odd projections2023-11-30T14:43:57+01:00Pierre PaleoProjections subsampling: reconstruct from even or odd projections## About
ID16 needs to reconstruct from even / odd projections (see #397 ).
With nabu it's possible (using `[dataset] projections_subsampling`) to reconstruct from a subset of the projections. In this case the simplest is probably to de...## About
ID16 needs to reconstruct from even / odd projections (see #397 ).
With nabu it's possible (using `[dataset] projections_subsampling`) to reconstruct from a subset of the projections. In this case the simplest is probably to define something like `subsampling_step:beginning` (eg. `2:1` to reconstruct from odd projections).
Close #397
## To do
- [x] Update `nabu_config` and validators
- [x] Update `ProcessConfig` and validation
- [x] Update chunk reader
- [x] Unit test
- [x] Update changelog/documentation
- [x] End-to-end reconstruction test
## Notes
The implementation is not so obvious. For the record:
- When parsing a dataset, `tomoscan` build a dictionary of projections in the form `{idx: data_url}` (see example below)
- However for performances, it's better to read a big data subvolume rather than individual images (eg. `h5_dataset[begin:end, :, :]` rather than many calls to `h5_dataset[i, :, :]`)
- So nabu uses a `get_compacted_dataslices()` function which builds a minimal collection of `DataUrl(..., data_slice=(begin, end, step))`, so that only a few calls to `h5_read` are done. The implementation of `get_compacted_dataslices` is not so trivial when subsampling is considered.
Example:
```python
from nabu.io.reader import ChunkReader
from nabu.resources.dataset_analyzer import analyze_dataset
from nabu.io.utils import get_compacted_dataslices
di = analyze_dataset("/tmp/nabu_testdata_paleo/bamboo_reduced.nx")
reader_full = ChunkReader(di.projections)
reader_odd = ChunkReader(di.projections, dataset_subsampling=(2,1))
reader_even = ChunkReader(di.projections, dataset_subsampling=(2,0))
# projection indices: [26, 27, ..., 525] [551, ...., 1050]
# (the "jump" in the middle is due to the presence of a series of flats)
# subsampling-odd: [27, 29, 31, ..., 523, 525] [552, 554, 556, ..., 1048, 1050]
# subsampling-even: [26, 28, ..., 522, 524] [551, 553, ..., 1047, 1049]
```
then
```python
get_compacted_dataslices(reader_full.files)
```
returns
```
{26: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(26, 526, 1)),
27: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(26, 526, 1)),
28: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(26, 526, 1)),
29: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(26, 526, 1)),
# ...
524: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(26, 526, 1)),
525: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(26, 526, 1)),
551: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(551, 1051, 1)),
552: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(551, 1051, 1)),
# ...
1049: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(551, 1051, 1)),
1050: DataUrl(valid=True, scheme=None, file_path='/tmp/nabu_testdata_paleo/bamboo_reduced.nx', data_path='/entry0000/instrument/detector/data', data_slice=slice(551, 1051, 1))}
```
(note the "jump" in indices in the middle).
Then
```python
slice_to_tuple = lambda s: (s.start, s.stop, s.step)
set([slice_to_tuple(u.data_slice()) for u in get_compacted_dataslices(reader_even.files, subsampling=reader_even.dataset_subsampling, begin=reader_even._files_begin_idx).values()])
# returns {(26, 526, 2), (551, 1051, 2)}
set([slice_to_tuple(u.data_slice()) for u in get_compacted_dataslices(reader_odd.files, subsampling=reader_odd.dataset_subsampling, begin=reader_odd._files_begin_idx).values()])
# returns {(27, 526, 2), (552, 1051, 2)}
```R 2024.1.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/328Fix CPU/OpenCL pipeline2023-08-22T14:24:28+02:00Pierre PaleoFix CPU/OpenCL pipeline## About
It seems that the numpy/CPU (+OpenCL) pipeline is broken.
This MR also removes some unused imports, which made things crash (eg. `fbp_base` when no pyopencl was available).
## To do
- [x] Remove unused imports
- [x] Fix...## About
It seems that the numpy/CPU (+OpenCL) pipeline is broken.
This MR also removes some unused imports, which made things crash (eg. `fbp_base` when no pyopencl was available).
## To do
- [x] Remove unused imports
- [x] Fix CPU pipeline
- [x] In `OpenclProcessing`: `devicetype="GPU"` by default - what if no GPU ?
- [x] `pipeline/reconstruction.py`: check that cuda is available before trying to use `CudaReconstructor`
- [x] Fix input/output numpy arrays in `OpenclBackprojector`
- [x] `OpenclBackprojector`: fix `centered_axis` (does not work when using halftomo!)
- [x] End to end testsR 2023.2.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/299Fix pipeline writers2023-08-10T10:18:46+02:00Pierre PaleoFix pipeline writers## About
This MR fixes the inconsistencies for pipeline writers.
Close #381
## To do
- [x] Update `pipeline.WriterManager`
- [x] Update/add individual writers in `nabu.io.writer` when necessary
- [~] Unit tests
- [x] End-to-...## About
This MR fixes the inconsistencies for pipeline writers.
Close #381
## To do
- [x] Update `pipeline.WriterManager`
- [x] Update/add individual writers in `nabu.io.writer` when necessary
- [~] Unit tests
- [x] End-to-end reconstruction test
## Notes
* Define one writer per file type, with the same API, as it was done by `nabu.io.writer`
* Each writer can be backed by tomoscan volume
* `WriterManager` does minimal things to assemble them and dispatch parametersR 2023.2.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/246CoR improvements2023-01-13T17:41:42+01:00Pierre PaleoCoR improvements## About
This MR fixes a couple of bugs and brings simplifications in the codebase.
## To do
- [x] Create a `CORDispatcher` class for pipeline. It takes `dataset_info` (and other pipeline-related params) and dispatches to the suitab...## About
This MR fixes a couple of bugs and brings simplifications in the codebase.
## To do
- [x] Create a `CORDispatcher` class for pipeline. It takes `dataset_info` (and other pipeline-related params) and dispatches to the suitable estimators.
- [ ] ~~Simplify estimators to act on simpler data structures~~
- [x] Fix #368
- [ ] ~~(Bonus) Save CoR result~~
- [x] End-to-end reconstruction testR 2023.1.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/175Fix horizontal/vertical translations in pipeline2022-05-02T16:34:49+02:00Pierre PaleoFix horizontal/vertical translations in pipeline## About
This MR aims at bringing (back ?) the "translations" feature in the pipeline.
## To do
- [x] Plug `ctf_translations` and `translation_movements_file` to the pipeline
- [x] Check if `ctf_translations` is actually a duplica...## About
This MR aims at bringing (back ?) the "translations" feature in the pipeline.
## To do
- [x] Plug `ctf_translations` and `translation_movements_file` to the pipeline
- [x] Check if `ctf_translations` is actually a duplicate of `translation_movements_file`
- [x] End-to-end reconstruction tests
Close #299, #300R 2022.2.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/162Full-field pipeline without needing GPU2022-04-07T13:27:23+02:00Pierre PaleoFull-field pipeline without needing GPUThis MR is a first step toward a pipeline that does not need GPU at all.
With the current implementation (`CudaFullFieldPipeline` inheriting from `FullFieldPipeline`), it is theoretically possible to run a reconstruction without needi...This MR is a first step toward a pipeline that does not need GPU at all.
With the current implementation (`CudaFullFieldPipeline` inheriting from `FullFieldPipeline`), it is theoretically possible to run a reconstruction without needing Cuda. However:
- Some steps are still very slow with the numpy backend
- `Reconstructor`s *require* cuda in the current state - it should not be!
- Imports can be sometimes overlooked (crash if cuda is not present in some situations)
**This MR focuses on simple solutions (numpy + multithread). OpenCL will be considered later**.
## To do
- [x] List the performance bottleneck steps on CPU --> See [below](#note_161949)
- [x] FFTW support for `CTFPhaseRetrieval`
- [x] Unit test
- [x] Integration in pipeline
- [x] Multi-thread support for Histogram
- [x] Multi-thread support for binning
- [ ] ~~Multi-thread support for Unsharp~~ will be done later
- [ ] ~~Multi-thread support for Log~~ will be done later
- [x] Pipeline: ensure `nabu nabu.conf` works even without Cuda installed
- [x] End-to-end tests, measure the speed of CPU-only pipeline
Bonus:
- [x] Update `CTFPhaseRetrieval` so that API follows the design decisions
- [x] Update `test_ctf`
- [x] End-to-end CTF test
- [x] Update documentation for CTF and `get_num_threads` behavior
- [x] Refactor `Reconstructor`s - close #260
Close #230, #261
## Notes
- When referring to "CPU-only pipeline", the "backend" terminology should be used to avoid confusion. "CPU-numpy", "CPU-opencl", "GPU-cuda", "GPU-opencl", ...
- Some steps might still be very slow even with numpy + multi-thread: histogram, backprojection. Only a dedicated implementation (C++/OpenCL) can bring reasonable performances for these.R 2022.1.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/150Cuda context handling for Cuda >= 112021-12-09T18:37:49+01:00Pierre PaleoCuda context handling for Cuda >= 11This MR is an attempt at fixing #247.
A reconstructor may instantiate several pipelines objects in a reconstruction (since each pipeline is tied to a certain chunk/group size). It is critical to release the Cuda memory when re-initiali...This MR is an attempt at fixing #247.
A reconstructor may instantiate several pipelines objects in a reconstruction (since each pipeline is tied to a certain chunk/group size). It is critical to release the Cuda memory when re-initializing a pipeline.
In theory, pycuda transparently releases the resources when deleting the object (RAII model),
but Python's multiple references and garbage collection make things more difficult.
Therefore, doing `self.pipeline.d_radios = None` is not enough to actually release the memory.
The only working method was found to be `self.pipeline = None; import gc; gc.collect()`.
Prior to cuda 11, this worked well.
But from Cuda >= 11, we get "invalid context handle" errors for some reason.
For now, the workaround is to manually handle cuda contexts:
- Context is created (or pushed) at each new `pipeline` instantiation (through `nabu.cuda.utils.get_cuda_context()`)
- When a new pipeline has to be re-initialized, the previous one is destroyed
- For Cuda < 11, this is fine
- For Cuda >= 11, we also have to re-push the cuda context
- Create a new pipeline instance
The drawback is that he context handling is now done by `Reconstructor`.
**Update: new workaround found** - see https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/150#note_152136
Close #247Release 2021.2.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/113Detect and auto-correct camera tilt for centered CoR2021-03-31T15:48:52+02:00Nicola ViganoDetect and auto-correct camera tilt for centered CoR## About
This MR is part of a series of MRs on:
Automatic detection and correction of detector's pixel columns tilts with respect to the rotation axis.
### General
Ultimate goal is to be able to specify the following in the configurat...## About
This MR is part of a series of MRs on:
Automatic detection and correction of detector's pixel columns tilts with respect to the rotation axis.
### General
Ultimate goal is to be able to specify the following in the configuration file:
Add a new key in the same spirit of `rotation_axis_position`, called `tilt_correction = [auto|algo1|algo2|value]` where:
* `auto` is the default automatic tilt computation method (could be different between rotation axis in the center and rotation axis on the side).
* `algo1`, `algo2` refer to tilt computation methods
* `value` is a scalar chosen by the user
Follow-up of !102.
### Specific
Here we implement a new method for the tilt detection based on the fft-polar transform (log could be an option, to be discussed), and its integration in the pipeline.
This MR only targets acquisitions with rotation axis in the center. Acquisition with rotation axis on the side, will be tackled in a different MR, because they require extra care.
## To do
- [x] Implement fft-ploar transform based tilt fitting in `CameraTilt` (method and docstring)
- [x] Implement handling of the configuration file
- [x] Unit tests
- [x] Integrate in pipeline
- [x] End-to-end reconstruction test
## Notes
The new interface of `CameraTilt`, from `namu/preproc/alignment.py` will be:
```python
class CameraTilt(CenterOfRotation):
def compute_angle(
self,
img_1: np.ndarray,
img_2: np.ndarray,
method="1d-correlation", # New argument
roi_yxhw=None,
median_filt_shape=None,
padding_mode=None,
peak_fit_radius=1,
high_pass=None,
low_pass=None,
):
```
Where `method` is one out of:
* `"1d-correlation"`: current behavior, with 1D FFT-based horizontal correlation.
* `"fft-polar"`: correlation over the polar warp transform of the 2D-FFT transforms of the two images (at 0 and 180 degrees). A second 2D-FFT correlation will be used for finding the CoR on the rotated images, to honor the current interface.Release 2021.1.0Nicola ViganoNicola Viganohttps://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/116Fix 2282021-03-16T14:08:33+01:00Pierre PaleoFix 228## About
This MR fixes #228.
## To do
- [x] Lift limitation `delta_z > chunk_size`
- [x] End-to-end reconstruction test, notably with "full radios"## About
This MR fixes #228.
## To do
- [x] Lift limitation `delta_z > chunk_size`
- [x] End-to-end reconstruction test, notably with "full radios"Release 2021.1.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/102Rotate projections2021-03-16T12:22:25+01:00Pierre PaleoRotate projections## About
This MR adds the ability to rotate the projection images in the full-field pipeline. It also provides a general-purpose image rotation in the library.
Close #210
## To do
- [x] Implement `Rotation`
- [x] Implement `Cuda...## About
This MR adds the ability to rotate the projection images in the full-field pipeline. It also provides a general-purpose image rotation in the library.
Close #210
## To do
- [x] Implement `Rotation`
- [x] Implement `CudaRotation`
- [x] Unit tests
- [x] Integrate in nabu config and process config
- [x] Integrate in `FullFieldPipeline`
- [x] Integrate in `CudaFullFieldPipeline`
- [x] Integrate in `CudaFullFieldPipelineLimitedMemory`
- [x] End-to-end reconstruction test
## Notes
- The python-numpy backend will require `scikit-image`, as its rotation function is both faster and more flexible (rotation center) than the one of `scipy`.
- It would have been ideal to use `arrayfire` for the cuda backend, but its rotation function does not allow for choosing the center.Release 2021.1.0https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/100Sinogram based rings removal2020-12-17T12:15:02+01:00Pierre PaleoSinogram based rings removal## About
This MR aims at integrating sinogram-based rings removal into the full-field pipeline.
Close #95
## To do
- [x] Re-work `Deringer` and `CudaDeringer` API
- [x] Unit tests
- [x] Integrate in nabu config
- [x] Integrat...## About
This MR aims at integrating sinogram-based rings removal into the full-field pipeline.
Close #95
## To do
- [x] Re-work `Deringer` and `CudaDeringer` API
- [x] Unit tests
- [x] Integrate in nabu config
- [x] Integrate in `FullFieldPipeline`
- [x] Integrate in `CudaFullFieldPipeline`
- [x] Integrate in `CudaFullFieldPipelineLimitedMemory`
- [x] End-to-end reconstruction test
## Notes
The cuda backend needs [pycudwt](https://pypi.org/project/pycudwt).https://gitlab.esrf.fr/tomotools/nabu/-/merge_requests/80Histogram: cuda backend2020-10-27T12:00:24+01:00Pierre PaleoHistogram: cuda backend## Description
This MR adds a cuda backend for `nabu.misc.histogram` (`PartialHistogram`).
Computing a (sub)volume histogram with numpy can be very slow (20 mins on 16 GB of data), so a cuda implementation will be beneficial even with...## Description
This MR adds a cuda backend for `nabu.misc.histogram` (`PartialHistogram`).
Computing a (sub)volume histogram with numpy can be very slow (20 mins on 16 GB of data), so a cuda implementation will be beneficial even with Host-to-Device / Device-to-Host copies.
Close #172.
## To do
- [x] Implement `CudaPartialHistogram`
- [x] Unit tests
- [x] Integrate in full-field pipeline
- [x] Test on "big"
## Notes
The "fixed_bins_width" is not implemented in this backend.Release 2020.5.0