Support full-radio processing up to sinograms generation
The current Full-Field pipeline implementations process the data by slabs of projections: each FullFieldPipeline
object handles a sub-volume radios[:, start_z:end_z, :]
.
However, in some settings, it is necessary to process the full radio (ex. CTF phase retrieval with a filter spanning the whole image). In this case, a high-level view of the pipeline consists in these main steps:
- Processing steps on full radios
- Build the sinograms from the radios
- Processing steps on stack of sinograms
Here are two possible solutions.
Solution 1: using two classes
A class FullRadiosPipeline
processes groups of radios. Ex: radios[100:200, :, :]
, radios[200:300, :, :]
, etc.
Another class SinosStackProcessing
processes stacks of sinograms. Ex: radios[:, 100:200, :] == sinos[100:200, :, :]
.
The main difficulty is how the data is passed from one class to the other, especially for datasets not fitting in memory (which is usually the case).
Therefore, a group of radios processed by a FullRadiosPipeline
instance is either dumped in a file (nabu_processes.h5
, temporary file) or kept in memory.
Once all the radios are processed (either by sequentially instantiating FullRadiosPipeline
objects, or by distributing them), the class SinosStackProcessing
is used to reconstruct slabs of slices.
This approach offers the advantage of easily allowing the workload distribution of both the radios processing and the sinos processing. There is one synchronization point (waiting that all the radios are processed to build the sinograms) but it cannot be avoided.
Solution 2: using one class
An alternative could be to write one specialized pipeline dealing with this use case, in one single class, like in CudaFullFieldPipelineLimitedMemory
.
While it might be simpler to implement (debatable), it would not easily allow to distribute the workload.
Remarks
I would advocate for solution 1 as it seems to be more flexible.
The main technical difficulty will be how we save the processed radios:
- If the configuration changes, new radios must be saved. Should it overwrite the old radios to save space ?
- Otherwise, existing radios must be loaded from an existing file (if any)
- Importantly, writing to files should be avoided when possible. For "small" datasets fitting in memory, only one
FullRadiosProcessing
instance can be used, and thenSinosStackProcessing
would use in-memory radios.