integrator issueshttps://gitlab.esrf.fr/tomotools/integrator/-/issues2021-08-27T15:48:16+02:00https://gitlab.esrf.fr/tomotools/integrator/-/issues/5Support sub-volume averaging2021-08-27T15:48:16+02:00Pierre PaleoSupport sub-volume averagingIn `intrange`, when the parameter `avexy` is set to True, the current stack is averaged along "z" axis and the resulting image undergoes the AI. So, we end up with one curve instead of `N_z`.
```python
aveim = np.average(fil[self.arrayp...In `intrange`, when the parameter `avexy` is set to True, the current stack is averaged along "z" axis and the resulting image undergoes the AI. So, we end up with one curve instead of `N_z`.
```python
aveim = np.average(fil[self.arraypath][()], axis=0)
xye = self.ai.integrate1d_legacy(
aveim, self.args.npt, unit=self.args.unit,
polarization_factor=1.0,
correctSolidAngle=False,
azimuth_range=self.azims[w],
error_model=self.args.error,
method=self.aimethod
)
```https://gitlab.esrf.fr/tomotools/integrator/-/issues/6Support detector attenuation2021-08-24T15:49:11+02:00Pierre PaleoSupport detector attenuationCurrently, detector attenuation is not enabled in `intrange`:
- [self.set_detector_atten](https://gitlab.esrf.fr/steche/intrange/-/blob/master/hdintrange.py#L92) updates `self.det_att`, but `self.det_att` is unused (it should be [saved...Currently, detector attenuation is not enabled in `intrange`:
- [self.set_detector_atten](https://gitlab.esrf.fr/steche/intrange/-/blob/master/hdintrange.py#L92) updates `self.det_att`, but `self.det_att` is unused (it should be [saved in the output file](https://gitlab.esrf.fr/steche/intrange/-/blob/master/hdintrange.py#L128) )https://gitlab.esrf.fr/tomotools/integrator/-/issues/14Decompress data on GPU2022-11-17T12:02:40+01:00Pierre PaleoDecompress data on GPUThe data is compressed with bitshuffle/LZ4.
Currently, the processing sequence is roughly
```
1. load raw data (from GPFS usually)
2. decompress on CPU (bitshuffle/LZ4 implementation from hdf5plugin)
3. send full frame on GPU
4. Perfo...The data is compressed with bitshuffle/LZ4.
Currently, the processing sequence is roughly
```
1. load raw data (from GPFS usually)
2. decompress on CPU (bitshuffle/LZ4 implementation from hdf5plugin)
3. send full frame on GPU
4. Perform AI
```
Now if a GPU decompressor is available, the processing sequence becomes
```
1. load raw data (from GPFS usually)
2. send raw data to GPU
3. decompress on GPU
4. Perform AI
```
We benefit from both sending raw data to GPU (10X less data) and (hopefully) decompressing on GPU (5X faster ?).
There are, however, two difficulties.
The first difficulty is to read raw data (instead of transparently-decompressed by hdf5plugin).
This can be done as follows:
```python
for i in range(ds.id.get_num_chunks()):
filter_mask, chunk = ds.id.read_direct_chunk(ds.id.get_chunk_info(i).chunk_offset)
```
The second difficulty is that the above code won't work for virtual datasets.
So the distribution of integration tasks has to be done on the `scan????/pilatus_????.h5` files directly.
But `StackIntegrator` is designed to process data from `start_idx` and `end_idx`, which is incompatible with direct chunk read. Only entry points like `process_full_dataset()` should be used.
To do:
- Add a init parameter: `decompress_on_GPU`
- browse the current dataset: list the virtual sources, for each chunk, get the path to the actual data.
- Add `load_data_raw` (see above snippet)
- Add a `decompress_lz4` (using GPU decompressor)
- Modify `process_full_dataset`
Note that the original `process_stack()` will be incompatible with `decompress_on_GPU=True`https://gitlab.esrf.fr/tomotools/integrator/-/issues/15chmod g+w still not working (?)2022-11-21T10:51:35+01:00Pierre Paleochmod g+w still not working (?)Beamline feedback suggest that "chmod 755" is still not working.
Perhaps it would be good to do
- `umask 0002` at startup
- a chmod after integration:
```python
# https://stackoverflow.com/a/2853934
def recursive_chmod(path, mode):
...Beamline feedback suggest that "chmod 755" is still not working.
Perhaps it would be good to do
- `umask 0002` at startup
- a chmod after integration:
```python
# https://stackoverflow.com/a/2853934
def recursive_chmod(path, mode):
os.chmod(path, mode)
for dirpath, dirnames, filenames in os.walk(path):
os.chmod(dirpath, mode)
for fname in filenames:
os.chmod(os.path.join(dirpath, fname), mode)
```https://gitlab.esrf.fr/tomotools/integrator/-/issues/16trailing slash in output file makes integrator crash2023-02-27T11:30:12+01:00Pierre Paleotrailing slash in output file makes integrator crashin `[output] location`, an extra `/` in the end makes the whole thing crash.in `[output] location`, an extra `/` in the end makes the whole thing crash.