Add support for sparse VDS in scan files
In a scan file, we have a VDS to aggregate Lima's dense dataset (written over multiple files), likewise we need a way to aggregate a sparse data written over multiple files.
Sparse data are made of three 1D datasets:
-
frame_ptr
: the index of the first pixel in theindex
dataset starting from 0 in each subfile (should be 64 bit integer to accommodate long scan) -
index
: the index of the pixel in the 2D image (32 bit integer) -
intensity
: the intensity of the pixel in the 2D image (depends on the dtype)
In the scan master files
-
frame_ptr
would be rewritten to have a continuous numbering -
index
: is a VDS that concatenate the 1D dataset of each individual data-file -
intensity
: is a VDS that concatenate the 1D dataset of each individual data-file
All the datasets are prefixed with the name of the channel (e.g. eiger2_4m_sparse_frame_ptr
, eiger2_4m_sparse_index
, eiger2_4m_sparse_intensity
)
Here is a snippet of code which does the rewriting of the frame_ptr
from J. Kieffer:
cfpt = numpy.zeros(0, dtype=int)
offset = 0
for f in fptrs:
cfpt = numpy.concatenate((cfpt, f[:-1] + offset))
offset += f[-1]
cfpt = numpy.concatenate((cfpt, [offset]))
The recompaction of frame_ptr
for 100 files of 10000 frames each takes ~48ms on my computer...