HDF5: metadata slows down writing because of big dict
Dumping a large dictionary with silx.io.dictdump.dicttoh5
is very costly because it creates one HDF5 dataset for every key.
The culprit is mostly the processing_options["read_chunk"]["files"]
dictionary which is a dict
with one key per projection.
Solutions:
- Short term: Don't write metadata for reconstruction subvolumes (it was never needed needed so far). Write metadata only for the final master file.
- Later: Convert the
files
to another data structure, while preserving the mapping between projection indices and data urls.