Compressed datasets can be very slow to fetch
I don't know if it comes from h5grove, h5web or a combination of the two.
This needs further investigation. It can be probably reproduced outside of Hibou by using a dataset with the same metadata:
Dataset | |
---|---|
Type | Integer (unsigned), 32-bit, little-endian |
Shape | 2000 x 2167 x 2070 = 8971380000 |
Chunk shape | 1 x 2167 x 2070 = 4485690 |
Compression filters | |
---|---|
32008 | bitshuffle; see https://github.com/kiyo-masui/bitshuffle |
Also observed in jupyterlab-h5web
Edited by Loic Huder