Foresee to load data from memory
The entry point to get data is the ChunkReader
class. This class expects
- A list of silx
DataUrl
which can be paths to EDF files, HDF5 entries, etc. - A sub-region for chunk reading
For "on-line data processing", it would be good to foresee that data is not on disk. Possible scenarios:
- Data is retrieved from a memcached server
- Data is available in-memory as a numpy array
- Data is available in-memory as a raw C pointer with relevant information
- Data is available in GPU memory
Cases (1) and (2) should be foreseen in the near future. Case (3) falls back to (2) with numpy/ctypes. Case (4) is not clear for now.
There are several options:
- Extend silx
DataUrl
to support these cases - Extend the
ChunkReader
class - For case (1), use another class to make explicit that data is retrieved from a TCP server.
Edited by Pierre Paleo