For my own reference: you can read from multiple processing with file locking enabled. So HDF5 file locking is not just locking the file, it is locking the file for writing. So this is actually a shared lock with flock rather than an exclusive lock.
Wout De Nolf (83e9b07b) at 22 Feb 11:57
https://gitlab.esrf.fr/workflow/ewoksapps/ewoksfluo/-/jobs/944224
while pytest src/ewoksfluo/tests/tasks/test_fit_tasks.py::test_fit_tasks_widget -vx --full-trace;do echo ok;done
Yes the main process only has output.h5
open and the subprocesses open file{filenr}.h5
and some subprocesses open the same file
-> results = executor.map(sub_main, queues, filenames, groups, chunksize=1)
(Pdb) filenames
('file0.h5', 'file0.h5', 'file1.h5', 'file1.h5')
(Pdb) groups
('grp0', 'grp1', 'grp0', 'grp1')
[141035-SPAWNED] HDF5_USE_FILE_LOCKING=TRUE
[141035-SPAWNED] OPEN file0.h5
[141036-SPAWNED] HDF5_USE_FILE_LOCKING=TRUE
[141036-SPAWNED] OPEN file0.h5 >>> this file should be locked by process 141035
[141037-SPAWNED] HDF5_USE_FILE_LOCKING=TRUE
[141037-SPAWNED] OPEN file1.h5
[141038-SPAWNED] HDF5_USE_FILE_LOCKING=TRUE
[141038-SPAWNED] OPEN file1.h5
[141035-SPAWNED] file0.h5 CLOSED
[141036-SPAWNED] file0.h5 CLOSED
[141037-SPAWNED] file1.h5 CLOSED
[141038-SPAWNED] file1.h5 CLOSED
I think there's an issue with your sample code: the "file{filenr}.h5"
are closed when you start the subprocesses that read from them, only the "output.h5"
file is opened but no subprocess access it.
Wout De Nolf (83e9b07b) at 20 Feb 10:45
read without locking in pymca fit subprocess
I had a look here but no luck: https://github.com/h5py/h5py/blob/master/h5py/_objects.pyx
I did see with print_reg
that there are no objects when spawning. Still weird that the spawned subprocess can lock the file locked by the parent process.
Wout De Nolf (c6587693) at 15 Feb 18:50
read without locking in pymca fit subprocess
Wout De Nolf (0345dc43) at 15 Feb 18:47
read without locking in pymca fit subprocess
@tvincent Is there a way in h5py to clear all global state and re-import it?
I'm opening the same file without locking in a subprocess and I need to spawn the subprocess instead of fork. Otherwise the subprocess occasionally hangs when calling h5py.File
(not crash, actually hang).
Wout De Nolf (e25986f3) at 15 Feb 17:16
read without locking in pymca fit subprocess
Wout De Nolf (83814a8c) at 15 Feb 16:28
read without locking in pymca fit subprocess
Wout De Nolf (1461be72) at 15 Feb 16:07
Wout De Nolf (893332c0) at 15 Feb 16:07
Merge branch 'fix_nxdata_order' into 'main'
... and 2 more commits
Wout De Nolf (1461be72) at 15 Feb 15:46
demo: concentrations and regrid with interpolation
... and 1 more commit
Here is what I had in mind for #20: a simple task to pick a single element from a list.
Example usecase: we now expose both detector_names
and detector_name
in the example_xrf_data
task. With this task, we would only need the expose detector_names
and use the task to pick a single detector name:
Closing this since the API of tasks was heavily reworked in the mean time to avoid resorting to a task like this