safe hdf5 group iteration

@wright, @sole, @payno, @andy.gotz, @favre

I'm trying to implement some useful utilities for concurrent h5py in silx (h5py2.py will become silx/io/h5py.py):

https://github.com/silx-kit/silx/issues/3354

I tried to derive classes from h5py to incorporate catch-retry but I gave up. These classes are not derive friendly. In addition, closing a file also invalidates all Dataset and Group instances, even if you open the file again.

So I will probably just derive from h5py.File and some decorators: a generic "retry" and "top_level_groups" which yields all top level groups in a safe way.

However the errors I'm getting are different from before (run test.sh and see test.log):

top_level_groups(filename) is a context manager which yields all Group objects of FINISHED scans (with internal catch-retry). I would think that these groups are safe to access by the reader but this doesn't seem to be the case.

INFO:root:[1123420] writing scan 282.1
INFO:root:[1123420] finished writing scan 282.1
INFO:root:[1123420] writing scan 283.1
INFO:root:[1123421] reading scan /125.1
INFO:root:[1123420] finished writing scan 283.1
INFO:root:[1123420] writing scan 284.1
INFO:root:[1123421] reading scan /126.1
INFO:root:[1123420] finished writing scan 284.1
INFO:root:[1123420] writing scan 285.1
INFO:root:[1123421] reading scan /127.1
ERROR:root:[1123421] <HDF5 group "/127.1" (3 members)>
INFO:root:[1123420] finished writing scan 285.1
INFO:root:[1123420] writing scan 286.1
INFO:root:[1123420] finished writing scan 286.1
INFO:root:[1123420] writing scan 287.1
INFO:root:[1123420] finished writing scan 287.1
INFO:root:[1123420] writing scan 288.1
INFO:root:[1123420] finished writing scan 288.1
INFO:root:[1123420] writing scan 289.1
INFO:root:[1123420] finished writing scan 289.1
INFO:root:[1123420] writing scan 290.1
INFO:root:[1123420] finished writing scan 290.1
INFO:root:[1123420] writing scan 291.1
INFO:root:[1123420] finished writing scan 291.1
INFO:root:[1123420] writing scan 292.1
INFO:root:[1123420] finished writing scan 292.1
INFO:root:[1123420] writing scan 293.1
INFO:root:[1123420] finished writing scan 293.1
INFO:root:[1123420] writing scan 294.1
INFO:root:[1123420] finished writing scan 294.1
INFO:root:[1123420] writing scan 295.1
INFO:root:[1123420] finished writing scan 295.1
INFO:root:[1123420] writing scan 296.1
Traceback (most recent call last):
  File "/home/denolf/dev/concurrent_h5py/h5py2.py", line 132, in wrapper
    yield next(result)
  File "safe_open_hdf5.py", line 54, in reader
    process_scan(scan)
  File "safe_open_hdf5.py", line 42, in process_scan
    for k in measurement:
  File "/users/denolf/virtualenvs/bliss/ubuntu_20_04/lib/python3.7/site-packages/h5py/_hl/group.py", line 409, in __iter__
    for x in self.id.__iter__():
  File "h5py/h5g.pyx", line 472, in h5py.h5g.GroupID.__iter__
  File "h5py/h5g.pyx", line 473, in h5py.h5g.GroupID.__iter__
  File "h5py/h5g.pyx", line 100, in h5py.h5g.GroupIter.__init__
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5g.pyx", line 331, in h5py.h5g.GroupID.get_num_objs
RuntimeError: Can't determine (addr overflow, addr = 5704337, size = 328, eoa = 5703249)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "safe_open_hdf5.py", line 109, in <module>
    reader("test.h5")
  File "safe_open_hdf5.py", line 57, in reader
    raise
  File "/users/denolf/virtualenvs/bliss/ubuntu_20_04/lib/python3.7/contextlib.py", line 161, in __exit__
    raise RuntimeError("generator didn't stop after throw()")
RuntimeError: generator didn't stop after throw()
INFO:root:[1123420] finished writing scan 296.1
INFO:root:[1123420] writing scan 297.1
INFO:root:[1123420] finished writing scan 297.1
INFO:root:[1123420] writing scan 298.1

@sole I thought that once I got the Group (after catch-retry) we could safely use that group recursively. Could you check? Maybe I'm missing something obvious but I can't seems to find it. h5py.h5g.GroupID.get_num_objs should only access the group itself ...