Nexus writer: finalize HDF5 one bad scan improve traceback
I saw this in ID10 logs. They were restarting things so not a real issue but the
INFO 2023-06-12 07:28:16,969 nexus_writer_service.subscribers.session_writer: [cdi-2 (RUNNING)] [1070_dscan-2 (ON)] [axis (ENABLED)] Scan marked as SUCCEEDED in HDF5
INFO 2023-06-12 07:28:16,970 nexus_writer_service.subscribers.session_writer: [cdi-2 (RUNNING)] [1070_dscan-2 (ON)] Finished writing to '/data/visitor/mi1396/id10/20230608/RAW_DATA/cogd1dot_az2/cogd1dot_az2_0010/cogd1dot_az2_0010.h5' (121pts-121pts 2.2GB 0:02:23.032555)
INFO 2023-06-12 07:28:16,970 nexus_writer_service.subscribers.session_writer: [cdi-2 (RUNNING)] [1070_dscan-2 (ON)] Stop listening to Redis events
INFO 2023-06-12 07:28:16,982 nexus_writer_service.subscribers.session_writer: [cdi-2 (RUNNING)] [1070_dscan-2 (OFF)] Finished succesfully
INFO 2023-06-12 07:28:16,982 nexus_writer_service.subscribers.session_writer: [cdi-2 (ON)] [1070_dscan-2 (OFF)] Greenlet exits
INFO 2023-06-12 09:25:09,714 nexus_writer_service.subscribers.session_writer: [cdi-0 (ON)] No scan writers to stop
INFO 2023-06-12 09:25:09,714 nexus_writer_service.subscribers.session_writer: [cdi-0 (ON)] 0 scan writers (0 active)
INFO 2023-06-12 09:25:09,715 nexus_writer_service.subscribers.session_writer: [cdi-0 (ON)] Stop listening to Redis events
ERROR 2023-06-12 09:25:09,715 nexus_writer_service.subscribers.session_writer: [cdi-0 (FAULT)] GreenletExit
WARN 2023-06-12 09:25:09,715 nexus_writer_service.subscribers.session_writer: [cdi-0 (FAULT)] Stop listening to Redis events (greenlet killed)
INFO 2023-06-12 09:25:09,715 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (ON)] Finalize writing to /data/visitor/mi1396/id10/20230608/RAW_DATA/CoGd1/CoGd1_0002/CoGd1_0002.h5
INFO 2023-06-12 09:25:09,716 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (ON)] Fetch last data
ERROR 2023-06-12 09:25:09,716 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)]
ERROR 2023-06-12 09:25:09,716 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)] Captured finalization exception
Traceback (most recent call last):
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 364, in _capture_finalize_exceptions
yield
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 324, in _finalize_hdf5
self._fetch_data(node, last=True)
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 1551, in _fetch_data
nproxy = self._node_proxy(node)
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 1334, in _node_proxy
return self._dataset_proxy(node, full_init=full_init)
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 1352, in _dataset_proxy
dproxy = subscan.datasets.get(node.fullname)
File "/home/blissadm/local/bliss.git/blissdata/blissdata/data/nodes/channel.py", line 158, in fullname
return self.info.get("fullname")
File "/home/blissadm/local/bliss.git/blissdata/blissdata/settings.py", line 89, in _read
value = func(self, *args, **keys)
File "/home/blissadm/local/bliss.git/blissdata/blissdata/settings.py", line 924, in get
v = super().raw_get(key)
File "/home/blissadm/local/bliss.git/blissdata/blissdata/settings.py", line 592, in raw_get
return cnx.hget(self.name, *keys)
File "/users/blissadm/conda/miniconda/envs/bliss_1_11/lib/python3.9/site-packages/redis/client.py", line 3010, in hget
return self.execute_command('HGET', name, key)
File "/users/blissadm/conda/miniconda/envs/bliss_1_11/lib/python3.9/site-packages/redis/client.py", line 901, in execute_command
return self.parse_response(conn, command_name, **options)
File "/users/blissadm/conda/miniconda/envs/bliss_1_11/lib/python3.9/site-packages/redis/client.py", line 915, in parse_response
response = connection.read_response()
File "/users/blissadm/conda/miniconda/envs/bliss_1_11/lib/python3.9/site-packages/redis/connection.py", line 730, in read_response
response = self._parser.read_response()
File "/users/blissadm/conda/miniconda/envs/bliss_1_11/lib/python3.9/site-packages/redis/connection.py", line 464, in read_response
self.read_from_socket()
File "/users/blissadm/conda/miniconda/envs/bliss_1_11/lib/python3.9/site-packages/redis/connection.py", line 421, in read_from_socket
bufflen = recv_into(self._sock, self._buffer)
File "/users/blissadm/conda/miniconda/envs/bliss_1_11/lib/python3.9/site-packages/redis/_compat.py", line 75, in recv_into
return sock.recv_into(*args, **kwargs)
File "/users/blissadm/conda/miniconda/envs/bliss_1_11/lib/python3.9/site-packages/gevent/_socketcommon.py", line 693, in recv_into
self._wait(self._read_event)
File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
File "src/gevent/_hub_primitives.py", line 304, in gevent._gevent_c_hub_primitives._primitive_wait
File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
greenlet.GreenletExit
INFO 2023-06-12 09:25:09,743 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)] Save the last data
INFO 2023-06-12 09:25:09,751 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)] Save dataset metadata
INFO 2023-06-12 09:25:09,759 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)] Save device metadata
ERROR 2023-06-12 09:25:09,759 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)] Captured finalization exception
Traceback (most recent call last):
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 440, in _listen_event_loop
for event_type, node, event_data in self._walk_events(
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 474, in _walk_events
yield from self.node.walk_events(**kwargs)
File "/home/blissadm/local/bliss.git/blissdata/blissdata/data/node.py", line 1085, in walk_events
yield from self._iter_reader(
File "/home/blissadm/local/bliss.git/blissdata/blissdata/data/node.py", line 1147, in _iter_reader
for stream, events in reader:
File "/home/blissadm/local/bliss.git/blissdata/blissdata/streaming.py", line 807, in __iter__
for item in self._queue:
File "src/gevent/queue.py", line 412, in gevent._gevent_cqueue.Queue.__next__
File "src/gevent/queue.py", line 350, in gevent._gevent_cqueue.Queue.get
File "src/gevent/queue.py", line 327, in gevent._gevent_cqueue.Queue._Queue__get_or_peek
File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
greenlet.GreenletExit
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 364, in _capture_finalize_exceptions
yield
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 286, in _event_loop_finalize
self._finalize_hdf5()
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 347, in _finalize_hdf5
self._fetch_device_metadata(subscan)
File "/home/blissadm/local/bliss.git/nexus_writer_service/subscribers/scan_writer_base.py", line 1736, in _fetch_device_metadata
channel_info = channels.get(fullname)
AttributeError: 'NoneType' object has no attribute 'get'
INFO 2023-06-12 09:25:09,761 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)] Finished writing to '/data/visitor/mi1396/id10/20230608/RAW_DATA/CoGd1/CoGd1_0002/CoGd1_0002.h5' (9pts-11pts 165.0MB 3 days, 16:42:58.119106)
INFO 2023-06-12 09:25:09,761 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)] Stop listening to Redis events
WARN 2023-06-12 09:25:09,958 nexus_writer_service.subscribers.session_writer: [cdi-1 (FAULT)] [39_dscan-1 (FAULT)] Stop listening to Redis events (greenlet killed)
setting BLISS environment
Using BLISS bliss_1_11
This has moved to ScanWriter._save_device_metadata
but the improvement is the same.