Redis connection closed by server due to buffer limits
I get this exception when listing to a scan node for a while during a timescan:
ERROR:nexus_writer_service.session_writer: [nexus_writer_config (RUNNING)] [46_timescan (FAULT)] Stop writer due to exception:
Traceback (most recent call last):
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/nexus_writer_service/scan_writers/writer_base.py", line 844, in _run
self._listen_scan_events()
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/nexus_writer_service/scan_writers/writer_base.py", line 876, in _listen_scan_events
for event_type, node in iterator.walk_events():
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/bliss/data/node.py", line 332, in walk_events
for event_type, event_data in self.wait_for_event(pubsub, filter=filter):
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/bliss/data/node.py", line 355, in wait_for_event
msg = pubsub.get_message()
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/client.py", line 3297, in get_message
response = self.parse_response(block=False, timeout=timeout)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/client.py", line 3185, in parse_response
response = self._execute(conn, conn.read_response)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/client.py", line 3159, in _execute
return command(*args, **kwargs)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 700, in read_response
response = self._parser.read_response()
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 350, in read_response
response = [self.read_response() for i in xrange(length)]
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 350, in <listcomp>
response = [self.read_response() for i in xrange(length)]
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 344, in read_response
response = self._buffer.read(length)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 223, in read
self._read_from_socket(length - self.length)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 187, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
I don't see much on the server log except this info message
INFO 2019-12-11 13:52:11,182 beacon.redis: 121984:M 11 Dec 2019 13:52:11.182 # Client id=2593 addr=/tmp/bliss_testenv_grxkrl7l/redis_xz70rs.sock:0 fd=108 name= age=689 idle=2 flags=PU db=1 sub=0 psub=3 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=502 omem=10293008 events=rw cmd=psubscribe scheduled to be closed ASAP for overcoming of output buffer limits.
I guess this has to do with https://redis.io/topics/clients. @matias.guijarro how should I handle this in the writer? Should I expect it to happen and try to connect again until some timeout?