nexus_writer: Redis is loading the dataset in memory
Hi @denolf , on ID22 we experienced some errors with nexus_writer, it did get the following exception, starting it manually (from Jive the start command) then fixed:
INFO 2020-03-02 08:20:47 nexus_writer_service.subscribers.session_writer: [optics-1 (ON)] Session writer started with options {'flat': True, 'multivalue_positioners': False, 'allow_external_nonhdf5': False, 'allow_external_hdf5': True, 'copy_non_external': False, 'stack_mcas': False, 'resource_profiling': False, 'configurable': True}
Ready to accept request
INFO 2020-03-02 08:20:47 nexus_writer_service.subscribers.session_writer: [exp-0 (ON)] No scan writers to stop
INFO 2020-03-02 08:20:47 nexus_writer_service.subscribers.session_writer: [exp-0 (OFF)] Stop listening to Redis events
ERROR 2020-03-02 08:20:47 nexus_writer_service.subscribers.session_writer: [exp-0 (FAULT)] Redis is loading the dataset in memory
ERROR 2020-03-02 08:20:47 nexus_writer_service.subscribers.session_writer: [exp-0 (FAULT)] Stop listening due to exception:
Traceback (most recent call last):
File "/users/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 234, in _node_iterator
return self._greenlet._iterator
AttributeError: 'Greenlet' object has no attribute '_iterator'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/users/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 223, in node
return self._greenlet._node
AttributeError: 'Greenlet' object has no attribute '_node'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/users/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 317, in __greenlet_main
self._listen_event_loop()
File "/users/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 341, in _listen_event_loop
stream_stop_reading_handler=self._stop_handler
File "/users/blissadm/local/bliss.git/nexus_writer_service/subscribers/session_writer.py", line 176, in _walk_events
yield from self._node_iterator.walk_on_new_events(
File "/users/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 236, in _node_iterator
it = self._greenlet._iterator = self.node.iterator
File "/users/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 225, in node
node = self._greenlet._node = get_node(self.node_type, self.db_name)
File "/users/blissadm/local/bliss.git/nexus_writer_service/subscribers/base_subscriber.py", line 56, in get_node
node = _get_node(db_name)
File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 108, in get_node
return get_nodes(db_name, connection=connection)[0]
File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 114, in get_nodes
connection = client.get_redis_connection(db=1)
File "/users/blissadm/local/bliss.git/bliss/config/conductor/client.py", line 44, in f
return func(*args, **keys)
File "/users/blissadm/local/bliss.git/bliss/config/conductor/client.py", line 105, in get_redis_connection
return connection.get_redis_connection(db=db)
File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 402, in get_redis_connection
conn.client_setname(my_name)
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/client.py", line 1183, in client_setname
return self.execute_command('CLIENT SETNAME', name)
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/client.py", line 878, in execute_command
return self.parse_response(conn, command_name, **options)
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/client.py", line 892, in parse_response
response = connection.read_response()
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/connection.py", line 734, in read_response
response = self._parser.read_response()
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/connection.py", line 333, in read_response
raise error
redis.exceptions.BusyLoadingError: Redis is loading the dataset in memory
This is on diffract22.esrf.fr, file /var/log/nexuswriter_1.log