Bliss and gevent monitoring
In implementing the Tango server for the external writer, I notice that the server blocks sometimes. I suspect something is blocking the async loop so I wanted to debug it with gevent's monitoring thread. The PeriodicMonitoringThread thread crashes however (see below).
Now my question: I noticed Bliss patches all but threads monkey.patch_all(thread=False)
. Why? Does this mean that something is using the native threading library? Could this be an issue for gevent monitoring (which I assume is using gevent's threading)? Has anyone ever tried to debug blocking async loops with gevent? Any suggestions are welcome, I'm on shaky grounds here.
Traceback (most recent call last):
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/gevent/_monitor.py", line 202, in __call__
f(hub)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/gevent/_monitor.py", line 235, in monitor_blocking
dict(greenlet_stacks=False, current_thread_ident=self.monitor_thread_ident))
File "src/gevent/_tracer.py", line 103, in gevent.__tracer.GreenletTracer.did_block_hub_report
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/nexus_writer_service/scan_writers/writer_base.py", line 142, in __str__
return "{} ({})".format(self.scan_node.name, self.state.name)
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/bliss/common/greenlet_utils.py", line 70, in func
return fu(*args, **kwargs)
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/bliss/data/node.py", line 487, in name
return self._struct.name
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/bliss/config/settings.py", line 1170, in __getattribute__
return self._proxy.get(name)
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/bliss/config/settings.py", line 101, in _read
value = func(self, *args, **keys)
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/bliss/config/settings.py", line 947, in get
v = super().raw_get(key)
File "/mntdirect/_data_id21_inhouse/wout/dev/blissmain/bliss/config/settings.py", line 599, in raw_get
return cnx.hget(self.name, *keys)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/client.py", line 2713, in hget
return self.execute_command('HGET', name, key)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/client.py", line 839, in execute_command
return self.parse_response(conn, command_name, **options)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/client.py", line 853, in parse_response
response = connection.read_response()
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 699, in read_response
response = self._parser.read_response()
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 309, in read_response
response = self._buffer.readline()
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 241, in readline
self._read_from_socket()
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/connection.py", line 183, in _read_from_socket
data = recv(self._sock, socket_read_size)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/redis/_compat.py", line 58, in recv
return sock.recv(*args, **kwargs)
File "/data/id21/inhouse/wout/dev/virtualenvs/rnice8/bliss/py37/lib/python3.7/site-packages/gevent/_socket3.py", line 382, in recv
self._wait(self._read_event)
File "src/gevent/_hub_primitives.py", line 284, in gevent.__hub_primitives.wait_on_socket
File "src/gevent/_hub_primitives.py", line 289, in gevent.__hub_primitives.wait_on_socket
File "src/gevent/_hub_primitives.py", line 271, in gevent.__hub_primitives._primitive_wait
File "src/gevent/_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
File "src/gevent/_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
File "src/gevent/_hub_primitives.py", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
File "src/gevent/_waiter.py", line 151, in gevent.__waiter.Waiter.get
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
greenlet.error: cannot switch to a different thread
2019-12-05T20:57:19Z
<PeriodicMonitoringThread at 0x7f60ed394b90 in thread 0x7f60e1dcd700 greenlet <greenlet.greenlet object at 0x7f60e1e3e520> for <Hub '' at 0x7f60e1e96f50 epoll default pending=0 ref=10 fileno=12 resolver=<gevent.resolver_thread.Resolver at 0x7f60d86135d0 pool=<ThreadPool at 0x7f60ed8d04d0 0/1/10 hub=<Hub at 0x7f60e1e96f50 thread_ident=0x140054369560320>>> threadpool=<ThreadPool at 0x7f60ed8d04d0 0/1/10 hub=<Hub at 0x7f60e1e96f50 thread_ident=0x140054369560320>> thread_ident=0x7f60f2f26700>> failed with error