Flint: Stucked for a while
Here is an issue happening the 2021-02-10 at ID15A.
- Opening the colormap dialog (of the image view) make Flint stuck for about 5 or 10 minutes.
- Flint was fully stucked
- One BLISS process was about 100%
- No extra memory consumption
- Nothing in the remote logs
- We have enabled
SCAN_DISPLAY.flint_output_enabled = True
(there was nothing displayed) - I send 3 times an interruption
kill -USR1
to reach logs, nothing - At one point Flint was responding again
Here is few logs finally displayed
In the BLISS shell:
Displayed when FLint was not anymore stucked.
Here we lose part of it.
In BLISS remote logs:
2021-02-09 19:28:10,882 eh3 user_input INFO : SCAN_DISPLAY.flint_output_enabled=True
2021-02-09 19:29:11,554 eh3 flint.output ERROR : --- Logging error ---
2021-02-09 19:29:11,555 eh3 flint.output ERROR : Traceback (most recent call last):
2021-02-09 19:29:11,557 eh3 flint.output ERROR :
2021-02-09 19:29:11,558 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/logging/__init__.py", line 1029, in emit
self.flush()
2021-02-09 19:29:11,559 eh3 flint.output ERROR :
2021-02-09 19:29:11,560 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/logging/__init__.py", line 1009, in flush
self.stream.flush()
2021-02-09 19:29:11,562 eh3 flint.output ERROR :
2021-02-09 19:29:11,563 eh3 flint.output ERROR : BrokenPipeError: [Errno 32] Broken pipe
2021-02-09 19:29:11,564 eh3 flint.output ERROR :
2021-02-09 19:29:11,566 eh3 flint.output ERROR : Call stack:
2021-02-09 19:29:11,605 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
2021-02-09 19:29:11,606 eh3 flint.output ERROR :
2021-02-09 19:29:11,606 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
2021-02-09 19:29:11,607 eh3 flint.output ERROR :
2021-02-09 19:29:11,607 eh3 flint.output ERROR : File "/users/blissadm/local/bliss.git/bliss/flint/__main__.py", line 10, in <module>
main()
2021-02-09 19:29:11,607 eh3 flint.output ERROR :
2021-02-09 19:29:11,608 eh3 flint.output ERROR : File "/users/blissadm/local/bliss.git/bliss/flint/flint.py", line 495, in main
sys.exit(qapp.exec_())
2021-02-09 19:29:11,608 eh3 flint.output ERROR :
2021-02-09 19:29:11,608 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/silx/gui/plot/backends/BackendBase.py", line 330, in postRedisplay
def postRedisplay(self):
2021-02-09 19:29:11,608 eh3 flint.output ERROR :
2021-02-09 19:29:11,609 eh3 flint.output ERROR : File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 118, in triggered
_logger.info("Interrupted by %s", signum)
2021-02-09 19:29:11,609 eh3 flint.output ERROR :
2021-02-09 19:29:11,609 eh3 flint.output ERROR : Message: 'Interrupted by %s'
Arguments: (10,)
2021-02-09 19:29:11,634 eh3 flint.output ERROR : --- Logging error ---
2021-02-09 19:29:11,634 eh3 flint.output ERROR : Traceback (most recent call last):
2021-02-09 19:29:11,635 eh3 flint.output ERROR :
2021-02-09 19:29:11,635 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/logging/__init__.py", line 1029, in emit
self.flush()
2021-02-09 19:29:11,635 eh3 flint.output ERROR :
2021-02-09 19:29:11,636 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/logging/__init__.py", line 1009, in flush
self.stream.flush()
2021-02-09 19:29:11,636 eh3 flint.output ERROR :
2021-02-09 19:29:11,636 eh3 flint.output ERROR : BrokenPipeError: [Errno 32] Broken pipe
2021-02-09 19:29:11,636 eh3 flint.output ERROR :
2021-02-09 19:29:11,637 eh3 flint.output ERROR : Call stack:
2021-02-09 19:29:11,637 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
2021-02-09 19:29:11,637 eh3 flint.output ERROR :
2021-02-09 19:29:11,637 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
2021-02-09 19:29:11,638 eh3 flint.output ERROR :
2021-02-09 19:29:11,638 eh3 flint.output ERROR : File "/users/blissadm/local/bliss.git/bliss/flint/__main__.py", line 10, in <module>
main()
2021-02-09 19:29:11,638 eh3 flint.output ERROR :
2021-02-09 19:29:11,638 eh3 flint.output ERROR : File "/users/blissadm/local/bliss.git/bliss/flint/flint.py", line 495, in main
sys.exit(qapp.exec_())
2021-02-09 19:29:11,639 eh3 flint.output ERROR :
2021-02-09 19:29:11,639 eh3 flint.output ERROR : File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/silx/gui/plot/backends/BackendBase.py", line 330, in postRedisplay
def postRedisplay(self):
2021-02-09 19:29:11,639 eh3 flint.output ERROR :
2021-02-09 19:29:11,640 eh3 flint.output ERROR : File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 120, in triggered
self.initMonitoring()
2021-02-09 19:29:11,640 eh3 flint.output ERROR :
2021-02-09 19:29:11,640 eh3 flint.output ERROR : File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 79, in initMonitoring
_logger.info("Start monitoring")
2021-02-09 19:29:11,640 eh3 flint.output ERROR :
2021-02-09 19:29:11,641 eh3 flint.output ERROR : Message: 'Start monitoring'
Arguments: ()
2021-02-09 19:30:09,550 eh3 user_input INFO : SCAN_DISPLAY.flint_output_enabled=False
In Flint remote log file:
2021-02-09 19:29:11,521 eh3 6833 bliss.flint.manager.watchdog INFO : Interrupted by 10
2021-02-09 19:29:11,633 eh3 6833 bliss.flint.manager.watchdog INFO : Start monitoring
2021-02-09 19:30:49,232 eh3 6833 bliss.flint.manager.watchdog INFO : Interrupted by 10
2021-02-09 19:30:49,269 eh3 6833 bliss.flint.manager.watchdog INFO : Log monitoring
2021-02-09 19:30:49,301 eh3 6833 bliss.flint.manager.watchdog INFO : ---------- threads traceback ----------
ThreadID: 139821091772160
File: "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/threadpool.py", line 187, in run
task = task_queue_get(task_queue_cookie)
File: "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_threading.py", line 158, in get
self._not_empty.wait(cookie)
File: "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_threading.py", line 68, in wait
wait_lock.acquire() # Block on the native lock
ThreadID: 139821328946944
File: "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/threadpool.py", line 187, in run
task = task_queue_get(task_queue_cookie)
File: "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_threading.py", line 158, in get
self._not_empty.wait(cookie)
File: "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_threading.py", line 68, in wait
wait_lock.acquire() # Block on the native lock
ThreadID: 139821811357440
File: "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/hub.py", line 647, in run
loop.run()
ThreadID: 139821893007104
File: "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/hub.py", line 647, in run
loop.run()
File: "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 123, in triggered
self.logMonitoring()
File: "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 86, in logMonitoring
log_threads_traceback()
File: "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 35, in log_threads_traceback
for filename, lineno, name, line in traceback.extract_stack(stack):
2021-02-09 19:30:49,389 eh3 6833 bliss.flint.manager.watchdog INFO : ---------- greenlet traceback ----------
********************************************************************************
* Threads
********************************************************************************
Thread 0x7f2aa281d700 (idle threadpool worker)
********************************************************************************
Thread 0x7f2ab0a4d700 (idle threadpool worker)
********************************************************************************
Thread 0x7f2acd65d700 (None)
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/hub.py", line 647, in run
loop.run()
********************************************************************************
Thread 0x7f2ad243b700 (MainThread) (CURRENT)
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/hub.py", line 647, in run
loop.run()
File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 123, in triggered
self.logMonitoring()
File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 87, in logMonitoring
log_greenlets_traceback()
File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 49, in log_greenlets_traceback
log += gevent.util.format_run_info()
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 152, in format_run_info
_format_thread_info(lines, thread_stacks, limit, current_thread_ident)
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 189, in _format_thread_info
lines.append(''.join(traceback.format_stack(frame, limit)))
********************************************************************************
* Greenlets
********************************************************************************
---- Thread boundary
<greenlet.greenlet object at 0x7f2ad0f5db90>
: Parent: None
: Running:
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 193, in _run_module_as_main
: "__main__", mod_spec)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 85, in _run_code
: exec(code, run_globals)
: File "/users/blissadm/local/bliss.git/bliss/flint/__main__.py", line 10, in <module>
: main()
: File "/users/blissadm/local/bliss.git/bliss/flint/flint.py", line 495, in main
: sys.exit(qapp.exec_())
: File "/users/blissadm/local/bliss.git/bliss/flint/flint.py", line 307, in process_gevent
: gevent.sleep(0.01)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/hub.py", line 166, in sleep
: hub.wait(t)
+--- <Greenlet "Greenlet-0" at 0x7f2ab0ed85f0: <bound method FlintServer._task of <bliss.flint.helper.rpc_server.FlintServer object at 0x7f2ab0e5c0d0>>(<bliss.flint.flint_api.FlintApi object at 0x7f2ab3, <gevent._gevent_cevent.AsyncResult object at 0x7f2)>
: Parent: <Hub '' at 0x7f2ac1ed4940 epoll default pending=0 ref=5 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f2ab10c3e90 pool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>> threadpool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>> thread_ident=0x7f2ad243b700>
: Running:
: File "/users/blissadm/local/bliss.git/bliss/flint/helper/rpc_server.py", line 60, in _task
: gevent.wait([stop, task], count=1)
: Spawned at:
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 193, in _run_module_as_main
: "__main__", mod_spec)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 85, in _run_code
: exec(code, run_globals)
: File "/users/blissadm/local/bliss.git/bliss/flint/__main__.py", line 10, in <module>
: main()
: File "/users/blissadm/local/bliss.git/bliss/flint/flint.py", line 489, in main
: server = FlintServer(flintModel.flintApi())
: File "/users/blissadm/local/bliss.git/bliss/flint/helper/rpc_server.py", line 54, in __init__
: self.thread = gevent.spawn(self._task, flintApi, self.stop)
| +--- <Greenlet "Greenlet-2" at 0x7f2ab0ed8830: <bound method _ServerObject.run of <bliss.comm.rpc._ServerObject object at 0x7f2ab3a5d210>>>
: Parent: <Hub '' at 0x7f2ac1ed4940 epoll default pending=0 ref=5 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f2ab10c3e90 pool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>> threadpool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>> thread_ident=0x7f2ad243b700>
: Running:
: File "/users/blissadm/local/bliss.git/bliss/comm/rpc.py", line 282, in run
: new_client, addr = server_socket.accept()
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_socket3.py", line 217, in accept
: self._wait(self._read_event)
: Spawned at:
: File "/users/blissadm/local/bliss.git/bliss/flint/helper/rpc_server.py", line 58, in _task
: with safe_rpc_server(flint) as (task, url):
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/contextlib.py", line 112, in __enter__
: return next(self.gen)
: File "/users/blissadm/local/bliss.git/bliss/flint/helper/rpc_server.py", line 32, in safe_rpc_server
: task = gevent.spawn(server.run)
| | +--- <Greenlet "Greenlet-4" at 0x7f2ab0198b90: <bound method _ServerObject._client_poll of <bliss.comm.rpc._ServerObject object at 0x7f2ab3a5d210>>(<gevent._socket3.socket at 0x7f2aa284b7c0 object, )>
: Parent: <Hub '' at 0x7f2ac1ed4940 epoll default pending=0 ref=5 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f2ab10c3e90 pool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>> threadpool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>> thread_ident=0x7f2ad243b700>
: Running:
: File "/users/blissadm/local/bliss.git/bliss/comm/rpc.py", line 334, in _client_poll
: msg = client_sock.recv(READ_BUFFER_SIZE)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_socketcommon.py", line 663, in recv
: self._wait(self._read_event)
: Spawned at:
: File "/users/blissadm/local/bliss.git/bliss/comm/rpc.py", line 295, in run
: self._clients.append(gevent.spawn(self._client_poll, new_client))
| +--- <Greenlet "BeaconListenTask" at 0x7f2ab0ed8950: <bound method Connection._raw_read of <bliss.config.conductor.connection.Connection object at 0x7f2ab0adba50>>>
: Parent: <Hub '' at 0x7f2ac1ed4940 epoll default pending=0 ref=5 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f2ab10c3e90 pool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>> threadpool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>> thread_ident=0x7f2ad243b700>
: Running:
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 734, in _raw_read
: self.__raw_read()
: File "/users/blissadm/local/bliss.git/bliss/common/greenlet_utils.py", line 69, in func
: return fu(*args, **kwargs)
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 745, in __raw_read
: raw_data = self._socket.recv(16 * 1024)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_socketcommon.py", line 663, in recv
: self._wait(self._read_event)
: Spawned at:
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/contextlib.py", line 112, in __enter__
: return next(self.gen)
: File "/users/blissadm/local/bliss.git/bliss/flint/helper/rpc_server.py", line 45, in maintain_value
: redis = get_redis_proxy()
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/client.py", line 46, in f
: return func(*args, **keys)
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/client.py", line 121, in get_redis_proxy
: return connection.get_redis_proxy(db=db, caching=caching, shared=shared)
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 487, in get_redis_proxy
: return self._get_shared_redis_proxy(proxyid)
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 460, in _get_shared_redis_proxy
: pool = self._get_redis_conn_pool(proxyid)
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 429, in _get_redis_conn_pool
: pool = self._create_redis_conn_pool(poolid)
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 438, in _create_redis_conn_pool
: address = self.get_redis_connection_address()
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 66, in f
: self.connect()
: File "/users/blissadm/local/bliss.git/bliss/config/conductor/connection.py", line 262, in connect
: self._raw_read_task = gevent.spawn(self._raw_read)
+--- <Greenlet "Greenlet-1" at 0x7f2ab0198ef0: watch_session_scans('eh3', <bound method ScanManager.new_scan of <bliss.flint, <bound method ScanManager.new_scan_child of <bliss, <bound method ScanManager.new_scan_data of <bliss., <bound method ScanManager.end_scan of <bliss.flint, ready_event=<gevent._gevent_cevent.Event object at 0x7f2ab013f, watch_scan_group=True, exclude_existing_scans=True)>
: Parent: <Hub '' at 0x7f2ac1ed4940 epoll default pending=0 ref=5 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f2ab10c3e90 pool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>> threadpool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>> thread_ident=0x7f2ad243b700>
: Running:
: File "/users/blissadm/local/bliss.git/bliss/data/scan.py", line 128, in watch_session_scans
: started_event=ready_event,
: File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 1145, in walk_on_new_events
: yield from self.walk_events(first_index=streaming.DataStream.now_index(), **kw)
: File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 1136, in walk_events
: started_event=started_event,
: File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 1190, in _iter_reader
: for stream, events in reader:
: File "/users/blissadm/local/bliss.git/bliss/config/streaming.py", line 772, in __iter__
: for item in self._queue:
: Spawned at:
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 193, in _run_module_as_main
: "__main__", mod_spec)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/runpy.py", line 85, in _run_code
: exec(code, run_globals)
: File "/users/blissadm/local/bliss.git/bliss/flint/__main__.py", line 10, in <module>
: main()
: File "/users/blissadm/local/bliss.git/bliss/flint/flint.py", line 495, in main
: sys.exit(qapp.exec_())
: File "/users/blissadm/local/bliss.git/bliss/flint/flint.py", line 492, in <lambda>
: qt.QTimer.singleShot(10, lambda: start_flint(flintModel, options, splash))
: File "/users/blissadm/local/bliss.git/bliss/flint/flint.py", line 168, in start_flint
: result = manager.updateBlissSessionName(options.bliss_session)
: File "/users/blissadm/local/bliss.git/bliss/flint/manager/manager.py", line 222, in updateBlissSessionName
: flintModel.setBlissSessionName(sessionName)
: File "/users/blissadm/local/bliss.git/bliss/flint/model/flint_model.py", line 151, in setBlissSessionName
: self.blissSessionChanged.emit()
: File "/users/blissadm/local/bliss.git/bliss/flint/manager/scan_manager.py", line 145, in __bliss_session_changed
: self._spawn_scans_session_watch(session_name)
: File "/users/blissadm/local/bliss.git/bliss/flint/manager/scan_manager.py", line 171, in _spawn_scans_session_watch
: exclude_existing_scans=True,
| +--- <Greenlet "Greenlet-5" at 0x7f2ab0198a70: <bound method DataStreamReader._read_task_main of <bliss.config.streaming.DataStreamReader object at 0x7f2ab0adba90>>>
: Parent: <Hub '' at 0x7f2ac1ed4940 epoll default pending=0 ref=5 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f2ab10c3e90 pool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>> threadpool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>> thread_ident=0x7f2ad243b700>
: Running:
: File "/users/blissadm/local/bliss.git/bliss/config/streaming.py", line 667, in _read_task_main
: lst = self._read_active_streams()
: File "/users/blissadm/local/bliss.git/bliss/config/streaming.py", line 608, in _read_active_streams
: streams_to_read, count=self._count, block=self._block
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/client.py", line 2573, in xread
: return self.execute_command('XREAD', *pieces)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/client.py", line 901, in execute_command
: return self.parse_response(conn, command_name, **options)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/client.py", line 915, in parse_response
: response = connection.read_response()
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/connection.py", line 730, in read_response
: response = self._parser.read_response()
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/connection.py", line 464, in read_response
: self.read_from_socket()
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/connection.py", line 421, in read_from_socket
: bufflen = recv_into(self._sock, self._buffer)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/redis/_compat.py", line 75, in recv_into
: return sock.recv_into(*args, **kwargs)
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_socketcommon.py", line 693, in recv_into
: self._wait(self._read_event)
: Spawned at:
: File "/users/blissadm/local/bliss.git/bliss/data/scan.py", line 128, in watch_session_scans
: started_event=ready_event,
: File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 1145, in walk_on_new_events
: yield from self.walk_events(first_index=streaming.DataStream.now_index(), **kw)
: File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 1136, in walk_events
: started_event=started_event,
: File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 1186, in _iter_reader
: yield_events=yield_events,
: File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 1495, in _subscribe_all_streams
: "children_list", reader, create=True, first_index=first_index
: File "/users/blissadm/local/bliss.git/bliss/data/node.py", line 1324, in _subscribe_stream
: reader.add_streams(stream, node=self, first_index=first_index, **kw)
: File "/users/blissadm/local/bliss.git/bliss/config/streaming.py", line 501, in add_streams
: self._streams[stream.name] = sinfo
: File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/contextlib.py", line 119, in __exit__
: next(self.gen)
: File "/users/blissadm/local/bliss.git/bliss/config/streaming.py", line 462, in _update_streams_context
: self._start_read_task()
: File "/users/blissadm/local/bliss.git/bliss/config/streaming.py", line 390, in _start_read_task
: self._read_task = gevent.spawn(self._read_task_main)
+--- <Hub '' at 0x7f2ac1ed4940 epoll default pending=0 ref=5 fileno=4 resolver=<gevent.resolver.thread.Resolver at 0x7f2ab10c3e90 pool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>> threadpool=<ThreadPool at 0x7f2ab0a5e830 tasks=0 size=1 maxsize=10 hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>> thread_ident=0x7f2ad243b700>
Parent: <greenlet.greenlet object at 0x7f2ad0f5db90>
Running:
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/hub.py", line 647, in run
loop.run()
File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 123, in triggered
self.logMonitoring()
File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 87, in logMonitoring
log_greenlets_traceback()
File "/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py", line 49, in log_greenlets_traceback
log += gevent.util.format_run_info()
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 153, in format_run_info
_format_greenlet_info(lines, greenlet_stacks, limit)
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 214, in _format_greenlet_info
'running_stack_limit': limit,
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 361, in format_lines
for l in self._render(tree)]
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 434, in _render
self.__render_children(tree)
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 454, in __render_children
child_tree = child._render(tree.deeper())
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 418, in _render
tree.details['running_stack_limit'])
File "/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py", line 377, in __render_tb
tb = ''.join(traceback.format_stack(frame, limit))
---- Thread boundary
<ThreadPoolWorker at 0x7f2aa2bc3a10 thread_ident=0x7f2aa281d700 threadpool-hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>
: Parent: <greenlet.greenlet object at 0x7f2aa2bc3ad0>
---- Thread boundary
<ThreadPoolWorker at 0x7f2ab0af0410 thread_ident=0x7f2ab0a4d700 threadpool-hub=<Hub at 0x7f2ac1ed4940 thread_ident=0x7f2ad243b700>>
: Parent: <greenlet.greenlet object at 0x7f2ab0af04d0>
---- Thread boundary
<greenlet.greenlet object at 0x7f2ab3b84b90>
: Parent: None
+--- <Hub '' at 0x7f2ab3bc1ef0 epoll pending=0 ref=1 fileno=22 thread_ident=0x7f2acd65d700>
Parent: <greenlet.greenlet object at 0x7f2ab3b84b90>
2021-02-09 19:30:49,475 eh3 6833 bliss.flint.manager.watchdog INFO : ---------- tracemalloc ----------
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/silx/gui/plot/backends/BackendOpenGL.py:896: size=9675 KiB (+9675 KiB), count=4 (+4), average=2419 KiB
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/linecache.py:137: size=1253 KiB (+1253 KiB), count=12588 (+12588), average=102 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:224: size=14.4 KiB (+14.4 KiB), count=213 (+213), average=69 B
<frozen importlib._bootstrap>:219: size=13.4 KiB (+13.4 KiB), count=93 (+93), average=147 B
<frozen importlib._bootstrap_external>:525: size=10.2 KiB (+10.2 KiB), count=119 (+119), average=88 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/silx/gui/colors.py:1109: size=4720 B (+4720 B), count=25 (+25), average=189 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/local.py:363: size=2705 B (+2705 B), count=14 (+14), average=193 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/local.py:254: size=2300 B (+2300 B), count=13 (+13), average=177 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:398: size=2274 B (+2274 B), count=8 (+8), average=284 B
/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py:36: size=2094 B (+2094 B), count=12 (+12), average=174 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:434: size=2080 B (+2080 B), count=4 (+4), average=520 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/local.py:235: size=2024 B (+2024 B), count=10 (+10), average=202 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/local.py:275: size=1918 B (+1918 B), count=9 (+9), average=213 B
/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py:41: size=1892 B (+1892 B), count=1 (+1), average=1892 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/local.py:215: size=1850 B (+1850 B), count=8 (+8), average=231 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/linecache.py:175: size=1752 B (+1752 B), count=11 (+11), average=159 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/local.py:212: size=1747 B (+1747 B), count=10 (+10), average=175 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:454: size=1464 B (+1464 B), count=3 (+3), average=488 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/louie/dispatcher.py:292: size=1440 B (+1440 B), count=20 (+20), average=72 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:164: size=1344 B (+1344 B), count=3 (+3), average=448 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/threadpool.py:361: size=1256 B (+1256 B), count=3 (+3), average=419 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/silx/gui/plot/backends/BackendOpenGL.py:1398: size=1080 B (+1080 B), count=15 (+15), average=72 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:334: size=1008 B (+1008 B), count=7 (+7), average=144 B
/users/blissadm/local/bliss.git/bliss/flint/manager/watchdog.py:38: size=941 B (+941 B), count=11 (+11), average=86 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:207: size=880 B (+880 B), count=2 (+2), average=440 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/silx/gui/plot/PlotWidget.py:2808: size=840 B (+840 B), count=5 (+5), average=168 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/weakref.py:409: size=824 B (+824 B), count=7 (+7), average=118 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/OpenGL/wrapper.py:678: size=816 B (+816 B), count=10 (+10), average=82 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/OpenGL/wrapper.py:677: size=792 B (+792 B), count=9 (+9), average=88 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/OpenGL/wrapper.py:867: size=768 B (+768 B), count=6 (+6), average=128 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:357: size=672 B (+672 B), count=4 (+4), average=168 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/util.py:533: size=640 B (+640 B), count=2 (+2), average=320 B
/users/blissadm/local/bliss.git/bliss/flint/widgets/log_widget.py:191: size=608 B (+608 B), count=4 (+4), average=152 B
/users/blissadm/local/bliss.git/bliss/flint/widgets/log_widget.py:190: size=608 B (+608 B), count=4 (+4), average=152 B
/users/blissadm/local/bliss.git/bliss/flint/widgets/log_widget.py:186: size=608 B (+608 B), count=4 (+4), average=152 B
/users/blissadm/local/bliss.git/bliss/flint/widgets/log_widget.py:185: size=608 B (+608 B), count=4 (+4), average=152 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/silx/gui/plot/backends/BackendOpenGL.py:114: size=592 B (+592 B), count=2 (+2), average=296 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/silx/gui/plot/backends/glutils/GLPlotCurve.py:1126: size=576 B (+576 B), count=1 (+1), average=576 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/hub.py:647: size=576 B (+576 B), count=1 (+1), average=576 B
/users/blissadm/conda/miniconda/envs/bliss_dev/lib/python3.7/site-packages/gevent/_util.py:154: size=576 B (+576 B), count=1 (+1), average=576 B
Edited by Valentin Valls