Pool cleanup
Even though the pipeline passes, I sometimes see this
Exception ignored in: <function Pool.__del__ at 0x7f8d2a85cb80>
Traceback (most recent call last):
File "/users/denolf/.pyenv/ubuntu_20_04/versions/3.8.5/lib/python3.8/multiprocessing/pool.py", line 268, in __del__
self._change_notifier.put(None)
File "/users/denolf/.pyenv/ubuntu_20_04/versions/3.8.5/lib/python3.8/multiprocessing/queues.py", line 368, in put
self._writer.send_bytes(obj)
File "/users/denolf/.pyenv/ubuntu_20_04/versions/3.8.5/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/users/denolf/.pyenv/ubuntu_20_04/versions/3.8.5/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/users/denolf/.pyenv/ubuntu_20_04/versions/3.8.5/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
OSError: [Errno 9] Bad file descriptor
It disappears when destroying the pool explicitly:
class AsyncFactory:
def call(self, *args, **kwargs):
asyncresult = self.pool.apply_async(...)
self.pool = None # this makes to issue go away
Seems to work but I'm not sure whether it is safe to do so.
@svensson In general the logic does not allow for proper cleanup. We don't do anything with the asyncresult
handler and since pool destruction seems to work, there must be some resources somewhere to which we have no reference too, leaving to door open for resource leaks.