Broken pipe unrecoverable
From time to time on id21 (mostly experiment changes), we notice that the metaexp_id21 device is in state "UNKNOWN" and cannot start new experiments. Restarting fixes this but its not clear how it ends up in this state.
The log indicates broken pipe which i guess is the connection to the active mq server? Should the metadata devices auto-reconnect to these if the connection is dropped?
1599636229 [140609350272768] INFO id21/metaexp/session_sxm Creating new investigation: blc12343
1599636229 [140609350272768] ERROR id21/metaexp/cyril Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599636229 [140609350272768] ERROR id21/metaexp/perceval Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599636229 [140609350272768] ERROR id21/metaexp/session_dcm Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599636229 [140609350272768] ERROR id21/metaexp/session_sxm Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599636229 [140609350272768] ERROR id21/metaexp/session_temp Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599636229 [140609350272768] ERROR id21/metaexp/wout Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645348 [140609350272768] INFO id21/metaexp/session_sxm Creating new investigation: blc12343
1599645348 [140609350272768] ERROR id21/metaexp/cyril Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645348 [140609350272768] ERROR id21/metaexp/perceval Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645348 [140609350272768] ERROR id21/metaexp/session_dcm Error sending message, trying again: Could nterminate called after throwing an instance of 'Tango::DevFailed'
ot send to connection [[Errno 32] Broken pipe] beamline=id21
1599645348 [140609350272768] ERROR id21/metaexp/session_sxm Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645348 [140609350272768] ERROR id21/metaexp/session_temp Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645348 [140609350272768] ERROR id21/metaexp/wout Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645793 [140609350272768] INFO id21/metaexp/session_sxm Creating new investigation: blc12343
1599645793 [140609350272768] ERROR id21/metaexp/cyril Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645793 [140609350272768] ERROR id21/metaexp/perceval Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645793 [140609350272768] ERROR id21/metaexp/session_dcm Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645793 [140609350272768] ERROR id21/metaexp/session_sxm Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645793 [140609350272768] ERROR id21/metaexp/session_temp Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
1599645793 [140609350272768] ERROR id21/metaexp/wout Error sending message, trying again: Could not send to connection [[Errno 32] Broken pipe] beamline=id21
@denolf we saw this before a few weeks back i think.