Collect and process data: hangs when a CBF file does not appear

https://requests.esrf.fr/browse/DPC-339?focusedCommentId=113361&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-113361

2024-11-08 01:14:31,165 INFO  [672d57bef1c2ecb8c565b65b] [MXPressA] [<EwoksPythonActor> Collect and process data] started
2024-11-08 01:14:31,167 INFO  [672d57bef1c2ecb8c565b65b] pypushflow concurrency: process
2024-11-08 01:14:31,168 INFO  [672d57bef1c2ecb8c565b65b] pypushflow process pool context: 'NoneType'
2024-11-08 01:14:31,180 INFO  [672d57bef1c2ecb8c565b65b] [job '672d57bef1c2ecb8c565b65b'] [workflow 'MXPressA'] [node 'Collect and process data'] [task 'bes.actors.execute_mesh_startDataCollection.run'] task started
...
2024-11-08 01:16:02,351 DEBUG [672d57bef1c2ecb8c565b65b] Waiting for path: /data/visitor/fx72/id30a1/20241106/RAW_DATA/STAG1/STAG1-CD042391_G07-2_BX009A-02/run_01_MXPressA/run_01_02_mesh/mesh-STAG1-CD042391_G07-2_BX009A-02_1_2_2024.cbf
2024-11-08 01:16:02,399 INFO  [672d57bef1c2ecb8c565b65b] Waiting for file /data/visitor/fx72/id30a1/20241106/RAW_DATA/STAG1/STAG1-CD042391_G07-2_BX009A-02/run_01_MXPressA/run_01_02_mesh/mesh-STAG1-CD042391_G07-2_BX009A-02_1_2_2024.cbf
2024-11-08 01:16:02,403 DEBUG [672d57bef1c2ecb8c565b65b] ExecDozor batch size: 100
2024-11-08 01:16:02,439 DEBUG [672d57bef1c2ecb8c565b65b] Wait file timeOut set to 300 s
2024-11-08 01:16:02,494 INFO  [672d57bef1c2ecb8c565b65b] Waiting for file /data/visitor/fx72/id30a1/20241106/RAW_DATA/STAG1/STAG1-CD042391_G07-2_BX009A-02/run_01_MXPressA/run_01_02_mesh/mesh-STAG1-CD042391_G07-2_BX009A-02_1_2_2024.cbf
2024-11-08 01:16:02,583 INFO  [672d57bef1c2ecb8c565b65b] Reading header from image /data/visitor/fx72/id30a1/20241106/RAW_DATA/STAG1/STAG1-CD042391_G07-2_BX009A-02/run_01_MXPressA/run_01_02_mesh/mesh-STAG1-CD042391_G07-2_BX009A-02_1_2_1901.cbf
2024-11-08 01:21:02,946 WARNING [672d57bef1c2ecb8c565b65b] Timeout while waiting for file /data/visitor/fx72/id30a1/20241106/RAW_DATA/STAG1/STAG1-CD042391_G07-2_BX009A-02/run_01_MXPressA/run_01_02_mesh/mesh-STAG1-CD042391_G07-2_BX009A-02_1_2_2024.cbf
2024-11-08 01:21:03,983 ERROR [672d57bef1c2ecb8c565b65b] Time-out while waiting for image /data/visitor/fx72/id30a1/20241106/RAW_DATA/STAG1/STAG1-CD042391_G07-2_BX009A-02/run_01_MXPressA/run_01_02_mesh/mesh-STAG1-CD042391_G07-2_BX009A-02_1_2_2024.cbf
2024-11-08 01:21:03,984 DEBUG [672d57bef1c2ecb8c565b65b] Done waiting for path: /data/visitor/fx72/id30a1/20241106/RAW_DATA/STAG1/STAG1-CD042391_G07-2_BX009A-02/run_01_MXPressA/run_01_02_mesh/mesh-STAG1-CD042391_G07-2_BX009A-02_1_2_2024.cbf
#
# !!! no other messages from 672d57bef1c2ecb8c565b65b
#
2024-11-08 06:10:44,021 INFO  [672d9d51f1c2ecb8c565b65c] [job '672d9d51f1c2ecb8c565b65c'] job started
2024-11-08 06:10:44,022 INFO  [672d9d51f1c2ecb8c565b65c] [MXPressA] 

A normal workflow

2024-11-08 09:24:09,224 INFO  [672dca82f1c2ecb8c565b686] [MXPressA] [<EwoksPythonActor> Collect and process data] started
2024-11-08 09:24:09,226 INFO  [672dca82f1c2ecb8c565b686] pypushflow concurrency: process
2024-11-08 09:24:09,227 INFO  [672dca82f1c2ecb8c565b686] pypushflow process pool context: 'NoneType'
2024-11-08 09:24:09,239 INFO  [672dca82f1c2ecb8c565b686] [job '672dca82f1c2ecb8c565b686'] [workflow 'MXPressA'] [node 'Collect and process data'] [task 'bes.actors.execute_mesh_startDataCollection.run'] task started
...
2024-11-08 09:25:01,074 DEBUG [672dca82f1c2ecb8c565b686] Waiting for path: /data/visitor/fx72/id30a1/20241106/RAW_DATA/PI3K/PI3K-CD042467_E09-3_BX013A-08/run_01_MXPressA/run_01_02_mesh/mesh-PI3K-CD042467_E09-3_BX013A-08_1_2_0700.cbf
2024-11-08 09:25:01,105 INFO  [672dca82f1c2ecb8c565b686] Waiting for file /data/visitor/fx72/id30a1/20241106/RAW_DATA/PI3K/PI3K-CD042467_E09-3_BX013A-08/run_01_MXPressA/run_01_02_mesh/mesh-PI3K-CD042467_E09-3_BX013A-08_1_2_0700.cbf
2024-11-08 09:25:01,113 DEBUG [672dca82f1c2ecb8c565b686] ExecDozor batch size: 100
2024-11-08 09:25:01,135 DEBUG [672dca82f1c2ecb8c565b686] Wait file timeOut set to 300 s
2024-11-08 09:25:01,173 INFO  [672dca82f1c2ecb8c565b686] Waiting for file /data/visitor/fx72/id30a1/20241106/RAW_DATA/PI3K/PI3K-CD042467_E09-3_BX013A-08/run_01_MXPressA/run_01_02_mesh/mesh-PI3K-CD042467_E09-3_BX013A-08_1_2_0700.cbf
2024-11-08 09:25:01,177 INFO  [672dca82f1c2ecb8c565b686] Reading header from image /data/visitor/fx72/id30a1/20241106/RAW_DATA/PI3K/PI3K-CD042467_E09-3_BX013A-08/run_01_MXPressA/run_01_02_mesh/mesh-PI3K-CD042467_E09-3_BX013A-08_1_2_0501.cbf
2024-11-08 09:25:03,271 DEBUG [672dca82f1c2ecb8c565b686] Done waiting for path: /data/visitor/fx72/id30a1/20241106/RAW_DATA/PI3K/PI3K-CD042467_E09-3_BX013A-08/run_01_MXPressA/run_01_02_mesh/mesh-PI3K-CD042467_E09-3_BX013A-08_1_2_0700.cbf
...
2024-11-08 09:26:05,327 INFO  [672dca82f1c2ecb8c565b686] [job '672dca82f1c2ecb8c565b686'] [workflow 'MXPressA'] [node 'Collect and process data'] [task 'bes.actors.execute_mesh_startDataCollection.run'] task finished
2024-11-08 09:26:05,328 INFO  [672dca82f1c2ecb8c565b686] [MXPressA] [<EwoksPythonActor> Collect and process data] finished

MXPressA.log.7.gz