ID23-2 Grenades reprocessing fails
ID23-2 Grenades reprocessing fails as reported by @nanao
Parameters: 30, 2, P212121
Celery worker logs contain the Slurm logs:
[2024-09-05 11:57:53,042: ERROR/MainProcess]
STDOUT: /tmp_14_days/ewoks_opid23/slurm_logs/pyslurmutils.ewoksserver1.17046804.out
------------------------------------------------------------------------------------
python3 is /cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/bin/python3
Python version: 3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:33:10)
[GCC 12.3.0]
working directory: /tmp
Connecting to ewoksserver1.esrf.fr:59088 ...
Receiving function and arguments from ewoksserver1.esrf.fr:59088 ...
Executing function: <function execute_graph at 0x14db5c4718b0>
Arguments: ({'graph': {'id': 'MXDataReductionPipelines_ID23-2', 'label': 'MXDataReductionPipelines (ID23-2)', 'keywords': {'cardinality': '1', 'definition': '*', 'instrumentName': 'ID23-2', 'scanType': '*'}, 'input_schema': {'description': '', 'properties': {'anomalous': {'default': True, 'description': '', 'title': 'Anomalous', 'type': 'boolean'}, 'exclude': {'title': 'Exclude data range', 'description': 'Excluded images will not be processed by XDS.', 'type': 'string'}, 'forced_spacegroup': {'title': 'Forced spacegroup', 'type': 'string'}, 'high_res_limit': {'title': 'High resolution limit', 'type': 'number'}, 'low_res_limit': {'title': 'Low resolution limit', 'type': 'number'}, 'mx_pipeline_name': {'default': ['EDNA_proc'], 'items': {'enum': ['EDNA_proc', 'autoPROC', 'XIA2_DIALS', 'grenades_fastproc'], 'type': 'string'}, 'title': 'Pipeline', 'type': 'array', 'uniqueItems': True}}, 'title': 'Automatic reprocessing of X-Ray diffraction images', 'type': 'object'}, 'ui_schema': {'mx_pipeline_name': {'ui:widget': 'checkboxes'}, 'exclude': {'ui:widget': 'dozorExcludeRange'}, 'ui:order': ['mx_pipeline_name', 'exclude', 'low_res_limit', 'high_res_limit', 'forced_spacegroup', 'anomalous', '*']}, 'execute_arguments': {'_slurm_spawn_arguments': {'parameters': {'current_working_directory': '/tmp_14_days/opid23', 'environment': {'EDNA_SITE': 'ESRF_ID23EH2', 'USER': 'opid23'}, 'partition': 'nice', 'time_limit': 360}, 'pre_script': 'ORG_PATH=$PATH;module load bes_id23eh2;PATH=$PATH'}, 'engine': 'ppf'}, 'worker_options': {'queue': 'id23eh2'}, 'category': 'MX'}, 'nodes': [{'id': 'ewoksmx.tasks.WaitForFinished.WaitForFinished3', 'label': 'grenades_fastproc Wait For Finished', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForFinished.WaitForFinished', 'uiProps': {'icon': 'default.png', 'position': {'x': 1065, 'y': 45}, 'nodeWidth': 249}}, {'id': 'ewoksmx.tasks.WaitForStarted.WaitForStarted3', 'label': 'grenades_fastproc Wait For Started', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForStarted.WaitForStarted', 'uiProps': {'icon': 'default.png', 'position': {'x': 780, 'y': 45}, 'nodeWidth': 233}}, {'id': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit3', 'label': 'grenades_fastproc Slurm Submit', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit', 'uiProps': {'icon': 'default.png', 'position': {'x': 510, 'y': 45}, 'nodeWidth': 218}}, {'id': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'label': 'MXPipelineSynchronize', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize', 'uiProps': {'icon': 'graphOutput.svg', 'position': {'x': 1725, 'y': -120}, 'nodeWidth': 191}}, {'id': 'ewoksmx.tasks.WaitForStarted.WaitForStarted2', 'label': 'XIA2_DIALS Wait For Started', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForStarted.WaitForStarted', 'uiProps': {'icon': 'default.png', 'position': {'x': 780, 'y': -90}, 'nodeWidth': 209}}, {'id': 'ewoksmx.tasks.WaitForFinished.WaitForFinished2', 'label': 'XIA2_DIALS Wait For Finished', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForFinished.WaitForFinished', 'uiProps': {'icon': 'default.png', 'position': {'x': 1065, 'y': -90}, 'nodeWidth': 199}}, {'id': 'ewoksmx.tasks.WaitForFinished.WaitForFinished1', 'label': 'autoPROC Wait For Finished', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForFinished.WaitForFinished', 'uiProps': {'icon': 'default.png', 'position': {'x': 1065, 'y': -210}, 'nodeWidth': 204}}, {'id': 'ewoksmx.tasks.WaitForStarted.WaitForStarted1', 'label': 'autoPROC Wait For Started', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForStarted.WaitForStarted', 'uiProps': {'icon': 'default.png', 'position': {'x': 780, 'y': -210}, 'nodeWidth': 203}}, {'id': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit2', 'label': 'XIA2_DIALS Slurm Submit', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit', 'uiProps': {'icon': 'default.png', 'position': {'x': 510, 'y': -90}, 'nodeWidth': 202}}, {'id': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit1', 'label': 'autoPROC Slurm Submit', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit', 'uiProps': {'icon': 'default.png', 'position': {'x': 510, 'y': -210}, 'nodeWidth': 200}}, {'id': 'ewoksmx.tasks.WaitForFinished.WaitForFinished0', 'label': 'EDNA_proc Wait For Finished', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForFinished.WaitForFinished', 'uiProps': {'icon': 'default.png', 'position': {'x': 1065, 'y': -405}, 'nodeWidth': 222}}, {'id': 'ewoksmx.tasks.WaitForStarted.WaitForStarted0', 'label': 'EDNA_proc Wait For Started\n', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForStarted.WaitForStarted', 'uiProps': {'icon': 'default.png', 'position': {'x': 780, 'y': -405}, 'nodeWidth': 194}}, {'id': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit0', 'label': 'EDNA_proc Slurm Submit', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit', 'uiProps': {'icon': 'default.png', 'position': {'x': 465, 'y': -405}, 'nodeWidth': 210}}, {'id': 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline0', 'label': 'Grenades_fastproc_pipeline', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline', 'uiProps': {'icon': 'default.png', 'position': {'x': 195, 'y': 45}, 'nodeWidth': 198}}, {'id': 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline0', 'label': 'XIA2_DIALS_pipeline', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline', 'uiProps': {'icon': 'default.png', 'position': {'x': 210, 'y': -90}, 'nodeWidth': 176}}, {'id': 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline0', 'label': 'AutoPROC_pipeline', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline', 'inputs_complete': False, 'default_error_node': False, 'uiProps': {'icon': 'default.png', 'position': {'x': 210, 'y': -210}, 'moreHandles': False, 'withImage': True, 'withLabel': True, 'nodeWidth': 165}}, {'id': 'EDNA_proc_pipeline0', 'label': 'EDNA_proc_pipeline', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.EDNA_proc_pipeline.EDNA_proc_pipeline', 'inputs_complete': False, 'default_error_node': False, 'uiProps': {'icon': 'default.png', 'position': {'x': 210, 'y': -405}, 'moreHandles': False, 'withImage': True, 'withLabel': True, 'nodeWidth': 142}}, {'id': 'MXPipelineInput0', 'label': 'MXPipelineInput', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.MXPipelineInput.MXPipelineInput', 'inputs_complete': False, 'default_error_node': False, 'uiProps': {'icon': 'graphInput.svg', 'position': {'x': -540, 'y': -75}, 'moreHandles': False, 'withImage': True, 'withLabel': True, 'nodeWidth': 138}}], 'links': [{'source': 'MXPipelineInput0', 'target': 'EDNA_proc_pipeline0', 'data_mapping': [{'source_output': 'metadata', 'target_input': 'metadata'}, {'source_output': 'EDNA_proc', 'target_input': 'EDNA_proc'}, {'source_output': 'autoPROC', 'target_input': 'autoPROC'}, {'source_output': 'XIA2_DIALS', 'target_input': 'XIA2_DIALS'}], 'conditions': [{'source_output': 'EDNA_proc', 'value': True}], 'on_error': False, 'required': False, 'map_all_data': False, 'uiProps': {'label': 'EDNA_proc: "true"', 'type': 'default', 'animated': False}}, {'source': 'MXPipelineInput0', 'target': 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline0', 'data_mapping': [{'source_output': 'metadata', 'target_input': 'metadata'}, {'source_output': 'EDNA_proc', 'target_input': 'EDNA_proc'}, {'source_output': 'autoPROC', 'target_input': 'autoPROC'}, {'source_output': 'XIA2_DIALS', 'target_input': 'XIA2_DIALS'}], 'conditions': [{'source_output': 'autoPROC', 'value': True}], 'on_error': False, 'required': False, 'map_all_data': False, 'uiProps': {'label': 'autoPROC: "true"', 'type': 'default', 'animated': False}}, {'source': 'MXPipelineInput0', 'target': 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline0', 'data_mapping': [{'source_output': 'metadata', 'target_input': 'metadata'}, {'source_output': 'EDNA_proc', 'target_input': 'EDNA_proc'}, {'source_output': 'autoPROC', 'target_input': 'autoPROC'}, {'source_output': 'XIA2_DIALS', 'target_input': 'XIA2_DIALS'}], 'conditions': [{'source_output': 'XIA2_DIALS', 'value': True}], 'on_error': False, 'map_all_data': False, 'uiProps': {'label': 'XIA2_DIALS: "true"'}}, {'source': 'EDNA_proc_pipeline0', 'target': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit0', 'data_mapping': [{'source_output': 'pipeline_name', 'target_input': 'pipeline_name'}, {'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit0', 'target': 'ewoksmx.tasks.WaitForStarted.WaitForStarted0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForStarted.WaitForStarted0', 'target': 'ewoksmx.tasks.WaitForFinished.WaitForFinished0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline0', 'target': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit1', 'data_mapping': [{'source_output': 'pipeline_name', 'target_input': 'pipeline_name'}, {'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline0', 'target': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit2', 'data_mapping': [{'source_output': 'pipeline_name', 'target_input': 'pipeline_name'}, {'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit1', 'target': 'ewoksmx.tasks.WaitForStarted.WaitForStarted1', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForStarted.WaitForStarted1', 'target': 'ewoksmx.tasks.WaitForFinished.WaitForFinished1', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit2', 'target': 'ewoksmx.tasks.WaitForStarted.WaitForStarted2', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForStarted.WaitForStarted2', 'target': 'ewoksmx.tasks.WaitForFinished.WaitForFinished2', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForFinished.WaitForFinished0', 'target': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForFinished.WaitForFinished1', 'target': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForFinished.WaitForFinished2', 'target': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'MXPipelineInput0', 'target': 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline0', 'data_mapping': [{'source_output': 'metadata', 'target_input': 'metadata'}, {'source_output': 'grenades_fastproc', 'target_input': 'grenades_fastproc'}, {'source_output': 'EDNA_proc', 'target_input': 'EDNA_proc'}, {'source_output': 'autoPROC', 'target_input': 'autoPROC'}, {'source_output': 'XIA2_DIALS', 'target_input': 'XIA2_DIALS'}], 'conditions': [{'source_output': 'grenades_fastproc', 'value': True}], 'on_error': False, 'map_all_data': False, 'uiProps': {'label': 'grenades_fastproc: "true"'}}, {'source': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit3', 'target': 'ewoksmx.tasks.WaitForStarted.WaitForStarted3', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForStarted.WaitForStarted3', 'target': 'ewoksmx.tasks.WaitForFinished.WaitForFinished3', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline0', 'target': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit3', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}, {'source_output': 'pipeline_name', 'target_input': 'pipeline_name'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForFinished.WaitForFinished3', 'target': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}]},)
Keyword arguments: {'engine': 'ppf', 'startargs': {'anomalous': True, 'mx_pipeline_name': ['grenades_fastproc'], 'exclude': '123-513', 'low_res_limit': 50, 'high_res_limit': 2, 'forced_spacegroup': 'P212121', 'datasets': [{'id': 1883932075, 'name': 'XIA2_DIALS [2024-08-31 03:53:10]', 'startDate': '2024-08-31T03:53:08.454+02:00', 'endDate': '2024-08-31T03:53:08.454+02:00', 'location': '/data/visitor/mx2651/id23eh2/20240830/PROCESSED_DATA/Sample-3-1-10/run_02_datacollection/autoprocessing/XIA2_DIALS/XIA2_DIALS', 'investigation': {'id': 1824856659, 'createId': 'ingester', 'createTime': '2024-07-17T00:00:56.195+02:00', 'modId': 'ingester', 'modTime': '2024-08-31T08:00:13.879+02:00', 'dataCollectionInvestigations': [], 'datasets': [], 'doi': '10.15151/ESRF-ES-1824856659', 'endDate': '2024-08-31T08:00:00.000+02:00', 'fileCount': 0, 'fileSize': 0, 'fundingReferences': [], 'investigationFacilityCycles': [], 'investigationGroups': [], 'investigationInstruments': [{'id': 1884303114, 'createId': 'ingester', 'createTime': '2024-08-31T07:34:26.864+02:00', 'modId': 'ingester', 'modTime': '2024-08-31T07:34:26.864+02:00', 'instrument': {'id': 39153691, 'createId': '255362', 'createTime': '2017-06-16T15:35:40.631+02:00', 'modId': 'bodin/265147', 'modTime': '2024-04-22T09:58:10.276+02:00', 'datasetInstruments': [], 'description': 'Structural Biology Microfocus Beamline', 'fullName': 'ID23-2', 'instrumentScientists': [], 'investigationInstruments': [], 'name': 'ID23-2', 'shifts': [], 'type': 'Beamline', 'url': 'https://www.esrf.fr/UsersAndScience/Experiments/MX/About_our_beamlines/ID23-2'}}], 'investigationUsers': [], 'keywords': [], 'name': 'MX-2651', 'parameters': [], 'publications': [], 'releaseDate': '2027-08-31T08:00:00.000+02:00', 'samples': [], 'shifts': [], 'startDate': '2024-08-30T09:30:00.000+02:00', 'studyInvestigations': [], 'summary': 'The Austrian Crystallographic Diffraction Consortium (ACDC) BAG aims at supporting several macromolecular structural biology groups across Austria (in total\r\n20 research groups at 9 institutions in 4 cities). We employ mainly the technique of X-ray crystallography to characterize macromolecular biological complexes,\r\nbut also solution scattering approaches are frequently used to characterize dynamic assemblies. Approval of this BAG will be essential for researchers in all\r\nour groups to carry out their structural studies on various health-related and basic research-oriented topics, which are targeted at characterizing large molecular\r\nassemblies or proteins currently under-represented in the PDB, including membrane proteins. Corresponding high-resolution diffraction data are only obtainable\r\nthrough the use of synchrotron radiation sources. Thus regular synchrotron access is highly desirable to maintain a competitive advantage in our individual\r\nresearch fields.', 'title': 'Austrian Crystallographic Diffraction Consortium', 'type': {'id': 312, 'createId': 'root', 'createTime': '2014-06-18T15:00:43.317+02:00', 'modId': 'root', 'modTime': '2014-06-18T15:00:43.317+02:00', 'description': 'Macromolecular Crystallography', 'investigations': [], 'name': 'MX'}, 'visitId': 'MX/2651 ID23-2 30-08-2024/31-08-2024'}, 'type': 'processed', 'sampleName': 'Sample-3-1-10', 'sampleId': 1883835906, 'parameters': [{'name': 'scanType', 'value': 'integration', 'id': 1883932185, 'units': 'None', 'datasetId': 1883932075}, {'name': 'endDate', 'value': '2024-08-31T03:53:08.454420', 'id': 1883932189, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'Sample_name', 'value': 'Sample-3-1-10', 'id': 1883932184, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'startDate', 'value': '2024-08-31T03:53:08.454416', 'id': 1883932188, 'units': 'NA', 'datasetId': 1883932075}, {'name': '__fileCount', 'value': '15', 'id': 1883932195, 'units': 'NA', 'datasetId': 1883932075}, {'name': '__volume', 'value': '68429313', 'id': 1883932196, 'units': 'Bytes', 'datasetId': 1883932075}, {'name': '__elapsedTime', 'value': '0', 'id': 1883932197, 'units': 's', 'datasetId': 1883932075}, {'name': 'datasetName', 'value': 'XIA2_DIALS', 'id': 1883932193, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'input_datasets', 'value': '/data/visitor/mx2651/id23eh2/20240830/RAW_DATA/Sample-3-1-10/run_02_datacollection', 'id': 1883932187, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'input_datasetIds', 'value': '1883929032', 'id': 1883932190, 'units': 'NA', 'datasetId': 1883932075}, {'name': '__full_input_datasetIds', 'value': '1883929032', 'id': 1883932205, 'units': 'NA', 'datasetId': 1883932075}, {'name': '__full_input_datasetNames', 'value': 'run_02_datacollection', 'id': 1883932206, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_beta', 'value': '101.724707', 'id': 1883932141, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_resolution_limit_high', 'value': '4.827686171697526', 'id': 1883932162, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_completeness', 'value': '98.85474800146523', 'id': 1883932144, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_merge', 'value': '14.308336852215678', 'id': 1883932150, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_completeness', 'value': '97.35746982937994', 'id': 1883932157, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_cc_half', 'value': '0.9563052909054404', 'id': 1883932156, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_merge', 'value': '6.369120272905746', 'id': 1883932163, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_resolution_limit_low', 'value': '1.8094320636908312', 'id': 1883932175, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_cc_half', 'value': '0.2515337148878142', 'id': 1883932183, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_mean_I_over_sigI', 'value': '40.18795385195578', 'id': 1883932168, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_alpha', 'value': '90.0', 'id': 1883932140, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_anomalous_multiplicity', 'value': '1.8523825296388516', 'id': 1883932147, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_resolution_limit_high', 'value': '1.778758417415428', 'id': 1883932176, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_merge', 'value': '74.90074632325022', 'id': 1883932177, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_completeness', 'value': '98.94759921069941', 'id': 1883932170, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_c', 'value': '118.583447', 'id': 1883932137, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_meas_within_IPlus_IMinus', 'value': '105.33081846936821', 'id': 1883932178, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_a', 'value': '64.153227', 'id': 1883932133, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_resolution_limit_low', 'value': '65.64389801025389', 'id': 1883932148, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_meas_all_IPlus_IMinus', 'value': '9.272653445205377', 'id': 1883932165, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_anomalous_multiplicity', 'value': '1.882438805299798', 'id': 1883932160, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_multiplicity', 'value': '3.5830305620859155', 'id': 1883932159, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_mean_I_over_sigI', 'value': '1.0960728236334958', 'id': 1883932182, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_mean_I_over_sigI', 'value': '10.695375365467381', 'id': 1883932155, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_gamma', 'value': '90.0', 'id': 1883932142, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_anomalous_completeness', 'value': '91.60099076784508', 'id': 1883932171, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_resolution_limit_high', 'value': '1.7787593061413856', 'id': 1883932149, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_meas_within_IPlus_IMinus', 'value': '8.993147895562203', 'id': 1883932164, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_resolution_limit_low', 'value': '65.68861198534033', 'id': 1883932161, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_pim_within_IPlus_IMinus', 'value': '6.347248780550601', 'id': 1883932166, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_cc_half', 'value': '0.9730803959988407', 'id': 1883932169, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_anomalous_completeness', 'value': '93.52370266479664', 'id': 1883932145, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_pim_within_IPlus_IMinus', 'value': '14.223198285917874', 'id': 1883932153, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'Process_program', 'value': 'XIA2_DIALS', 'id': 1883932186, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_space_group', 'value': 'P 1 2 1', 'id': 1883932131, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_pim_all_IPlus_IMinus', 'value': '10.240392227105216', 'id': 1883932154, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_meas_all_IPlus_IMinus', 'value': '102.0355441527951', 'id': 1883932179, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_multiplicity', 'value': '3.5351424990463736', 'id': 1883932146, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_anomalous', 'value': '1', 'id': 1883932143, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_pim_within_IPlus_IMinus', 'value': '73.9812144665318', 'id': 1883932180, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_meas_within_IPlus_IMinus', 'value': '20.17998638545335', 'id': 1883932151, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_multiplicity', 'value': '3.516951030356747', 'id': 1883932172, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_anomalous_multiplicity', 'value': '1.8496678708775203', 'id': 1883932174, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_meas_all_IPlus_IMinus', 'value': '19.214985838033652', 'id': 1883932152, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_b', 'value': '65.643865', 'id': 1883932135, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_pim_all_IPlus_IMinus', 'value': '4.857884056242371', 'id': 1883932167, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_pim_all_IPlus_IMinus', 'value': '53.89944846102994', 'id': 1883932181, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_anomalous_completeness', 'value': '98.27946989072309', 'id': 1883932158, 'units': 'NA', 'datasetId': 1883932075}], 'instrumentName': 'ID23-2', 'techniques': [{'id': 1580426249, 'datasetId': 1883932075, 'pid': 'PaNET:PaNET01012', 'name': 'x-ray probe'}], 'meta': {'page': {'totalWithoutFilters': 1, 'total': 1, 'totalPages': 1, 'currentPage': 1}}}], 'callback': 'https://icatplus.esrf.fr/ewoks/394fe3f6-0fae-440b-896e-922c069a374b/jobs?id=66d98095385da400137bdee9', 'raw_data_path': ['/data/visitor/mx2651/id23eh2/20240830/PROCESSED_DATA/Sample-3-1-10/run_02_datacollection/autoprocessing/XIA2_DIALS/XIA2_DIALS']}, 'execinfo': {'handlers': [{'class': 'ewoksjob.events.handlers.RedisEwoksEventHandler', 'arguments': [{'name': 'url', 'value': 'redis://ewoksserver1:25004/0'}, {'name': 'ttl', 'value': 86400}]}], 'job_id': 'ff1afb62-c68b-4c7b-a1f8-a522a155d430'}}
Sending result <class 'tuple'> to ewoksserver1.esrf.fr:59088 ...
STDERR: /tmp_14_days/ewoks_opid23/slurm_logs/pyslurmutils.ewoksserver1.17046804.err
------------------------------------------------------------------------------------
slurmstepd-hpc7-01: error: couldn't chdir to `/tmp_14_days/opid23': No such file or directory: going to /tmp instead
slurmstepd-hpc7-01: error: couldn't chdir to `/tmp_14_days/opid23': No such file or directory: going to /tmp instead
[MXDataReductionPipelines (ID23-2)] [<EwoksPythonActor> MXPipelineInput] Error in python actor 'MXPipelineInput'!
Not running down stream actors ["Conditional actor between 'MXPipelineInput0' and 'EDNA_proc_pipeline0'", "Conditional actor between 'MXPipelineInput0' and 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline0'", "Conditional actor between 'MXPipelineInput0' and 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline0'", "Conditional actor between 'MXPipelineInput0' and 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline0'"]
Exception:Task 'MXPipelineInput' failed
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 401, in execute
self.run()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/ewoksmx/src/ewoksmx/tasks/MXPipelineInput.py", line 48, in run
raise FileNotFoundError(str(metadata_path))
FileNotFoundError: /data/visitor/mx2651/id23eh2/20240830/PROCESSED_DATA/Sample-3-1-10/run_02_datacollection/autoprocessing/XIA2_DIALS/XIA2_DIALS/metadata.json
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/pypushflow/concurrent/interrupt/process.py", line 14, in task_main
return fn(*args, **kwargs)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoksppf/ppfrunscript.py", line 27, in run
task.execute()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 407, in execute
raise RuntimeError(f"Task '{self.label}' failed") from e
RuntimeError: Task 'MXPipelineInput' failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/pypushflow/concurrent/process.py", line 78, in cb
result = future.result()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
RuntimeError: Task 'MXPipelineInput' failed
[MXDataReductionPipelines (ID23-2)] Interrupt workflow execution: Task 'MXPipelineInput' failed
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 401, in execute
self.run()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/ewoksmx/src/ewoksmx/tasks/MXPipelineInput.py", line 48, in run
raise FileNotFoundError(str(metadata_path))
FileNotFoundError: /data/visitor/mx2651/id23eh2/20240830/PROCESSED_DATA/Sample-3-1-10/run_02_datacollection/autoprocessing/XIA2_DIALS/XIA2_DIALS/metadata.json
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/pypushflow/concurrent/interrupt/process.py", line 14, in task_main
return fn(*args, **kwargs)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoksppf/ppfrunscript.py", line 27, in run
task.execute()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 407, in execute
raise RuntimeError(f"Task '{self.label}' failed") from e
RuntimeError: Task 'MXPipelineInput' failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 50, in <module>
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoks/bindings.py", line 78, in execute_graph
result = mod.execute_graph(graph, execinfo=execinfo, **execute_options)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/events/contexts.py", line 25, in wrapper
return method(*args, execinfo=execinfo, **kw)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoksppf/bindings.py", line 599, in execute_graph
return ppfgraph.run(**execute_options)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoksppf/bindings.py", line 564, in run
raise ex
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/pypushflow/concurrent/process.py", line 78, in cb
result = future.result()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
RuntimeError: Task 'MXPipelineInput' failed
[2024-09-05 11:57:53,068: ERROR/MainProcess]
STDOUT: /tmp_14_days/ewoks_opid23/slurm_logs/pyslurmutils.ewoksserver1.17046803.out
------------------------------------------------------------------------------------
python3 is /cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/bin/python3
Python version: 3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:33:10)
[GCC 12.3.0]
working directory: /tmp
Connecting to ewoksserver1.esrf.fr:59049 ...
Receiving function and arguments from ewoksserver1.esrf.fr:59049 ...
Executing function: <function execute_graph at 0x14e89bb688b0>
Arguments: ({'graph': {'id': 'MXDataReductionPipelines_ID23-2', 'label': 'MXDataReductionPipelines (ID23-2)', 'keywords': {'cardinality': '1', 'definition': '*', 'instrumentName': 'ID23-2', 'scanType': '*'}, 'input_schema': {'description': '', 'properties': {'anomalous': {'default': True, 'description': '', 'title': 'Anomalous', 'type': 'boolean'}, 'exclude': {'title': 'Exclude data range', 'description': 'Excluded images will not be processed by XDS.', 'type': 'string'}, 'forced_spacegroup': {'title': 'Forced spacegroup', 'type': 'string'}, 'high_res_limit': {'title': 'High resolution limit', 'type': 'number'}, 'low_res_limit': {'title': 'Low resolution limit', 'type': 'number'}, 'mx_pipeline_name': {'default': ['EDNA_proc'], 'items': {'enum': ['EDNA_proc', 'autoPROC', 'XIA2_DIALS', 'grenades_fastproc'], 'type': 'string'}, 'title': 'Pipeline', 'type': 'array', 'uniqueItems': True}}, 'title': 'Automatic reprocessing of X-Ray diffraction images', 'type': 'object'}, 'ui_schema': {'mx_pipeline_name': {'ui:widget': 'checkboxes'}, 'exclude': {'ui:widget': 'dozorExcludeRange'}, 'ui:order': ['mx_pipeline_name', 'exclude', 'low_res_limit', 'high_res_limit', 'forced_spacegroup', 'anomalous', '*']}, 'execute_arguments': {'_slurm_spawn_arguments': {'parameters': {'current_working_directory': '/tmp_14_days/opid23', 'environment': {'EDNA_SITE': 'ESRF_ID23EH2', 'USER': 'opid23'}, 'partition': 'nice', 'time_limit': 360}, 'pre_script': 'ORG_PATH=$PATH;module load bes_id23eh2;PATH=$PATH'}, 'engine': 'ppf'}, 'worker_options': {'queue': 'id23eh2'}, 'category': 'MX'}, 'nodes': [{'id': 'ewoksmx.tasks.WaitForFinished.WaitForFinished3', 'label': 'grenades_fastproc Wait For Finished', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForFinished.WaitForFinished', 'uiProps': {'icon': 'default.png', 'position': {'x': 1065, 'y': 45}, 'nodeWidth': 249}}, {'id': 'ewoksmx.tasks.WaitForStarted.WaitForStarted3', 'label': 'grenades_fastproc Wait For Started', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForStarted.WaitForStarted', 'uiProps': {'icon': 'default.png', 'position': {'x': 780, 'y': 45}, 'nodeWidth': 233}}, {'id': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit3', 'label': 'grenades_fastproc Slurm Submit', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit', 'uiProps': {'icon': 'default.png', 'position': {'x': 510, 'y': 45}, 'nodeWidth': 218}}, {'id': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'label': 'MXPipelineSynchronize', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize', 'uiProps': {'icon': 'graphOutput.svg', 'position': {'x': 1725, 'y': -120}, 'nodeWidth': 191}}, {'id': 'ewoksmx.tasks.WaitForStarted.WaitForStarted2', 'label': 'XIA2_DIALS Wait For Started', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForStarted.WaitForStarted', 'uiProps': {'icon': 'default.png', 'position': {'x': 780, 'y': -90}, 'nodeWidth': 209}}, {'id': 'ewoksmx.tasks.WaitForFinished.WaitForFinished2', 'label': 'XIA2_DIALS Wait For Finished', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForFinished.WaitForFinished', 'uiProps': {'icon': 'default.png', 'position': {'x': 1065, 'y': -90}, 'nodeWidth': 199}}, {'id': 'ewoksmx.tasks.WaitForFinished.WaitForFinished1', 'label': 'autoPROC Wait For Finished', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForFinished.WaitForFinished', 'uiProps': {'icon': 'default.png', 'position': {'x': 1065, 'y': -210}, 'nodeWidth': 204}}, {'id': 'ewoksmx.tasks.WaitForStarted.WaitForStarted1', 'label': 'autoPROC Wait For Started', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForStarted.WaitForStarted', 'uiProps': {'icon': 'default.png', 'position': {'x': 780, 'y': -210}, 'nodeWidth': 203}}, {'id': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit2', 'label': 'XIA2_DIALS Slurm Submit', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit', 'uiProps': {'icon': 'default.png', 'position': {'x': 510, 'y': -90}, 'nodeWidth': 202}}, {'id': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit1', 'label': 'autoPROC Slurm Submit', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit', 'uiProps': {'icon': 'default.png', 'position': {'x': 510, 'y': -210}, 'nodeWidth': 200}}, {'id': 'ewoksmx.tasks.WaitForFinished.WaitForFinished0', 'label': 'EDNA_proc Wait For Finished', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForFinished.WaitForFinished', 'uiProps': {'icon': 'default.png', 'position': {'x': 1065, 'y': -405}, 'nodeWidth': 222}}, {'id': 'ewoksmx.tasks.WaitForStarted.WaitForStarted0', 'label': 'EDNA_proc Wait For Started\n', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.WaitForStarted.WaitForStarted', 'uiProps': {'icon': 'default.png', 'position': {'x': 780, 'y': -405}, 'nodeWidth': 194}}, {'id': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit0', 'label': 'EDNA_proc Slurm Submit', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit', 'uiProps': {'icon': 'default.png', 'position': {'x': 465, 'y': -405}, 'nodeWidth': 210}}, {'id': 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline0', 'label': 'Grenades_fastproc_pipeline', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline', 'uiProps': {'icon': 'default.png', 'position': {'x': 195, 'y': 45}, 'nodeWidth': 198}}, {'id': 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline0', 'label': 'XIA2_DIALS_pipeline', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline', 'uiProps': {'icon': 'default.png', 'position': {'x': 210, 'y': -90}, 'nodeWidth': 176}}, {'id': 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline0', 'label': 'AutoPROC_pipeline', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline', 'inputs_complete': False, 'default_error_node': False, 'uiProps': {'icon': 'default.png', 'position': {'x': 210, 'y': -210}, 'moreHandles': False, 'withImage': True, 'withLabel': True, 'nodeWidth': 165}}, {'id': 'EDNA_proc_pipeline0', 'label': 'EDNA_proc_pipeline', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.EDNA_proc_pipeline.EDNA_proc_pipeline', 'inputs_complete': False, 'default_error_node': False, 'uiProps': {'icon': 'default.png', 'position': {'x': 210, 'y': -405}, 'moreHandles': False, 'withImage': True, 'withLabel': True, 'nodeWidth': 142}}, {'id': 'MXPipelineInput0', 'label': 'MXPipelineInput', 'task_type': 'class', 'task_identifier': 'ewoksmx.tasks.MXPipelineInput.MXPipelineInput', 'inputs_complete': False, 'default_error_node': False, 'uiProps': {'icon': 'graphInput.svg', 'position': {'x': -540, 'y': -75}, 'moreHandles': False, 'withImage': True, 'withLabel': True, 'nodeWidth': 138}}], 'links': [{'source': 'MXPipelineInput0', 'target': 'EDNA_proc_pipeline0', 'data_mapping': [{'source_output': 'metadata', 'target_input': 'metadata'}, {'source_output': 'EDNA_proc', 'target_input': 'EDNA_proc'}, {'source_output': 'autoPROC', 'target_input': 'autoPROC'}, {'source_output': 'XIA2_DIALS', 'target_input': 'XIA2_DIALS'}], 'conditions': [{'source_output': 'EDNA_proc', 'value': True}], 'on_error': False, 'required': False, 'map_all_data': False, 'uiProps': {'label': 'EDNA_proc: "true"', 'type': 'default', 'animated': False}}, {'source': 'MXPipelineInput0', 'target': 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline0', 'data_mapping': [{'source_output': 'metadata', 'target_input': 'metadata'}, {'source_output': 'EDNA_proc', 'target_input': 'EDNA_proc'}, {'source_output': 'autoPROC', 'target_input': 'autoPROC'}, {'source_output': 'XIA2_DIALS', 'target_input': 'XIA2_DIALS'}], 'conditions': [{'source_output': 'autoPROC', 'value': True}], 'on_error': False, 'required': False, 'map_all_data': False, 'uiProps': {'label': 'autoPROC: "true"', 'type': 'default', 'animated': False}}, {'source': 'MXPipelineInput0', 'target': 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline0', 'data_mapping': [{'source_output': 'metadata', 'target_input': 'metadata'}, {'source_output': 'EDNA_proc', 'target_input': 'EDNA_proc'}, {'source_output': 'autoPROC', 'target_input': 'autoPROC'}, {'source_output': 'XIA2_DIALS', 'target_input': 'XIA2_DIALS'}], 'conditions': [{'source_output': 'XIA2_DIALS', 'value': True}], 'on_error': False, 'map_all_data': False, 'uiProps': {'label': 'XIA2_DIALS: "true"'}}, {'source': 'EDNA_proc_pipeline0', 'target': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit0', 'data_mapping': [{'source_output': 'pipeline_name', 'target_input': 'pipeline_name'}, {'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit0', 'target': 'ewoksmx.tasks.WaitForStarted.WaitForStarted0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForStarted.WaitForStarted0', 'target': 'ewoksmx.tasks.WaitForFinished.WaitForFinished0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline0', 'target': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit1', 'data_mapping': [{'source_output': 'pipeline_name', 'target_input': 'pipeline_name'}, {'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline0', 'target': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit2', 'data_mapping': [{'source_output': 'pipeline_name', 'target_input': 'pipeline_name'}, {'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit1', 'target': 'ewoksmx.tasks.WaitForStarted.WaitForStarted1', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForStarted.WaitForStarted1', 'target': 'ewoksmx.tasks.WaitForFinished.WaitForFinished1', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit2', 'target': 'ewoksmx.tasks.WaitForStarted.WaitForStarted2', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForStarted.WaitForStarted2', 'target': 'ewoksmx.tasks.WaitForFinished.WaitForFinished2', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForFinished.WaitForFinished0', 'target': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForFinished.WaitForFinished1', 'target': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForFinished.WaitForFinished2', 'target': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'MXPipelineInput0', 'target': 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline0', 'data_mapping': [{'source_output': 'metadata', 'target_input': 'metadata'}, {'source_output': 'grenades_fastproc', 'target_input': 'grenades_fastproc'}, {'source_output': 'EDNA_proc', 'target_input': 'EDNA_proc'}, {'source_output': 'autoPROC', 'target_input': 'autoPROC'}, {'source_output': 'XIA2_DIALS', 'target_input': 'XIA2_DIALS'}], 'conditions': [{'source_output': 'grenades_fastproc', 'value': True}], 'on_error': False, 'map_all_data': False, 'uiProps': {'label': 'grenades_fastproc: "true"'}}, {'source': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit3', 'target': 'ewoksmx.tasks.WaitForStarted.WaitForStarted3', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForStarted.WaitForStarted3', 'target': 'ewoksmx.tasks.WaitForFinished.WaitForFinished3', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline0', 'target': 'ewoksmx.tasks.SlurmSubmit.SlurmSubmit3', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}, {'source_output': 'pipeline_name', 'target_input': 'pipeline_name'}], 'map_all_data': False}, {'source': 'ewoksmx.tasks.WaitForFinished.WaitForFinished3', 'target': 'ewoksmx.tasks.MXPipelineSynchronize.MXPipelineSynchronize0', 'data_mapping': [{'source_output': 'slurm_params', 'target_input': 'slurm_params'}], 'map_all_data': False}]},)
Keyword arguments: {'engine': 'ppf', 'startargs': {'anomalous': True, 'mx_pipeline_name': ['grenades_fastproc'], 'exclude': '123-513', 'low_res_limit': 50, 'high_res_limit': 2, 'forced_spacegroup': 'P212121', 'datasets': [{'id': 1883932075, 'name': 'XIA2_DIALS [2024-08-31 03:53:10]', 'startDate': '2024-08-31T03:53:08.454+02:00', 'endDate': '2024-08-31T03:53:08.454+02:00', 'location': '/data/visitor/mx2651/id23eh2/20240830/PROCESSED_DATA/Sample-3-1-10/run_02_datacollection/autoprocessing/XIA2_DIALS/XIA2_DIALS', 'investigation': {'id': 1824856659, 'createId': 'ingester', 'createTime': '2024-07-17T00:00:56.195+02:00', 'modId': 'ingester', 'modTime': '2024-08-31T08:00:13.879+02:00', 'dataCollectionInvestigations': [], 'datasets': [], 'doi': '10.15151/ESRF-ES-1824856659', 'endDate': '2024-08-31T08:00:00.000+02:00', 'fileCount': 0, 'fileSize': 0, 'fundingReferences': [], 'investigationFacilityCycles': [], 'investigationGroups': [], 'investigationInstruments': [{'id': 1884303114, 'createId': 'ingester', 'createTime': '2024-08-31T07:34:26.864+02:00', 'modId': 'ingester', 'modTime': '2024-08-31T07:34:26.864+02:00', 'instrument': {'id': 39153691, 'createId': '255362', 'createTime': '2017-06-16T15:35:40.631+02:00', 'modId': 'bodin/265147', 'modTime': '2024-04-22T09:58:10.276+02:00', 'datasetInstruments': [], 'description': 'Structural Biology Microfocus Beamline', 'fullName': 'ID23-2', 'instrumentScientists': [], 'investigationInstruments': [], 'name': 'ID23-2', 'shifts': [], 'type': 'Beamline', 'url': 'https://www.esrf.fr/UsersAndScience/Experiments/MX/About_our_beamlines/ID23-2'}}], 'investigationUsers': [], 'keywords': [], 'name': 'MX-2651', 'parameters': [], 'publications': [], 'releaseDate': '2027-08-31T08:00:00.000+02:00', 'samples': [], 'shifts': [], 'startDate': '2024-08-30T09:30:00.000+02:00', 'studyInvestigations': [], 'summary': 'The Austrian Crystallographic Diffraction Consortium (ACDC) BAG aims at supporting several macromolecular structural biology groups across Austria (in total\r\n20 research groups at 9 institutions in 4 cities). We employ mainly the technique of X-ray crystallography to characterize macromolecular biological complexes,\r\nbut also solution scattering approaches are frequently used to characterize dynamic assemblies. Approval of this BAG will be essential for researchers in all\r\nour groups to carry out their structural studies on various health-related and basic research-oriented topics, which are targeted at characterizing large molecular\r\nassemblies or proteins currently under-represented in the PDB, including membrane proteins. Corresponding high-resolution diffraction data are only obtainable\r\nthrough the use of synchrotron radiation sources. Thus regular synchrotron access is highly desirable to maintain a competitive advantage in our individual\r\nresearch fields.', 'title': 'Austrian Crystallographic Diffraction Consortium', 'type': {'id': 312, 'createId': 'root', 'createTime': '2014-06-18T15:00:43.317+02:00', 'modId': 'root', 'modTime': '2014-06-18T15:00:43.317+02:00', 'description': 'Macromolecular Crystallography', 'investigations': [], 'name': 'MX'}, 'visitId': 'MX/2651 ID23-2 30-08-2024/31-08-2024'}, 'type': 'processed', 'sampleName': 'Sample-3-1-10', 'sampleId': 1883835906, 'parameters': [{'name': 'scanType', 'value': 'integration', 'id': 1883932185, 'units': 'None', 'datasetId': 1883932075}, {'name': 'endDate', 'value': '2024-08-31T03:53:08.454420', 'id': 1883932189, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'Sample_name', 'value': 'Sample-3-1-10', 'id': 1883932184, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'startDate', 'value': '2024-08-31T03:53:08.454416', 'id': 1883932188, 'units': 'NA', 'datasetId': 1883932075}, {'name': '__fileCount', 'value': '15', 'id': 1883932195, 'units': 'NA', 'datasetId': 1883932075}, {'name': '__volume', 'value': '68429313', 'id': 1883932196, 'units': 'Bytes', 'datasetId': 1883932075}, {'name': '__elapsedTime', 'value': '0', 'id': 1883932197, 'units': 's', 'datasetId': 1883932075}, {'name': 'datasetName', 'value': 'XIA2_DIALS', 'id': 1883932193, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'input_datasets', 'value': '/data/visitor/mx2651/id23eh2/20240830/RAW_DATA/Sample-3-1-10/run_02_datacollection', 'id': 1883932187, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'input_datasetIds', 'value': '1883929032', 'id': 1883932190, 'units': 'NA', 'datasetId': 1883932075}, {'name': '__full_input_datasetIds', 'value': '1883929032', 'id': 1883932205, 'units': 'NA', 'datasetId': 1883932075}, {'name': '__full_input_datasetNames', 'value': 'run_02_datacollection', 'id': 1883932206, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_beta', 'value': '101.724707', 'id': 1883932141, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_resolution_limit_high', 'value': '4.827686171697526', 'id': 1883932162, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_completeness', 'value': '98.85474800146523', 'id': 1883932144, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_merge', 'value': '14.308336852215678', 'id': 1883932150, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_completeness', 'value': '97.35746982937994', 'id': 1883932157, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_cc_half', 'value': '0.9563052909054404', 'id': 1883932156, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_merge', 'value': '6.369120272905746', 'id': 1883932163, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_resolution_limit_low', 'value': '1.8094320636908312', 'id': 1883932175, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_cc_half', 'value': '0.2515337148878142', 'id': 1883932183, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_mean_I_over_sigI', 'value': '40.18795385195578', 'id': 1883932168, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_alpha', 'value': '90.0', 'id': 1883932140, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_anomalous_multiplicity', 'value': '1.8523825296388516', 'id': 1883932147, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_resolution_limit_high', 'value': '1.778758417415428', 'id': 1883932176, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_merge', 'value': '74.90074632325022', 'id': 1883932177, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_completeness', 'value': '98.94759921069941', 'id': 1883932170, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_c', 'value': '118.583447', 'id': 1883932137, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_meas_within_IPlus_IMinus', 'value': '105.33081846936821', 'id': 1883932178, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_a', 'value': '64.153227', 'id': 1883932133, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_resolution_limit_low', 'value': '65.64389801025389', 'id': 1883932148, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_meas_all_IPlus_IMinus', 'value': '9.272653445205377', 'id': 1883932165, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_anomalous_multiplicity', 'value': '1.882438805299798', 'id': 1883932160, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_multiplicity', 'value': '3.5830305620859155', 'id': 1883932159, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_mean_I_over_sigI', 'value': '1.0960728236334958', 'id': 1883932182, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_mean_I_over_sigI', 'value': '10.695375365467381', 'id': 1883932155, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_gamma', 'value': '90.0', 'id': 1883932142, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_anomalous_completeness', 'value': '91.60099076784508', 'id': 1883932171, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_resolution_limit_high', 'value': '1.7787593061413856', 'id': 1883932149, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_meas_within_IPlus_IMinus', 'value': '8.993147895562203', 'id': 1883932164, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_resolution_limit_low', 'value': '65.68861198534033', 'id': 1883932161, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_pim_within_IPlus_IMinus', 'value': '6.347248780550601', 'id': 1883932166, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_cc_half', 'value': '0.9730803959988407', 'id': 1883932169, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_anomalous_completeness', 'value': '93.52370266479664', 'id': 1883932145, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_pim_within_IPlus_IMinus', 'value': '14.223198285917874', 'id': 1883932153, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'Process_program', 'value': 'XIA2_DIALS', 'id': 1883932186, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_space_group', 'value': 'P 1 2 1', 'id': 1883932131, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_pim_all_IPlus_IMinus', 'value': '10.240392227105216', 'id': 1883932154, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_meas_all_IPlus_IMinus', 'value': '102.0355441527951', 'id': 1883932179, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_multiplicity', 'value': '3.5351424990463736', 'id': 1883932146, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_anomalous', 'value': '1', 'id': 1883932143, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_pim_within_IPlus_IMinus', 'value': '73.9812144665318', 'id': 1883932180, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_meas_within_IPlus_IMinus', 'value': '20.17998638545335', 'id': 1883932151, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_multiplicity', 'value': '3.516951030356747', 'id': 1883932172, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_anomalous_multiplicity', 'value': '1.8496678708775203', 'id': 1883932174, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_overall_r_meas_all_IPlus_IMinus', 'value': '19.214985838033652', 'id': 1883932152, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegration_cell_b', 'value': '65.643865', 'id': 1883932135, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_r_pim_all_IPlus_IMinus', 'value': '4.857884056242371', 'id': 1883932167, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_outer_r_pim_all_IPlus_IMinus', 'value': '53.89944846102994', 'id': 1883932181, 'units': 'NA', 'datasetId': 1883932075}, {'name': 'MXAutoprocIntegrationScaling_inner_anomalous_completeness', 'value': '98.27946989072309', 'id': 1883932158, 'units': 'NA', 'datasetId': 1883932075}], 'instrumentName': 'ID23-2', 'techniques': [{'id': 1580426249, 'datasetId': 1883932075, 'pid': 'PaNET:PaNET01012', 'name': 'x-ray probe'}], 'meta': {'page': {'totalWithoutFilters': 1, 'total': 1, 'totalPages': 1, 'currentPage': 1}}}], 'callback': 'https://icatplus.esrf.fr/ewoks/394fe3f6-0fae-440b-896e-922c069a374b/jobs?id=66d98095385da400137bdee7', 'raw_data_path': ['/data/visitor/mx2651/id23eh2/20240830/PROCESSED_DATA/Sample-3-1-10/run_02_datacollection/autoprocessing/XIA2_DIALS/XIA2_DIALS']}, 'execinfo': {'handlers': [{'class': 'ewoksjob.events.handlers.RedisEwoksEventHandler', 'arguments': [{'name': 'url', 'value': 'redis://ewoksserver1:25004/0'}, {'name': 'ttl', 'value': 86400}]}], 'job_id': '66031988-0b70-44b9-aeba-59470fab8166'}}
Sending result <class 'tuple'> to ewoksserver1.esrf.fr:59049 ...
STDERR: /tmp_14_days/ewoks_opid23/slurm_logs/pyslurmutils.ewoksserver1.17046803.err
------------------------------------------------------------------------------------
slurmstepd-hpc7-01: error: couldn't chdir to `/tmp_14_days/opid23': No such file or directory: going to /tmp instead
slurmstepd-hpc7-01: error: couldn't chdir to `/tmp_14_days/opid23': No such file or directory: going to /tmp instead
[MXDataReductionPipelines (ID23-2)] [<EwoksPythonActor> MXPipelineInput] Error in python actor 'MXPipelineInput'!
Not running down stream actors ["Conditional actor between 'MXPipelineInput0' and 'EDNA_proc_pipeline0'", "Conditional actor between 'MXPipelineInput0' and 'ewoksmx.tasks.AutoPROC_pipeline.AutoPROC_pipeline0'", "Conditional actor between 'MXPipelineInput0' and 'ewoksmx.tasks.XIA2_DIALS_pipeline.XIA2_DIALS_pipeline0'", "Conditional actor between 'MXPipelineInput0' and 'ewoksmx.tasks.Grenades_fastproc_pipeline.Grenades_fastproc_pipeline0'"]
Exception:Task 'MXPipelineInput' failed
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 401, in execute
self.run()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/ewoksmx/src/ewoksmx/tasks/MXPipelineInput.py", line 48, in run
raise FileNotFoundError(str(metadata_path))
FileNotFoundError: /data/visitor/mx2651/id23eh2/20240830/PROCESSED_DATA/Sample-3-1-10/run_02_datacollection/autoprocessing/XIA2_DIALS/XIA2_DIALS/metadata.json
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/pypushflow/concurrent/interrupt/process.py", line 14, in task_main
return fn(*args, **kwargs)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoksppf/ppfrunscript.py", line 27, in run
task.execute()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 407, in execute
raise RuntimeError(f"Task '{self.label}' failed") from e
RuntimeError: Task 'MXPipelineInput' failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/pypushflow/concurrent/process.py", line 78, in cb
result = future.result()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
RuntimeError: Task 'MXPipelineInput' failed
[MXDataReductionPipelines (ID23-2)] Interrupt workflow execution: Task 'MXPipelineInput' failed
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 401, in execute
self.run()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/ewoksmx/src/ewoksmx/tasks/MXPipelineInput.py", line 48, in run
raise FileNotFoundError(str(metadata_path))
FileNotFoundError: /data/visitor/mx2651/id23eh2/20240830/PROCESSED_DATA/Sample-3-1-10/run_02_datacollection/autoprocessing/XIA2_DIALS/XIA2_DIALS/metadata.json
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/pypushflow/concurrent/interrupt/process.py", line 14, in task_main
return fn(*args, **kwargs)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoksppf/ppfrunscript.py", line 27, in run
task.execute()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 407, in execute
raise RuntimeError(f"Task '{self.label}' failed") from e
RuntimeError: Task 'MXPipelineInput' failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 50, in <module>
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoks/bindings.py", line 78, in execute_graph
result = mod.execute_graph(graph, execinfo=execinfo, **execute_options)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/events/contexts.py", line 25, in wrapper
return method(*args, execinfo=execinfo, **kw)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoksppf/bindings.py", line 599, in execute_graph
return ppfgraph.run(**execute_options)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewoksppf/bindings.py", line 564, in run
raise ex
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/pypushflow/concurrent/process.py", line 78, in cb
result = future.result()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
RuntimeError: Task 'MXPipelineInput' failed
[2024-09-05 11:57:53,338: ERROR/MainProcess] Task ewoksjob.apps.ewoks.execute_graph[ff1afb62-c68b-4c7b-a1f8-a522a155d430] raised unexpected: RuntimeError("Task 'MXPipelineInput' failed")
Traceback (most recent call last):
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/ewoksjob/apps/ewoks.py", line 25, in wrapper
return method(self, *args, **kwargs)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/ewoksjob/worker/task.py", line 19, in wrapper
return _executor(func, *args, **kwargs)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/ewoksjob/worker/slurm.py", line 75, in executor
return future.result()
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/pyslurmutils/client/job_io/tcp_io.py", line 67, in result
errors.reraise_remote_exception_from_tb(*result, tb)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/pyslurmutils/client/errors.py", line 42, in reraise_remote_exception_from_tb
raise remote_exception_from_tb(exc_cls, exc_msg, tb)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 401, in execute
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/ewoksmx/src/ewoksmx/tasks/MXPipelineInput.py", line 48, in run
RuntimeError: Task 'MXPipelineInput' failed
[2024-09-05 11:57:53,346: ERROR/MainProcess] Task ewoksjob.apps.ewoks.execute_graph[66031988-0b70-44b9-aeba-59470fab8166] raised unexpected: RuntimeError("Task 'MXPipelineInput' failed")
Traceback (most recent call last):
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/ewoksjob/apps/ewoks.py", line 25, in wrapper
return method(self, *args, **kwargs)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/ewoksjob/worker/task.py", line 19, in wrapper
return _executor(func, *args, **kwargs)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/ewoksjob/worker/slurm.py", line 75, in executor
return future.result()
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/pyslurmutils/client/job_io/tcp_io.py", line 67, in result
errors.reraise_remote_exception_from_tb(*result, tb)
File "/opt/ansible/venv/ewoks/lib/python3.8/site-packages/pyslurmutils/client/errors.py", line 42, in reraise_remote_exception_from_tb
raise remote_exception_from_tb(exc_cls, exc_msg, tb)
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/miniconda3/envs/id23eh2/lib/python3.9/site-packages/ewokscore/task.py", line 401, in execute
File "/cvmfs/sb.esrf.fr/software/packages/ubuntu20.04/x86_64/bes/20230110/id23eh2/ewoksmx/src/ewoksmx/tasks/MXPipelineInput.py", line 48, in run
RuntimeError: Task 'MXPipelineInput' failedPing @svensson
Edited by Wout De Nolf