Wout De Nolf requested to merge 25-cli-execute-submit-convert-multiple-workflows into main Feb 04, 2024

Add support for multiple workflows to the ewoks CLI.

Either provide multiple arguments

ewoks execute workflow1.json workflow2.json ...
ewoks submit workflow1.json workflow2.json ...

When the --search flag is set, the workflow argument(s) are searched for (glob or fnmatch in the case of --test)

ewoks execute mydir1/*.json mydir2/*.json ... --search
ewoks submit mydir1/*.json mydir2/*.json ... --search
ewoks convert mydir1/*.json _suffix.yaml --search

Note that "convert" only takes one workflow argument while "execute" and "submit" can take multiple.

The CLI main function does not raise workflow exceptions anymore because we want to execute all for them, even when one fails. It does print the traceback and the return_code with be set to 1.

I also decided to move the cliutils from ewokscore to ewoks. The reason why it was in ewokscore is that changes to the python API of execute_graph can be directly applied to the CLI in one go. However the ewoks CLI already adds arguments to the ewokscore CLI and if we want to change the CLI without changing the python API (which is what this MR does), we would have to modify both ewokscore and ewoks. So I think it is better to have all CLI stuff in the one and only project that exposes the CLI, which is ewoks.

Some examples below (acyclic2 is expected to fail).

Execute multiple workflows

ewoks execute acyclic* --test --search --outputs=all

###################################
# Execute workflow 'acyclic1'
###################################
{'task1': {'result': 1},
 'task2': {'result': 2},
 'task3': {'result': 4},
 'task4': {'result': 6},
 'task5': {'result': 10},
 'task6': {'result': 16}}
FINISHED

###################################
# Execute workflow 'acyclic2'
###################################
Traceback (most recent call last):
  File "/home/denolf/projects/ewoks/src/ewoks/__main__.py", line 52, in command_execute
    results = execute_graph(graph, engine=args.engine, **args.execute_options)
  File "/home/denolf/projects/ewoks/src/ewoks/bindings.py", line 78, in execute_graph
    result = mod.execute_graph(graph, execinfo=execinfo, **execute_options)
  File "/home/denolf/projects/ewokscore/src/ewokscore/events/contexts.py", line 25, in wrapper
    return method(*args, execinfo=execinfo, **kw)
  File "/home/denolf/projects/ewokscore/src/ewokscore/bindings.py", line 56, in execute_graph
    return sequential.execute_graph(taskgraph.graph, **execute_options)
  File "/home/denolf/projects/ewokscore/src/ewokscore/graph/execute/sequential.py", line 100, in execute_graph
    raise RuntimeError("cannot execute graphs with conditional links")
RuntimeError: cannot execute graphs with conditional links
FAILED

###################################
# Execute workflow 'acyclic3'
###################################
task7: {'result': 16}
task8: <no inputs>
{'task1': {'result': 1},
 'task2': {'result': 2},
 'task3': {'result': 4},
 'task4': {'result': 6},
 'task5': {'result': 10},
 'task6': {'result': 16},
 'task7': {},
 'task8': {}}
FINISHED

Submit multiple workflows

ewoks submit acyclic* --test --search --outputs=all
Workflow 'acyclic1' submitted (ID: 2209e83f-4048-465b-a749-aba679d47c12)
Workflow 'acyclic2' submitted (ID: 4085dbb0-1966-4532-b9cb-fa38a7bd6f07)
Workflow 'acyclic3' submitted (ID: e2636a9a-5b29-4a9c-b4ac-43c04de2dd21)

Submit multiple workflows (wait)

ewoks submit acyclic* --test --search --outputs=all --wait inf

Workflow 'acyclic1' submitted (ID: c175877b-27bb-4a84-a225-9b0359409137)
Workflow 'acyclic2' submitted (ID: 99eba6b3-c4e6-430d-86bf-2369dafe8326)
Workflow 'acyclic3' submitted (ID: e751a6ee-fb71-4b54-8298-ecaeb3d4417a)
Waiting for results ...

###########################################################################
# Result of workflow 'acyclic1' (ID: c175877b-27bb-4a84-a225-9b0359409137)
###########################################################################
{'task1': {'result': 1},
 'task2': {'result': 2},
 'task3': {'result': 4},
 'task4': {'result': 6},
 'task5': {'result': 10},
 'task6': {'result': 16}}
FINISHED

###########################################################################
# Result of workflow 'acyclic2' (ID: 99eba6b3-c4e6-430d-86bf-2369dafe8326)
###########################################################################
Traceback (most recent call last):
  File "/home/denolf/projects/ewoks/src/ewoks/__main__.py", line 107, in command_submit
    results = future.get(timeout=args.wait)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/result.py", line 251, in get
    return self.backend.wait_for_pending(
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/backends/asynchronous.py", line 223, in wait_for_pending
    return result.maybe_throw(callback=callback, propagate=propagate)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/result.py", line 365, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/result.py", line 358, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/vine/promises.py", line 235, in throw
    reraise(type(exc), exc, tb)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/vine/utils.py", line 26, in reraise
    raise value.with_traceback(tb)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/denolf/projects/ewoksjob/src/ewoksjob/apps/ewoks.py", line 26, in new_celery_task
    return celery_task(self, *args, **kwargs)
  File "/home/denolf/projects/ewoksjob/src/ewoksjob/apps/ewoks.py", line 45, in new_celery_task
    return celery_task(*args, **kwargs)
  File "/home/denolf/projects/ewoksjob/src/ewoksjob/apps/ewoks.py", line 61, in execute_graph
    return ewoks.execute_graph(*args, **kwargs)
  File "/home/denolf/projects/ewoks/src/ewoks/bindings.py", line 78, in execute_graph
    result = mod.execute_graph(graph, execinfo=execinfo, **execute_options)
  File "/home/denolf/projects/ewokscore/src/ewokscore/events/contexts.py", line 25, in wrapper
    return method(*args, execinfo=execinfo, **kw)
  File "/home/denolf/projects/ewokscore/src/ewokscore/bindings.py", line 56, in execute_graph
    return sequential.execute_graph(taskgraph.graph, **execute_options)
  File "/home/denolf/projects/ewokscore/src/ewokscore/graph/execute/sequential.py", line 100, in execute_graph
    raise RuntimeError("cannot execute graphs with conditional links")
RuntimeError: cannot execute graphs with conditional links
FAILED

###########################################################################
# Result of workflow 'acyclic3' (ID: e751a6ee-fb71-4b54-8298-ecaeb3d4417a)
###########################################################################
{'task1': {'result': 1},
 'task2': {'result': 2},
 'task3': {'result': 4},
 'task4': {'result': 6},
 'task5': {'result': 10},
 'task6': {'result': 16},
 'task7': {},
 'task8': {}}
FINISHED

Submit multiple workflows (wait timeout)

ewoks submit acyclic* --test --search --outputs=all --wait 0.1 -p delay=1

Workflow 'acyclic1' submitted (ID: 14cff081-dd0d-4a8e-b220-d455dc6c352d)
Workflow 'acyclic2' submitted (ID: 38dcc951-daa4-4c56-ae36-81517003485c)
Workflow 'acyclic3' submitted (ID: 4e890aa5-ea3a-4b3a-a7bb-427c1819b81e)
Waiting for results ...

###########################################################################
# Result of workflow 'acyclic1' (ID: 14cff081-dd0d-4a8e-b220-d455dc6c352d)
###########################################################################
Not finished after 0.1s

###########################################################################
# Result of workflow 'acyclic2' (ID: 38dcc951-daa4-4c56-ae36-81517003485c)
###########################################################################
Traceback (most recent call last):
  File "/home/denolf/projects/ewoks/src/ewoks/__main__.py", line 107, in command_submit
    results = future.get(timeout=args.wait)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/result.py", line 251, in get
    return self.backend.wait_for_pending(
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/backends/asynchronous.py", line 223, in wait_for_pending
    return result.maybe_throw(callback=callback, propagate=propagate)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/result.py", line 365, in maybe_throw
    self.throw(value, self._to_remote_traceback(tb))
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/result.py", line 358, in throw
    self.on_ready.throw(*args, **kwargs)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/vine/promises.py", line 235, in throw
    reraise(type(exc), exc, tb)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/vine/utils.py", line 26, in reraise
    raise value.with_traceback(tb)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/denolf/virtualenvs/ewoks/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/denolf/projects/ewoksjob/src/ewoksjob/apps/ewoks.py", line 26, in new_celery_task
    return celery_task(self, *args, **kwargs)
  File "/home/denolf/projects/ewoksjob/src/ewoksjob/apps/ewoks.py", line 45, in new_celery_task
    return celery_task(*args, **kwargs)
  File "/home/denolf/projects/ewoksjob/src/ewoksjob/apps/ewoks.py", line 61, in execute_graph
    return ewoks.execute_graph(*args, **kwargs)
  File "/home/denolf/projects/ewoks/src/ewoks/bindings.py", line 78, in execute_graph
    result = mod.execute_graph(graph, execinfo=execinfo, **execute_options)
  File "/home/denolf/projects/ewokscore/src/ewokscore/events/contexts.py", line 25, in wrapper
    return method(*args, execinfo=execinfo, **kw)
  File "/home/denolf/projects/ewokscore/src/ewokscore/bindings.py", line 56, in execute_graph
    return sequential.execute_graph(taskgraph.graph, **execute_options)
  File "/home/denolf/projects/ewokscore/src/ewokscore/graph/execute/sequential.py", line 100, in execute_graph
    raise RuntimeError("cannot execute graphs with conditional links")
RuntimeError: cannot execute graphs with conditional links
FAILED

###########################################################################
# Result of workflow 'acyclic3' (ID: 4e890aa5-ea3a-4b3a-a7bb-427c1819b81e)
###########################################################################
Not finished after 0.1s

Convert workflows

ewoks convert "acyclic*" json --test --search 
Converted acyclic1 -> acyclic1.json
Converted acyclic2 -> acyclic2.json
Converted acyclic3 -> acyclic3.json

ewoks convert "acyclic*" _conv.json --test --search
Converted acyclic1 -> acyclic1_conv.json
Converted acyclic2 -> acyclic2_conv.json
Converted acyclic3 -> acyclic3_conv.json

Edited Feb 04, 2024 by Wout De Nolf

Resolve "CLI: execute/submit/convert multiple workflows"

Execute multiple workflows

Submit multiple workflows

Submit multiple workflows (wait)

Submit multiple workflows (wait timeout)

Convert workflows

Merge request reports