Newer
Older
This project is meant to find the optimal workflow eco-system to be maintained by the ESRF DAU.
```bash
./devinstall.sh
```
Run the tests
```bash
pytest
```
python examples/running_taskgraphs.py --plot
Common runtime and persistent representation of tasks and task graphs
* *esrftaskgraph*: based on *networkx* with task and data management (currently a proof-of-concept with JSON files)
* *esrf2multiprocessing*: multiprocessing task scheduler (DAGs only)
* *esrf2paradag*: multitheading task scheduler (DAGs only)
### Technique specific projects
Only dependent on *esrftaskgraph*
* *tasklib*: example library of task implementations based on the *esrftaskgraph* abstraction.
* *graphlib*: example library with graphs of *tasklib* tasks. These are beamline/proposal specific graphs.
* *orange3widgetlib*: example library of Orange widgets for *tasklib* tasks.
Some task libraries could become part of the core projects (PCA, FFT, SIFT, ...).
* *Graph* (a.k.a. Task Graph, Workflow): list of Tasks and Links
* *Graph instance*: a *Graph* with fixed static inputs
* *Task instance*: a *Task* with fixed inputs
* *Pipeline*: a linear task *Graph*
* Directed graphs: edges have orientations (uni/bi-directional)
* Oriented graphs: no bidirectional edges
* Directed acyclic graphs (DAGs): no directed cycles
Apart from persistent representations (json, yaml, ...), graphs can be designed with GUIs. Often these GUIs also allow starting and monitoring a task scheduler.
Execute/distribute tasks from a *graph instance*
A *job* is one execution of a *graph instance* to completion, error or interrupt. A job could however be anything else, not related to task graphs.