sidecar merge requestshttps://gitlab.esrf.fr/workflow/sidecar/-/merge_requests2024-02-07T16:19:34+01:00https://gitlab.esrf.fr/workflow/sidecar/-/merge_requests/54Draft: Sidecar Server2024-02-07T16:19:34+01:00Stuart FisherDraft: Sidecar ServerCloses #31
This reworks sidecar to act only as a message handler between daiquiri (mimosa) and ewoksserver.
* The base package can be installed with minimal dependencies so the tasks can be used within ewoks. These tasks are auto-disc...Closes #31
This reworks sidecar to act only as a message handler between daiquiri (mimosa) and ewoksserver.
* The base package can be installed with minimal dependencies so the tasks can be used within ewoks. These tasks are auto-discoverable in ewoksserver/web via entry points
* pip install .[server] will install the server dependencies. The server will accept messages sent via kombu (redis or rabbitmq). It supports the old celery style messages so will continue to work with the existing daiquiri implementation. Internally it uses a new simpler message system that daiquiri should be migrated to in due course
* The server accepts messages from kombu and starts ewoks tasks via the REST API exposed by ewoksserver. It then checks the status of the launched jobs and updates mimosa with the relevant status so daiquiri knows whats going on. At task end all the ewoks events are captured and stored in a log file. We loose stdout capture as theres no way right now to capture this in ewoksserver.
* The server still injects a bunch of metadata into the first task of the workflow. `metadata` resolved info for the datacollection from mimosa, and the autoProcProgramId so that a workflow can create attachments, or put things into the database. Sidecar provides an `Input` task to fan-out these inputs to other tasks.
* Sidecar no longer has any knowledge of ewoks or its configuration, this is now completely decoupled.
* The tests e2e test kombu -> sidecar.server -> ewoksserver
To do:
- [ ] Allow extra inputs to be directed to specific tasks via task `id`
- [x] Fix mimas
- [x] Refactor messages with pydantic for validation
- [x] Update documentationStuart FisherStuart Fisherhttps://gitlab.esrf.fr/workflow/sidecar/-/merge_requests/52Draft: no more global inputs2024-01-31T16:15:00+01:00Stuart FisherDraft: no more global inputsI added `sidecar.ewoks.inputs.Inputs` to allow the sidecar inputs to be passed around if needed (use this as the first input to the workflow).
@valentin.valls if theres only one task in your tomo workflow this _shouldnt_ affect you (fam...I added `sidecar.ewoks.inputs.Inputs` to allow the sidecar inputs to be passed around if needed (use this as the first input to the workflow).
@valentin.valls if theres only one task in your tomo workflow this _shouldnt_ affect you (famous last words).Stuart FisherStuart Fisher