Skip to content

Draft: Sidecar Server

Stuart Fisher requested to merge sidecar-server into main

Closes #31

This reworks sidecar to act only as a message handler between daiquiri (mimosa) and ewoksserver.

  • The base package can be installed with minimal dependencies so the tasks can be used within ewoks. These tasks are auto-discoverable in ewoksserver/web via entry points
  • pip install .[server] will install the server dependencies. The server will accept messages sent via kombu (redis or rabbitmq). It supports the old celery style messages so will continue to work with the existing daiquiri implementation. Internally it uses a new simpler message system that daiquiri should be migrated to in due course
  • The server accepts messages from kombu and starts ewoks tasks via the REST API exposed by ewoksserver. It then checks the status of the launched jobs and updates mimosa with the relevant status so daiquiri knows whats going on. At task end all the ewoks events are captured and stored in a log file. We loose stdout capture as theres no way right now to capture this in ewoksserver.
  • The server still injects a bunch of metadata into the first task of the workflow. metadata resolved info for the datacollection from mimosa, and the autoProcProgramId so that a workflow can create attachments, or put things into the database. Sidecar provides an Input task to fan-out these inputs to other tasks.
  • Sidecar no longer has any knowledge of ewoks or its configuration, this is now completely decoupled.
  • The tests e2e test kombu -> sidecar.server -> ewoksserver

To do:

  • Allow extra inputs to be directed to specific tasks via task id
  • Fix mimas
  • Refactor messages with pydantic for validation
  • Update documentation
Edited by Stuart Fisher

Merge request reports