Skip to content

Test for handling CWL and cwl-runner

payno requested to merge proto_cwl into master

Description / Goal

See how we could handle the common workflow language.

TODO list

scenario

We want to execute a classic "spectroscopy workflow" using pymca. read data -> normalization -> exafs -> ft weight -> fourier transform

step 1

For now the spectrum read and the launch of the worklow is one thourgh the test.py file. And we provide a spec file. Then on test we read it and dump the xas_object to .hdf5 file that we are already able to read / write. And this is this file that will be overwrite at each step and that should contain an updated XASObject (according to the processes it encounter)

how to execute:

python test.py workflow_def_test_1.cwl --input-spec-file EXAFS_Cu.dat

current state:

  • it seems dumping file after XAS normalization does not contain enough information. Might need to create a 'saving point' or the writing or reading of normalization is missing some information
  • for now there is no output and the input is done using a string when File should be used. Have to check if this is possible.

step 2

save but this time the .cwl is using file to be pass from one step to another. But still overwriting the original spectrum file.

workflow_def_test_1.cwl

What could be done

  • Modify the input and output to use Python object and keep 'cwl-runner' compatibility
  • Add this management in pypushflow
  • browse existing viewers and see if one could be use for example https://github.com/rabix/composer or cwl-viewer
  • See if we could handle loop. Could have several output not always generated. But for this cases would prevent from using other softwares. I guess this simply cannot be handled.
Edited by payno

Merge request reports