pypushflow merge requestshttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests2020-04-16T08:13:32+02:00https://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/2Add representation2020-04-16T08:13:32+02:00paynoAdd representationGoal
====
embed 'representation' in pypushflow
TODO
====
- [x] add a setup.py ...
- [x] define concrete scenarii and use case
- [x] add some unittest
- add structure for unit test
- add unit test dedicated to repre...Goal
====
embed 'representation' in pypushflow
TODO
====
- [x] add a setup.py ...
- [x] define concrete scenarii and use case
- [x] add some unittest
- add structure for unit test
- add unit test dedicated to representation
- add general unit test for pypushflow
- [x] include representation in the project
- [ ] define clearly the user API
- [ ] add some tutorial
- [x] add some documentation
- add unit test for
- [x] executable class
- [x] class with handlers
- [x] function
- [x] script with a run function
extra
-----
close #1
Details
-------
Good
````
- add representation submodule (so you should now clone the project with --recursive option or if already clone you can go for git submodule update --init --recursive)
- add some unittest
- add documentation (at least the structure but there is plenty of TODO and uncomented function / class...)
- add automatic generation of the documentation. Once a PR is merge on the master it will call automatic generation of documentation and store it in https://workflow.gitlab-pages.esrf.fr/pypushflow/.
For now it has only be generated on my side. You can view it here: https://payno.gitlab-pages.esrf.fr/pypushflow/
- add ci
- some PEP8
Bad
```
- turn off the 'mongoDB' stuff for now. But as we discuss this should be an add-on. Once this part is done I can try to have a look at it.
- fail to execute the unit test from 'TestWorkflows' but it looks like it need to have mongodb install. So for now it is off.
- for the 'TestPythonActor' the full name of the script is given ('pypushflow.test.pythonActorTest.py'). PythonActor originally expect the given script contains a 'run' function to be executed. But from the 'representation' design it should be 'pypushflow.test.pythonActorTest.run' instead. But for now we let the 'script' variable. So when using this variable we make a workaround to translate in order to keep compatibility. But this workaround and the 'script' parameter should be removed and replace by giving directly the representation.scheme.node.Node class. Current workaround allows to keep code compatibility.
- maybe actor name need to be rework a bit.https://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/3add management of add-ons2021-01-06T16:42:50+01:00paynoadd management of add-onsGoal
----
Define and manage add-on to pypushflow.
/close #2
Info
----
This PR come with two add-on project:
- a project to explain add-on with an hello-world (https://gitlab.esrf.fr/workflow/ppfhelloworld)
- a function...Goal
----
Define and manage add-on to pypushflow.
/close #2
Info
----
This PR come with two add-on project:
- a project to explain add-on with an hello-world (https://gitlab.esrf.fr/workflow/ppfhelloworld)
- a functional add-on 'mongodb' to publish workflow execution on a mongodb database. (https://gitlab.esrf.fr/workflow/ppfmongodb)0.1https://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/4Integrate tomwer2020-08-04T08:34:22+02:00paynoIntegrate tomwerTODO
----
- [x] Integarte pypushflow with tomwer processes
- [x] manage dependancies on 'channels'
- [x] test previous casesTODO
----
- [x] Integarte pypushflow with tomwer processes
- [x] manage dependancies on 'channels'
- [x] test previous caseshttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/5Add script translation2021-01-06T16:41:04+01:00paynoAdd script translationhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/6Add black hook2020-08-21T14:16:27+02:00paynoAdd black hookhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/10Resolve "Refactor the StopActor"2021-06-30T10:32:01+02:00Wout De NolfResolve "Refactor the StopActor"Closes #7
`ThreadCountingActor` adds thread counting to `trigger`. Classes that derive from it:
* `AbstractActor`
* `StopActor`
* `Port`
The list of actor references is removed from `Workflow` and `Submodel`.Closes #7
`ThreadCountingActor` adds thread counting to `trigger`. Classes that derive from it:
* `AbstractActor`
* `StopActor`
* `Port`
The list of actor references is removed from `Workflow` and `Submodel`.Wout De NolfWout De Nolfhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/11Release 0.3.0.a2021-07-29T15:45:52+02:00Wout De NolfRelease 0.3.0.a@payno I only deployed on testpypi for now: https://test.pypi.org/project/pypushflow/0.3.0a0/@payno I only deployed on testpypi for now: https://test.pypi.org/project/pypushflow/0.3.0a0/Wout De NolfWout De Nolfhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/13apply_async error handler only receives one argument2021-09-16T11:43:03+02:00Wout De Nolfapply_async error handler only receives one argumenthttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/14Enforce style2021-09-16T11:43:44+02:00Wout De NolfEnforce stylehttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/15bump version2021-09-16T11:50:32+02:00Wout De Nolfbump versionhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/16simplify result/error handling in PythonActor2021-09-24T11:28:23+02:00Wout De Nolfsimplify result/error handling in PythonActorResult and exception handling in PythonActor was unnecessary complicated.Result and exception handling in PythonActor was unnecessary complicated.Wout De NolfWout De Nolfhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/12Resolve "Make Mongo logging configurable"2021-09-24T11:29:06+02:00Wout De NolfResolve "Make Mongo logging configurable"Closes #8
This MR refactors logging of workflow execution in Mongo.
In addition it allows installing actor data filters which will be used by `ewoksppf` to normalize the data (remove `_noinput` and dereference `Variable` objects)Closes #8
This MR refactors logging of workflow execution in Mongo.
In addition it allows installing actor data filters which will be used by `ewoksppf` to normalize the data (remove `_noinput` and dereference `Variable` objects)Wout De NolfWout De Nolfhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/17Release2021-09-24T11:36:48+02:00Wout De NolfReleasehttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/19Fixes olof2021-09-29T19:06:38+02:00Wout De NolfFixes olofreplaces !18replaces !18https://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/20Resolve "Errors in pythonactor result handler is not properly handled"2021-12-09T16:48:42+01:00Wout De NolfResolve "Errors in pythonactor result handler is not properly handled"Closes #9
In our last tests, the exception we got from putting data in `mongo` (a bug which has been fixed elsewhere) was not visible. This was because exceptions in async callbacks are not captured anywhere. This MR captures all except...Closes #9
In our last tests, the exception we got from putting data in `mongo` (a bug which has been fixed elsewhere) was not visible. This was because exceptions in async callbacks are not captured anywhere. This MR captures all exceptions in `PythonActor.resultHandler` and `PythonActor.errorHandler` and handles them appropriately.Wout De NolfWout De Nolfhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/21improve logging by adding log context (workflow name, actor name)2021-12-09T16:54:24+01:00Wout De Nolfimprove logging by adding log context (workflow name, actor name)I was trying to debug pypushflow but the logging was very confusing. I tried to clean it up by adding log context.
This is the logging of the triggering and execution of one python actor:
```python
DEBUG pypushflow:ThreadCounter.py:...I was trying to debug pypushflow but the logging was very confusing. I tried to clean it up by adding log context.
This is the logging of the triggering and execution of one python actor:
```python
DEBUG pypushflow:ThreadCounter.py:49 [submodel] Thread started for 'in' (2 threads running)
INFO pypushflow:PythonActor.py:114 [submodel] [<EwoksPythonActor> in] triggered with inData =
{'_noinput': {'node_attrs': {'task_identifier': 'ewoksppf.tests.test_ppf_workflow21.passthrough',
'task_type': 'method'},
'node_id': 'in',
'varinfo': {'root_uri': '/tmp/pytest-of-denolf/pytest-87/test_workflow21_True_True_args0'}},
'a': 20}
INFO pypushflow:AbstractActor.py:78 [submodel] [<EwoksPythonActor> in] started
DEBUG pypushflow:PythonActor.py:81 [submodel] [<EwoksPythonActor> in] asynchronous execution of 'ewoksppf.ppfrunscript.run'
DEBUG pypushflow:ThreadCounter.py:49 [submodel] Thread ended for 'Start' (1 threads running)
DEBUG pypushflow:StopActor.py:49 [submodel] [<StopActor> Stop] wait for scheduler threads to be finished
INFO pypushflow:AbstractActor.py:85 [submodel] [<EwoksPythonActor> in] finished
```Wout De NolfWout De Nolfhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/22bump version2021-12-09T17:16:53+01:00Wout De Nolfbump versionhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/23missing connect logging for RouterActor2022-01-10T14:28:12+01:00Wout De Nolfmissing connect logging for RouterActorWout De NolfWout De Nolfhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/24improve logs2022-01-29T16:49:28+01:00Wout De Nolfimprove logs* fix bug in logs
* better CI
* refactor Pool to reduce the chance for hanging (it still exists though).* fix bug in logs
* better CI
* refactor Pool to reduce the chance for hanging (it still exists though).Wout De NolfWout De Nolfhttps://gitlab.esrf.fr/workflow/pypushflow/-/merge_requests/25Resolve "Solve danging sub processes"2022-02-01T19:11:06+01:00Wout De NolfResolve "Solve danging sub processes"Related #10
Call `pool.join` after `pool.terminate` to wait until workers finished.
Add the possibility to use a common process pool for a workflow (used in the test suite).
Neither solve the freezing issue caused by `pytest-cov`Related #10
Call `pool.join` after `pool.terminate` to wait until workers finished.
Add the possibility to use a common process pool for a workflow (used in the test suite).
Neither solve the freezing issue caused by `pytest-cov`Wout De NolfWout De Nolf