LimaGroup issueshttps://gitlab.esrf.fr/groups/limagroup/-/issues2023-07-24T10:58:46+02:00https://gitlab.esrf.fr/limagroup/lima2/-/issues/167Add XPCS processing2023-07-24T10:58:46+02:00Samuel DebionneAdd XPCS processingXPCS processing would sparsify (remove 0 photon data) and save the data in the same sparse format than the one used for saving the peaks in the SMX processing. The format we use is basically 3 vectors:
```cpp
pixel_t pixel_value; // int...XPCS processing would sparsify (remove 0 photon data) and save the data in the same sparse format than the one used for saving the peaks in the SMX processing. The format we use is basically 3 vectors:
```cpp
pixel_t pixel_value; // intensity
int pixel_idx; // 1D index in the image
int frame_idx; // index of the beginning of the image in pixel_idx
```
An empty frame is just one entry in the `frame_idx` vector:
```
frame_idx = [0, 25, 36, 36, 54, 98... ]
^
frame #3 is an empty frame (same pixel index as the previous one)
```
**TODO**
- [ ] The `writer_sparse` needs to be refactored for this use case -the current implementation is too specific to SMX. The SMX specific part would be moved to the SMX processing.
- [ ] A sparsifier algorithm / node needs to be implemented. Implementations already exist in the benchmarks.
- [ ] A `sparse_frame`, similar to the one designed for SMX but without background, needs to be implemented. Refactoring needed.https://gitlab.esrf.fr/limagroup/lima2/-/issues/72Geometry Transformation node2020-09-29T16:23:54+02:00Samuel DebionneGeometry Transformation nodeRelated to #50
Add a `roi_flip_rot_node`Related to #50
Add a `roi_flip_rot_node`Samuel DebionneSamuel Debionnehttps://gitlab.esrf.fr/limagroup/lima/-/issues/122Follow-up from "Fix HDF5 parallel saving": formalize SaveContainer::Handler2020-09-03T16:51:41+02:00Alejandro Homs PuronFollow-up from "Fix HDF5 parallel saving": formalize SaveContainer::HandlerThe following discussion from !167 should be addressed:
- [ ] @alejandro.homs started a [discussion](https://gitlab.esrf.fr/limagroup/lima/-/merge_requests/167#note_77915):
> The problem is fixed for `HDF5`, but other `SaveContain...The following discussion from !167 should be addressed:
- [ ] @alejandro.homs started a [discussion](https://gitlab.esrf.fr/limagroup/lima/-/merge_requests/167#note_77915):
> The problem is fixed for `HDF5`, but other `SaveContainer`s are not state-less, like `EDF`.
>
> By formalizing `SaveContainer::Handler` we avoid this kind of issuesv1.10.0Alejandro Homs PuronAlejandro Homs Puronhttps://gitlab.esrf.fr/limagroup/lima/-/issues/111CmdThread do not catch and forward exceptions to the main thread2020-08-28T10:41:35+02:00Samuel DebionneCmdThread do not catch and forward exceptions to the main threadMany camera plugins use `CmdThread` to make commands that take a long time asynchronous.
Commands (see `execCmd`) are executed in `cmdLoop()` but there is no `try catch` block to handle exceptions, which means that the loop (and then th...Many camera plugins use `CmdThread` to make commands that take a long time asynchronous.
Commands (see `execCmd`) are executed in `cmdLoop()` but there is no `try catch` block to handle exceptions, which means that the loop (and then thread) is exited when an exception if thrown from a command. The plugin is then in dangling state, unable to execute any commands and a restart is required.
A possible implementation would be to catch and store the exception in the `cmdLoop()` and pop and rethrow it in the next call to `waitStatus()` / `waitNotStatus()`.
Most calls to `sendCmd` are followed by `waitStatus()` / `waitNotStatus()`.Samuel DebionneSamuel Debionnehttps://gitlab.esrf.fr/limagroup/lima2/-/issues/50Legacy (lima1) Processing Pipeline2020-09-29T16:23:54+02:00Samuel DebionneLegacy (lima1) Processing PipelineA Lima1-like pipeline with the following processing:
```
-> ImageAcquired
| Intensity reconstruction->Free Hw Buffer
| Binning->Free Hw Buffer
| ROI
| Flip
| Rotation
| PartialAssemble->Free Hw Buffer
| Buffer(queue_node)
| Imag...A Lima1-like pipeline with the following processing:
```
-> ImageAcquired
| Intensity reconstruction->Free Hw Buffer
| Binning->Free Hw Buffer
| ROI
| Flip
| Rotation
| PartialAssemble->Free Hw Buffer
| Buffer(queue_node)
| Image Assemble(multifuncion_node)
->ImageAssembled
| Accumulation
: SaturationMask: Given an intensity threshold
->Image with Nb of saturations per pixel per Accumulated image
: SaturationCallback: Given a threshold on the nb of pixels
->Event when the number of saturated pixels is above the threshold
->BaseImageReady
| Background - Substraction
| Flatfield - Correction
| Mask
| User - Transformation
->ImageReady
+ Saving
->ImageSaved
+ VideoImage
+ RoiCounters
+ Roi2Spectrum
+ BeamCalculations
+ User - Calculation
->CountersReady
```
### TBB nodes
- [ ] Intensity correction node (low priority)
- [ ] Image assembly nodes: geometry (flip, rotation, roi), see #18
- [ ] Saving node, see #22
- [ ] Event node (e.g. ImageAcquired, ImageReady)v0.1.0Alejandro Homs PuronAlejandro Homs Puronhttps://gitlab.esrf.fr/limagroup/lima2/-/issues/37Processing in Python2020-06-08T19:00:32+02:00Samuel DebionneProcessing in PythonEnable processing tasks to be written in python.
There are multiple options to consider:
- embed the python interpreter in the processing C++ executable
- write the processing in python (using `mpi4py` and/or Boost.MPI python binding)...Enable processing tasks to be written in python.
There are multiple options to consider:
- embed the python interpreter in the processing C++ executable
- write the processing in python (using `mpi4py` and/or Boost.MPI python binding)
* [ ] Check if Python interpreter can be called from multiple (TBB) threadsSamuel DebionneSamuel Debionne2020-06-15https://gitlab.esrf.fr/limagroup/lima2/-/issues/32Static Code Analysis2020-01-08T16:32:27+01:00Samuel DebionneStatic Code AnalysisMany options available these days:
* [clang-tidy](http://clang.llvm.org/extra/clang-tidy/): is a clang-based C++ “linter” tool. Its purpose is to provide an extensible framework for diagnosing and fixing typical programming errors, lik...Many options available these days:
* [clang-tidy](http://clang.llvm.org/extra/clang-tidy/): is a clang-based C++ “linter” tool. Its purpose is to provide an extensible framework for diagnosing and fixing typical programming errors, like style violations, interface misuse, or bugs that can be deduced via static analysis. clang-tidy is modular and provides a convenient interface for writing new checks.
* [iwyu](https://include-what-you-use.org/) (include what you use) : This parses C++ source files and determines the exact include files required to compile that file, no more no less.
* lwyu (link what you use): This is a built in CMake feature that uses options of ld and ldd to print out if executables link more libraries than they actually require.
* [cpplint](https://github.com/cpplint/cpplint): a C++ style checker following Google’s C++ style guide.
* [cppcheck](http://cppcheck.sourceforge.net/): a static analysis tool for C/C++ code. Cppcheck primarily detects the types of bugs that the compilers normally do not detect. The goal is to detect only real errors in the code (i.e. have zero false positives).
They are all pretty well integrated with CMake:
* `<LANG>_CLANG_TIDY` can be set with cache variable `CMAKE_<LANG>_CLANG_TIDY`
* `<LANG>_CPPCHECK` can be set with cache variable `CMAKE_<LANG>_CPPCHECK`
* `<LANG>_CPPLINT` can be set with cache variable `CMAKE_<LANG>_CPPLINT`
* `<LANG>_INCLUDE_WHAT_YOU_USE` can be set with cache variable `CMAKE_<LANG>_INCLUDE_WHAT_YOU_USE`
* `LINK_WHAT_YOU_USE` can be set with cache variable `CMAKE_LINK_WHAT_YOU_USE`
## Clang Tidy
* [x] Agree on a `.clang-tidy` configuration
* [x] Add a CMake option to enable `clang-tidy` on demand
* [ ] Apply clang-tidy suggestions to the codebase when sensible
## Include-what-you-use
* [ ] Add a CMake option to enable `iwyu` on demand
* [ ] Apply iwyu suggestions to the codebase when sensibleSamuel DebionneSamuel Debionnehttps://gitlab.esrf.fr/limagroup/lima2/-/issues/30SmartPix Camera Plugin2020-04-17T17:29:40+02:00Samuel DebionneSmartPix Camera Plugin* [x] Conda package for libsmpx
* [x] Implement `smpx` class, a the low level API to hide dance commands details (mapping command one-to-one)
* [x] Implement camera plugin with slow "controller image transfer"
* [ ] Implement camera ...* [x] Conda package for libsmpx
* [x] Implement `smpx` class, a the low level API to hide dance commands details (mapping command one-to-one)
* [x] Implement camera plugin with slow "controller image transfer"
* [ ] Implement camera plugin with fast RASHPA image transferSamuel DebionneSamuel Debionne2020-07-01https://gitlab.esrf.fr/limagroup/lima2/-/issues/29License and Copyright2020-05-06T08:41:24+02:00Laurent ClaustreLicense and CopyrightI propose to use a LGPLv3 license. The copyright is not intended to be held by staff members but ESRF (or BCU/ESRF), you can add a AUTHORS file if you want to be mentioned as developer. We could have a CONTRIBUTORS files as well.I propose to use a LGPLv3 license. The copyright is not intended to be held by staff members but ESRF (or BCU/ESRF), you can add a AUTHORS file if you want to be mentioned as developer. We could have a CONTRIBUTORS files as well.v0.1.02020-05-15https://gitlab.esrf.fr/limagroup/Lima-camera-slsdetector/-/issues/18Diagnostic the flow control failure on ID10 / ID02 Eiger-500k2020-04-01T14:17:16+02:00Samuel DebionneDiagnostic the flow control failure on ID10 / ID02 Eiger-500kTest the hardware script sent by Martin.Test the hardware script sent by Martin.Alejandro Homs PuronAlejandro Homs Puronhttps://gitlab.esrf.fr/limagroup/lima2/-/issues/22I/Os2020-04-28T15:06:44+02:00Samuel DebionneI/OsI/Os are standard processes that can be inserted in the processing pipeline:
* [x] Saving to file (HDF5)
* [ ] Saving to buffer
* [ ] Streaming (MPI Streams, ZMQ)
* [ ] Object storage (Memcached)I/Os are standard processes that can be inserted in the processing pipeline:
* [x] Saving to file (HDF5)
* [ ] Saving to buffer
* [ ] Streaming (MPI Streams, ZMQ)
* [ ] Object storage (Memcached)v0.1.0Samuel DebionneSamuel Debionne2020-04-24https://gitlab.esrf.fr/limagroup/lima2/-/issues/18Image Assembly2022-04-04T13:42:55+02:00Samuel DebionneImage AssemblyThe most generic approach to image assembly (reconstruction) is the definition by the HW plugin of a (set of) function(s) performing the task(s), to be called in dedicated processing node(s).
The system can then provide helpers that imp...The most generic approach to image assembly (reconstruction) is the definition by the HW plugin of a (set of) function(s) performing the task(s), to be called in dedicated processing node(s).
The system can then provide helpers that implement the function(s) based on:
* [ ] Detector Layout description in camera plugin
Low priority tasks:
* [ ] HW geometric transformation (i.e. flip, roi, bin)Alejandro Homs PuronAlejandro Homs Puronhttps://gitlab.esrf.fr/limagroup/lima2/-/issues/15Python Binding2020-04-17T17:48:30+02:00Samuel DebionnePython BindingSIP has three major drawbacks:
- Requires a preprocessor (sip -> cpp) that complicates the build system
- Uses a python module at runtime that poses all gkind of compatibility issues
- Verbose SIP files with duplicated C++ code
* ...SIP has three major drawbacks:
- Requires a preprocessor (sip -> cpp) that complicates the build system
- Uses a python module at runtime that poses all gkind of compatibility issues
- Verbose SIP files with duplicated C++ code
* [x] Evaluate pybind11
* [ ] Provide a generic binding for every camerasv0.1.0Samuel DebionneSamuel Debionnehttps://gitlab.esrf.fr/limagroup/lima2/-/issues/14Hardware Interface2020-04-17T17:21:25+02:00Samuel DebionneHardware InterfaceThe hardware interface should be split into control and acquisition interfaces.
* [ ] Survey on the SDKs for cameras that should be supported in distributed mode
* [ ] API for the control part of the Hardware Interface
* [ ] API for the...The hardware interface should be split into control and acquisition interfaces.
* [ ] Survey on the SDKs for cameras that should be supported in distributed mode
* [ ] API for the control part of the Hardware Interface
* [ ] API for the acquisition part of the Hardware InterfaceSamuel DebionneSamuel Debionne