CHANGELOG.rst 19 KB
Newer Older
1
2
3
CHANGELOG
=========

Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
4
0.6.1: 2021/09/29
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
5
6
7
8
9
10
11
-----------------
* Downgrade pypushflow to 0.1.0 for compatibility issues (MR !137)
* Core (MR !137):
    * `dataset.py`:
        * nica_nmf: Set 'init' as 'custom' when initializing NMF to use H and W absoulte values from NICA.
        * Change `cascade` to `waterfall`
        * Mosaicity only computable in two dimensions.
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
12
13
14
15
16
        * `compute_transformation`: Flip axes if rotate is checked (MR !138)
        * Add `Transformation` class (MR !138)
    * `mapping.py` (MR !138):
        * rsm: Convert to degrees and fix computation typo.
        * magnification: Convert to radians before computing sin and cos and add topography option.
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
17
18
* Decomposition (MR !137):
    * `nica.py`: Add epsilon to avoid division by 0.
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
19
20
21
22
23
* GUI:
    * `blindSourceSeparationWidget.py`: Change NNICA to NICA (MR !137)
    * `shiftCorrectionWidget.py`: Convert 'shift' attribute to numpy array after setter (MR !137)
    * `magnificationWidget.py`: Add topography checkbox and orientation axis combobox (MR !138)
    * `grainPlotWidget.py` & `zSumWidget.py`: Rotate image 90 degrees in case of active rotation (rsm), and set label corresponding to the transformation type (if so) (MR !138)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
24
25
26
* orangecontrib (MR !137):
    * `shiftcorrection.py`: Fix bug when clearing stack and loading and saving shift.

Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
0.6.0: 2021/09/14
-----------------
* Core:
    * Dataset:
        * Save data into hdf5 instead of numpy files, all images are saved into a single hdf5 file (MR !128):
            * `Data` class methods `apply_funcs` and `save` are modified to work with h5py and to only replace the necessary rows of the file.
            * `DataUrls` used in `Dataset` are linked to the corresponding image (using data slicing of h5py).
            * Temporary datasets are used to not lose information in case of crash or abortion.
        * Raise error if directory for treated data is read-only (MR !114)
        * Use treated data directory for dark files (MR !114)
        * Add `roi_dir` parameter in `apply_roi` as directory path for the new dataset (MR !122)
        * New treated files in disk automatically replace previous ones except said by parameter (MR !123)
        * Add `get_metadata_value` method to obtain metadata info from a key (MR !124)
        * Add attribute `transformation` with the transformation values and method `compute_transformation` to compute it (MR !125)
        * Modify `find_shift` and `apply_shift` methods parameters to use `steps` instead of `h_step` and `h_max` and use linear shift when applying shifts greater than 1 (MR !127)
        * Add methods `find_shift_along_dimension` and `apply_shift_along_dimension` that find and apply shift, respectively, in a loop along a dimensions values (MR !134)
    * Mapping:
        * Apply median filter to COM and std to remove Nans (MR !117)
        * Add functions `compute_rsm` and `compute_magnification` (MR !125)
    * Image registration:
        * Modify shift detection to work with higher values of shift than 0.5 (MR !127)
    * Process (MR !130):
        * Add new process `_GrainPlot`
        * Add new process `_Transformation`
        * Add step and chunk properties in `_NoiseRemoval` process
    * ROI:
        * Fix typo when computing 2D ROI (MR !130)
* GUI:
    * `grainPlotWidget.py`:
        * Orientation distribution: Increase resolution of hsv key, set coordinate labels as the angles of the motors, add checkbox to center angle values to 0 and fix bugs (MR !116)
        * Add export button that saves maps into Nexus file (MR !132)
        * Stop using opticolor (MR !117)
        * Modify axes if transformation is present and set global attributes origin and scale (MR !125)
    * `rockingCurvesWidget.py`:
        * Add two scatter points at the frame number position in the rocking curves (MR !119)
    * `noiseRemovalWidget.py`:
        * Fix bug when resetting data and applying new operations (MR !115)
        * Only show abort button if abortion is possible (MR !128)
        * Add step and chunks properties (MR !130)
    * `roiWidget.py`:
        * Return bg_dataset in `get_dataset` (MR !122)
        * Use same directory for consecutive roi applies (MR !122)
        * Only show abort button if abortion is possible (MR !128)
    * `displayComponentsWidget.py`: Set X values as motor positions when filtering by dimensions (MR !121)
    * `linkComponentsWidget.py`: Add colormap (orientation distribution maps) from the components (MR !126)
    * Add `rsmWidget.py` for Reciprocal Space Map 1D datasets (MR !124)
    * Add `magnificationWidget.py` for dataset with magnification values (MR !125)
    * `shiftCorrectionWidget.py`:
        * Add `ShiftCorrectionDialog` which includes `ShiftCorrectionWidget` as main widget (MR !127)
        * Add `_filtered_shift` attribute that contains the shift values, if any, along a dimension (MR !134)
        * Use methods for finding and applying shift along a dimension in `Dataset` (MR !134)
        * Modify checkbox to be used when shift should be applied to only selected value (MR !134)
        * Update values of vertical and horizontal shifts to show currently selected dimension value shift (MR !134)
    * Create `zSumWidget` to show plot of zsum (MR !131)
* pffaddon: Add alias for `GrainPlotWidgetOW`, for `TransformationWidgetOW` (MR !130) and for `zSumWidgetOW` (MR !131)
* app: Modify `ows_to_script.py` executable script arguments for Dataset (MR !130)
* orangecontrib:
    * Add `transformation.py` that shows either `magnificationWidget` or `RSMWidget` (MR !125, MR!133)
    * Add step and chunks Settings in `NoiseRemovalWidgetOW` (MR !130)
    * Add local in `shiftCorrectionWidgetOW` to avoid bugs depending on the local computers locale (MR !130)
    * Use `zSumWidget` in `zSumWidgetOW(MR !131)

Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
89
0.5.0: 2020/12/23
90
-----------------
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
91
* Add tqdm package to requirements.txt and setup.py
92
93
* Core:
    * Dataset:
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
94
        * Update constructor to admit `first_filename` parameter. (MR !106)
95
96
        * Fix reshaping: Correct the methodology used for reshaping to be consistent with the way data is obtained in id06 (MR !101)
        * Modify methods that work with dimensions (get_data, apply_shift) so that they work if more than one dimension is fixed (MR !102)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
97
        * Add `running_data` property which contains the data currently being used by any of the operations (if any). This data is stopped in case of `Abort` option called (MR !95)
98
99
100
101
102
        * apply_background_subtraction:
            * Use multiprocessing to chunk images (MR !87)
            * Add step option to compute median using only every step image (MR !87)
            * Add tests with step and chunks and add docstring (MR !92)
        * Add `apply_moments` method for computing orientation distribution and colorkey based on the dimensions (for now only works with two dimensions) (MR !96)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
103
104
        * Add `apply_fit` method for fitting the rocking curves of the data (a curve corresponds to a pixel around the stack), multiprocessing is used to chunk the images (MR !98)
        * Add `apply_threshold` method for thresholding the data (MR !108)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
105
106
    * Create new file `dimension.py` that contains the classes AcquisitionDims and Dimensions moved from `dataset.py` (MR !105)
    * Create new file `mapping.py` with several functions: fit a rocking curve and computation of moments (MR !98)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
107
    * `process.py`: Add new process `_RockingCurves` (MR !104)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
108
    * Add `test_dimension.py` file that contains the tests that use the Dimension and AcquisitionDim classes (MR !105)
109
* GUI:
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
110
    * Add setters and getters for colormaps in widgets that use them (MR !110)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
111
    * Disable buttons when computing operations (MR !109)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
112
113
114
    * `datasetSelectionWidget.py` (MR !106)
        * Add new tab for treated data.
        * Create class `DirSelectionWidget`
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
115
    * `noiseRemovalWidget.py`:
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
116
        * Add `setDataset` method in `NoiseRemovalDialog` to enable buttons after the dataset is set (MR !100)
117
118
        * Add widgets only visible when data in disk, which give option to use step or chunks when computing the median (MR !87, MR!90)
        * Fix bug when showing or not the in disk widget (MR !97)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
119
120
        * Make methods `toggleChunks` and `toggleInDiskWidget` private (MR !100)
        * Correctly set and get properties (background, method, etc) (MR !104)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
121
122
123
        * Add threshold removal option (MR !108)
        * Add signal that emits when method starts or ends computing (MR !109)
        * Fix some typos (MR !111)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
124
    * `shiftCorrectionWidget.py`:
125
126
        * Use `skimage.registration.phase_correlation` instead of `register_translation` when finding shift (MR !91)
        * Fix bug when shifting only using first dimension (MR !99)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
127
        * Round shift to 5 decimals (MR !105)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
128
    * `roiSelectionWidget.py`:
129
        * Run ROI operation in thread and add abort button(MR !95)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
130
    * `blindSourceSeparationWidget.py`:
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
131
        * Modify `Method` class so that its values are a description of the method. The keys are used in the `BlindSourceSeparationWidget` as text in the combobox, and the values are its tooltips (MR !100)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
132
        * Remove widgets to set a maximum number of components (MR !105)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
133
134
135
    * `pca.py` (MR !105):
        * Add title and labels to plot
        * Automatically compute PCA when creating widget and remove widgets to set a maximum number of components
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
136
137
    * `displayComponentsWidget.py`:
        * Add parameter in `setDimensions` to know the shape of the data, and use it to correctly take values from W (MR !107)
138
139
140
141
142
    * `ChooseDimensionWidget`: Add as many dimension widgets as dimensions are (minus 1), so all the dimensions are fixed except for one (MR !102)
    * Create `GrainPlotWidget`:
        * Show different COM, FWHM, Skewness, Kurtosis of the dataset around a certain dimension (MR !96)
        * Add a contours map to show the contours of the orientation distribution on top of the colorkey (MR !96)
        * Add a plot for the mosaicity map (only works with two dimensions) (MR !96)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
143
    * Create `RockingCurvesWidget` (MR !98, MR !111, MR !112):
144
145
146
147
        * Show the stack of images where the user can click any pixel.
        * Show a 1D plot with the rocking curve of the pixel selected in the stack.
        * A ChooseDimensionWidget allows the data to be filtered by the chosen dimension.
        * A button to fit the data shows a plot with 3 maps computed from the fitted data: Integrated intensity, FWHM and peak position.
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
148
        * If data is filtered by dimensions then the data is fitted in chunks, where every chunk is a certain value of the chosen dimension.
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
149
150
    * Create `LineProfileWidget` (MR !103):
        * Upload a *.npy image from disk and show intensity curve from a chosen y value of the image.
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
151
    * Create `ShowStackWidget` to show stack of data (MR !106)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
152
* pffaddon: Add alias for rockingCurveWidget (MR !104)
153
154
155
* orangecontrib:
    * Zsum: Use `ChooseDimensionDock` instead of `ChooseDimensionWidget` and fix bug when computing sum in disk (MR !92)
    * Add `GrainPlotWidgetOW` to show a `GrainPlotWidget` (MR !96)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
156
157
    * Add `RockingCurvesWidgetOW` to show a `RockingCurvesWidget` (MR !98)
    * Add `LineProfileWidgetOW` for `LineProfileWidget` (MR !103)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
158
159
    * Add properties in `NoiseRemovalWidgetOW` for threshold removal (MR !108)
    * Add colormap as input and output on widgets that use it (MR !110)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
160

Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
161
162
163
164
0.4.2: 2020/09/15
-----------------
* Use version 4.1.2.30 of opencv to avoid uncompatibility with PyQt5 latest release

165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
0.4.1: 2020/09/14
-----------------
* Modify `setup.py` (MR !82):
    * Add `opencv-python`, `scikit-image` and `silx` in `install_requires`.
    * Add package `full` in `extras` with `matplotlib`, `PyQt5` and `orange`.
    * Add package `test` in `extras` with `pillow`.
* Core:
    * `dataset.py`:
        * Add parameter bins when partitioning data in `Dataset` (MR !67)
        * Create `in_memory` setter in `Dataset` (MR !68)
        * Improve methods `flatten`, `reshape` and `take` in `Data` (MR !67)
        * Add try-except when fitting data in blind source separation methods to close file in case of crash (MR !71)
        * Add methods to convert dimensions to dictionary and viceversa in `AcquisitionDims` (MR !74)
        * Add `to_memory` method in `Dataset` to load only part of the data into memory (MR !77)
        * Add property `ndim` in `Data` (MR !80)
* Decomposition:
    * Fix bug in IPCA with cases where in the last iteration the chunk was smaller than the number of components (omit last iteration) (MR !73)
* GUI:
    * Temporarily register resources directory in when used in `DisplayComponentsWidget` (MR !84)
    * Change colormap default normalization to `log` for better visualization of the data (MR !85)
    * Add widget for bins in `DataPartitionWidget` (MR !67)
    * Add parameter for chunks in `NoiseRemovalWidget` (MR !68)
    * Add abort options in `dataPartitionWidget`, `NoiseRemovalWidget` and `ShiftCorrectionWidget` (MR !68)
    * Get next free axis when removing dimensions in `DimensionsWidget (MR !86)
    * Fix bug when clearing stack in `RoiSelectionWidget` (MR !67)
    * Fix bug when getting filtered indices in `DataPartitionWidget` (MR !68)
* IO:
    * New file `dataset_io.py` with functions to save and load data into/from json files (MR !74)
* Orangecontrib:
    * Remove progressBar due to bugs in the workflows (MR !84)
    * Fix bug with qthreads when creating and deleting the thread using orange widgets (MR !70)

Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
0.4.0: 2020/05/15
-----------------
* Refactors the project (MR !48)
    * Changes default colormap to cividis.
    * Core:
        * Dataset class stops inheriting from `silx.qt.QObject`
        * The dataset class takes control of all the operations done to the data.
        * Metadata is obtained from the frames using `fabioh5` from silx and then each frame is closed to spare space in memory.
        * Creates new class `Data` that inherits from numpy.ndarray and contains the corresponding urls and metadata of the data.
        * Adds new methods to `Dataset` for image operations, image registration, region of interest, and partition by intensity (filter data), that are applied differently depending on if the data is loaded into memory or is taken from disk in chunks.
        * Data can be saved into an Hdf5 file with images in the rows.
        * Implement in `Data` methods `shape`, `reshape`, `flatten`, `take`, `__getitem__` that work similarly to a numpy.ndarray.
        * Adds and modifies functions in `imageOperations.py` and `imageRegistration.py` to be used when data  is not loaded into memory.
        * Adds class `IPCA` in `blindSourceSeparation.py` that uses IncrementalPCA from scikit-learn to apply PCA in chunks.
        * Implements methods in `Dataset` to apply blind source separation algorithms to the data.
        * PCA and NMF algorithms used from scikit-learn when data is loaded into memory.
    * Decomposition:
        * A new package `decomposition` is created.
        * A base class `Base` is created for decomposition methods.
        * Adds class `NMF` that updates the matrices `H` and `W` using NMF method without having the data in memory.
        * Adds class `NICA` with the already implemented algorithm in `core`.
        * Adds class `PCA` currently not used (substitued for PCA in scikit-learn).
    * GUI:
        * Modifies all widgets to only have `Dataset` objects and call its corresponding methods.

* Core:
    * Adds method `in_memory` to Dataset to upload data from disk or remove data from memory (MR !51)
    * Adds method that returns intensity per frame (MR !58)
    * Number of bins can be passed as parameter when partitioning the data by intensity (MR !52)
    * Fix bugs when not closing Hdf5 files (MR !56)
    * Adds methos `zsum` in `Dataset` to sum images when not in memory (MR !57)
    * Fix bugs with `__getitem__` in `Data` (MR !57)

* Decomposition:
    * Adds IPCA to decomposition package (MR !50)
    * NICA implements IPCA for whitening when data is not in memory. Fix bugs when computing IPCA with data in the rows or in the columns (MR !53)
    * Allow matrices H and W in IPCA to be stored in disk using Hdf5 (MR !54)

* GUI:
    * Adds ComboBox in `DatasetSelectionWidget` to use data from disk or load it into memory (MR !57)
    * Adds widget to partition the data by intensity (MR !58)
    * Removes ComboBox in `DatasetSelectionWidget` to filter the data (MR !59)

Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
240
241
242
243
* Orangecontrib:
    * Add corresponding data partition widget (MR !58)
    * Update example tutorial (MR !64)

244
245
246
247
0.3.0: 2020/03/03
-----------------
* Adds resources package (MR !29)
* Adds configuration file for default parameters (MR !30)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
248
* Minor fixes (MR !12 !14 !15 !32 !37 !41 !46)
249
250
* Use silx backend when available (MR !44)
* Core:
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
251
    * Adds tolerance to dimensions that defines the uniqueness of its values (MR !17)
252
253
    * Modifies ROI performance (MR !23)
    * Implements blind source separation algorithms (MR !18 !20 !22 !28)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
254
    * Implements algorithms for linking components between datasets: ORB, SIFT, Euclidean distance (MR !36 !44 !45)
255
256
* GUI:
    * Creates a unified noise removal widget (MR !21 !24 !25)
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
257
    * Adds checkbox to filter the data in `DataSelectionWidget` (MR !19)
258
259
260
261
262
263
264
265
    * Improves `DimensionWidget` (MR !16 !17 !33)
    * Improves `roiSelectionWidget` (MR !23 !31 !43)
    * Implements widgets `blindSourceSeparationWidget` and `displayComponentsWidget` to display the components (MR !18 !22 !26 !29 !35)
    * Creates `PCAWidget` to show the eigenvalues of the PCA computation (MR !22)
    * Creates widget `LinkComponentsWidget` that implements the component matching algorithms from two sets of components (MR !36 !38)
* IO:
    * Creates functions for reading and writing the components of the BSS into Nexus files (MR !42)

Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
266
267
0.2.0: 2019/10/15
-----------------
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
268
269
* Renames project linkdick06 to darfix
* Refactors project into core, gui, io and test packages
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
270
271
272
* Adds unittests
* Adds orangecontrib package
* Adds examples package
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
* Adds continous integration with gitlab-ci
* Adds requirements.txt file with dependencies on numpy, silx, PyQt5, opencv-python, scikit-image
  and orange3 with version 3.22.0
* Core:
    * Dataset class inherits from `silx.qt.QObject`
    * Adds threshold to split data into high intensity images and low intensity images
    * Adds `Dimension` class to save information about how the frames are taken through the dataset
    * Adds roi file that implements region of interest functions
    * Image registration operations moved to new file `imageRegistration.py`
    * Adds shift detection using OpenCV
    * Adds `autofocus.py` to use at the shift detection
    * Adds `geneticShiftDetection` file to improve shift detection using a genetic algorithm
* GUI:
    * Adds `RoiSelectionWidget`
    * Adds `ShiftCorrectionWidget` to apply shift detection and shift correction
    * Adds widgets `BackgroundSubstractionWidget` and `HotPixelRemovalWidget` for noise removal
    * Adds widgets for data selection, metadata, and choosing dimensions
    * Adds a `QThread`, `operationThread`, to run widgets operations in a thread
    * Adds utils to create datasets from scratch
    * Removes `frameInterpretationWidget` and adds `DimensionWidget`
* IO:
    * Adds utils
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
295

Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
296
297
298
299
300
0.1.0: 2019/07/11
-----------------
* Adds project build, documentation and test structure
* Adds continuous integration set-up for Gitlab
* Adds EDF reader from Fabio
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
301
* Adds operations for correction of images in imageOperations
Julia Garriga Ferrer's avatar
Julia Garriga Ferrer committed
302
* Adds notebook tutorial