Commit 53880cab authored by Carsten Richter's avatar Carsten Richter

Merge branch 'doc' into 'master'

Update documentation

Closes #41

See merge request !109
parents 65c5bfdc 5f8bcfb7
Pipeline #7083 passed with stages
in 4 minutes and 12 seconds
Change Log
==========
Unreleased
2018/12/11
----------
* Command line:
- Add command `xsocs concat` to merge multiple HDF5 master files into one (MR: !54)
- Add option `--numcores` to set number of cores to use, e.g., `xsocs gui --numcores 2` (MR: !78)
- Add option `--no-3d` to disable OpenGL: `xsocs gui --no-3d` (MR: !84)
* Merge:
- Add image ROI support to only save part of input images (MR: !60)
- Read calibration and energy from spec when available (MR: !65)
- Allow to merge inconsistent commands (MR: !56)
* Intensity view:
- Allow to sort scans by any positioner not just eta (MR: !58)
- Add colorbar and option to change scatter symbols and size (MR: !64)
- Add selection of normalization (MR: !53)
- Shift editor: Allow to display any measurement rather than intensity (MR: !97)
* QSpace conversion:
- Add QSpace spherical coordinates system (MR: !89)
- Add image mask (MR: !59, !66)
- Add maxipix correction (MR: !69)
- Add multiple energies scan support (MR: !94, !98)
- Add optional normalization (MR: !53)
- Provide a default number of bins for QSpace histogram (MR: !73)
- Allow to override energy and calibration !50
- Update helper API (MR: !90)
- QSpace view: Add a stack view of the QSpace as an alternative to 3D view (MR: !72)
* Fit:
- Add background subtraction of constant and 'snip' background (MR: !85, !86, !92)
- Improve QSpace projection on axes: normalize after projection (MR: !101, !102)
* Compatibility:
- Fix Python3 compatibility issues (Merge requests (MR): !44, !46, !51)
- Add support of PyQt5 and drop PyQt4 support (MR: !61)
- Deprecates Python2 support
- Add dependency to fabio for EDF file reading (MR: !71, !77)
- Add Windows support (MR: !74)
* Miscellaneous:
- GUI: Usability improvements (MR: !47, !48, !49, !55, !68, !83, !95)
- Tests: Use gitlab-ci for continuous integration on Linux (MR: !76)
- Minor bug fixes (MR: !45, !70, !80, !91, !96, !99)
- Clean-up, code style and project structure (MR: !62, !63, !82, !87, !88, !93)
- Update documentation (MR: !52, !79)
- Update to newer versions of dependencies (MR: !81)
include README.rst
include CHANGELOG.rst
include copyright
include LICENSE
include MANIFEST.in
include run_tests.py
include version.py
include build-deb.sh
include requirements.txt
include requirements-dev.txt
recursive-include xsocs *.pyx *.pxd *.pxi
recursive-include xsocs *.h *.c *.cpp
recursive-include doc/source *.py *.rst *.png *.ico
recursive-include doc/source *.py *.rst *.png *.ico *.ipynb
global-exclude .ipynb_checkpoints/*
xsocs
=====
X-SOCS
======
The X-ray Strain Orientation Calculation Software (X-SOCS) is a user-friendly software,
developed for automatic analysis of 5D sets of data recorded during continuous mapping measurements.
......@@ -31,4 +31,4 @@ the `video tutorials <http://kmap.gitlab-pages.esrf.fr/xsocs/tutorials.html>`_.
License
-------
The source code of xsocs is licensed under the MIT license.
The source code of X-SOCS is licensed under the `MIT license <https://gitlab.esrf.fr/kmap/xsocs/blob/master/LICENSE>`_.
This diff is collapsed.
......@@ -14,10 +14,11 @@ films, surfaces or even embedded structures.
install.rst
using.rst
tutorials.rst
modules.rst
troubleshooting.rst
:doc:`install`
How to install *X-SOCS* on Linux
How to install *X-SOCS*
:doc:`using`
How to use *X-SOCS* to reduce KMAP data
......@@ -25,12 +26,16 @@ films, surfaces or even embedded structures.
:doc:`tutorials`
Video tutorials
:doc:`modules`
Documentation of the script API
:doc:`troubleshooting`
When things do not work as expected
Project Resources
- `Source code repository <https://gitlab.esrf.fr/kmap/xsocs>`_
- `Issue tracker <https://gitlab.esrf.fr/kmap/xsocs/issues>`_
Resources:
- `Source code repository <https://gitlab.esrf.fr/kmap/xsocs>`_
- `Issue tracker <https://gitlab.esrf.fr/kmap/xsocs/issues>`_
Indices and tables
==================
......
......@@ -14,12 +14,14 @@ The dependencies of *X-SOCS* are:
* `Python <https://www.python.org/>`_ >=3.4 or 2.7 (Note: The Graphical user interface is only tested with python3).
* `numpy <http://www.numpy.org>`_
* `h5py <http://www.h5py.org/>`_
* `silx <https://pypi.python.org/pypi/silx>`_
* `fabio <https://pypi.org/project/fabio/>`_
* `silx <https://pypi.org/project/silx>`_
* `xrayutilities <https://xrayutilities.sourceforge.io/>`_
* `scipy <https://pypi.python.org/pypi/scipy>`_
* `PyOpenGL <http://pyopengl.sourceforge.net/>`_
* `matplotlib <https://matplotlib.org/>`_
* `PyQt5 <https://riverbankcomputing.com/software/pyqt/intro>`_
* `setuptools <https://pypi.org/project/setuptools/>`_
In addition, OpenGL 2.1 is required for the 3D view of the QSpace.
......@@ -39,19 +41,23 @@ Building *X-SOCS* from the source requires some `Build dependencies`_.
Building from source
++++++++++++++++++++
Download the source from the `gitlab repository <https://gitlab.esrf.fr/kmap/xsocs.git>`_::
Clone the source `repository <https://gitlab.esrf.fr/kmap/xsocs.git>`_::
git clone https://gitlab.esrf.fr/kmap/xsocs.git
Then cd into xsocs::
Or download the source as a `zip file <https://gitlab.esrf.fr/kmap/xsocs/-/archive/master/xsocs-master.zip>`_ and unzip it.
Then go into the xsocs directory::
cd xsocs
And install xsocs, either system-wide (requires root privileges)::
And install xsocs either with minimal dependency (to use from scripts)::
pip install .
pip install . [--user]
or for the current user only::
or with all graphical user interface dependencies::
pip install . --user
pip install .[gui] [--user]
.. note::
The ``--user`` optional argument installs X-SOCS for the current user only.
===============
API Reference
===============
X-SOCS data processing is implemented in the :mod:`xsocs.process` package.
It provides:
- Merge SPEC+EDF to HDF5: :mod:`xsocs.process.merge`
- Q Space Conversion: :mod:`xsocs.process.qspace`
- Fit/COM: :mod:`xsocs.process.fit`
:mod:`xsocs.process.merge`: SPEC+EDF to HDF5
============================================
.. automodule:: xsocs.process.merge
:func:`~xsocs.process.merge.merge_scan_data`
--------------------------------------------
.. autofunction:: merge_scan_data
Sample Code
-----------
.. literalinclude:: ../../xsocs/examples/id01_merge.py
:lines: 31-
:mod:`xsocs.process.qspace`: QSpace conversion
==============================================
.. automodule:: xsocs.process.qspace
:func:`~xsocs.process.qspace.kmap_2_qspace`
-------------------------------------------
.. autofunction:: kmap_2_qspace
Sample Code
-----------
.. literalinclude:: ../../xsocs/examples/id01_qspace.py
:mod:`xsocs.process.fit`: Fit/COM
=================================
.. automodule:: xsocs.process.fit
:class:`~xsocs.process.fit.PeakFitter`
--------------------------------------
.. autoclass:: PeakFitter
.. autoclass:: FitTypes
.. autoclass:: BackgroundTypes
Sample Code
-----------
.. literalinclude:: ../../xsocs/examples/id01_peak.py
......@@ -4,13 +4,8 @@ Trouble shooting
Using XSocs through ssh
-----------------------
XSocs is using OpenGL2.1 for QSpace visualization.
When running through ssh, there are a few reasons that can avoid the display of the 3D QSpace visualization:
XSocs is using OpenGL 2.1 widgets from the `silx <http://www.silx.org/doc/silx/latest/>`_ toolkit for QSpace visualization.
When running through ssh, there are a few reasons that can avoid the display of the 3D QSpace visualization.
See `Using OpenGL through ssh <http://www.silx.org/doc/silx/latest/troubleshooting.html#using-opengl-through-ssh>`_ in `silx <http://www.silx.org/doc/silx/latest/>`_ documentation for more information.
- Make sure to use ``ssh -X`` to enable X11 forwarding.
- Both server and client computers must have the same kind of GPU drivers
(either both using proprietary NVidia drivers or both using open source drivers),
otherwise only OpenGL1.4 is available.
- "Indirect GLX" must be enabled on the local computer.
This can be set in the configuration of the display manager or the X server (requires root access).
On Debian 8, for kdm, add ``+iglx`` after ``ServerArgsLocal=...`` in ``/etc/kde4/kdm/kdmrc`` and restart the x server.
\ No newline at end of file
......@@ -30,3 +30,8 @@ Video tutorials
`Part 10: Q workspace Bragg peak fitting or COM <https://www.youtube.com/watch?v=eShYxO1ZF10>`_
`Part 11: Results workspace <https://www.youtube.com/watch?v=JbO9U7d-hPs>`_
.. note::
Since video tutorials were made, the graphical user interface has changed.
Yet the procedure remains valid.
.. _conversion_window:
Qspace conversion window
========================
.. |conversion_dialog| image:: img/conversion_dialog.png
+--------------------+
| Intensity view |
+====================+
| |conversion_dialog||
+--------------------+
This window lets you configure some parameters and options for the
conversion from real space to qspace.
.. |xray_init_area| replace:: experiment.HXRD.Ang2Q
.. _xray_init_area: https://xrayutilities.sourceforge.io/xrayutilities.html#xrayutilities.experiment.HXRD.Ang2Q
+--------------------+--------------------------------------------------------+
| Parameter / option |
+====================+========================================================+
| ``Median filter`` | If checked and set to anything else than [1, 1] then a |
| | median filter will be applied to all images before |
| | conversion. |
| | |
| | .. warning:: if non empty pixels in your images are |
| | few and far between, you might loose to much |
| | information if a median filter is applied. |
+--------------------+--------------------------------------------------------+
| ``Grid dimensions``| dimensions of your qspace cube. |
+--------------------+--------------------------------------------------------+
When all is set, click on ``Convert`` to start the conversion. When the conversion
is done, the result file (hdf5) can be found at the location written in the
``output`` field.
\ No newline at end of file
Q space conversion dialog
=========================
.. figure:: img/conversion_dialog.png
:width: 100%
:align: center
Q space conversion dialog
This window lets you configure some parameters for the conversion from images to Q space:
- `Acq. Parameters`_
- `Image Processing Parameters`_
- `Q space Parameters`_
The right panel allows to review the selected region of interest on the sample and the selected scan entries.
Click on **Convert** to start the conversion.
Once the conversion is done, a :ref:`Q space item <qspace_item>` is added to the :ref:`project tree <project_tree>`.
And the result is displayed in the :ref:`Q sspace view <qspace_view>`.
Acq. Parameters
...............
.. list-table::
:widths: 1 4
:header-rows: 1
* - Parameter
- Description
* - ``Beam energy``
- Energy of the beam, in eV
* - ``Direct beam``
- Position in pixels of the direct beam on the detector
* - ``Ch. per deg.``
- Channels per degree
Image Processing Parameters
...........................
All image processing steps are optional, check the corresponding check box to enable them.
.. list-table::
:widths: 1 4
:header-rows: 1
* - Parameter
- Description
* - ``Maxipix correction``
- Apply Maxipix detector module edges correction
* - ``Mask``
- Select a mask image to apply on input images.
The image MUST have the same size as input detector images.
Non zero value in the mask image discard the corresponding pixels.
* - ``Normalization``
- Select the measurement to use for normalization.
* - ``Median filter``
- Set the size (**w**\ idth and **h**\ eight) of the median filter window.
Q space Parameters
..................
.. list-table::
:widths: 1 4
:header-rows: 1
* - Parameter
- Description
* - ``Grid dimensions``
- Number of bins for each dimension of the Q space histogram.
* - ``Coordinates``
- Select the coordinate system of the QSpace histogram.
Either **Cartesian** or **Spherical**.
doc/source/usage/img/main_view.png

5.02 KB | W: | H:

doc/source/usage/img/main_view.png

7.11 KB | W: | H:

doc/source/usage/img/main_view.png
doc/source/usage/img/main_view.png
doc/source/usage/img/main_view.png
doc/source/usage/img/main_view.png
  • 2-up
  • Swipe
  • Onion skin
doc/source/usage/img/merge_window.png

39.5 KB | W: | H:

doc/source/usage/img/merge_window.png

40.5 KB | W: | H:

doc/source/usage/img/merge_window.png
doc/source/usage/img/merge_window.png
doc/source/usage/img/merge_window.png
doc/source/usage/img/merge_window.png
  • 2-up
  • Swipe
  • Onion skin
doc/source/usage/img/open_dialog.png

27.3 KB | W: | H:

doc/source/usage/img/open_dialog.png

36.9 KB | W: | H:

doc/source/usage/img/open_dialog.png
doc/source/usage/img/open_dialog.png
doc/source/usage/img/open_dialog.png
doc/source/usage/img/open_dialog.png
  • 2-up
  • Swipe
  • Onion skin
.. _intensity_view:
Converting to Q-Space (Intensity view)
======================================
Intensity View
==============
.. |intensity_view| image:: img/intensity_view.png
.. |draw_roi_button| image:: img/draw_roi_button.png
.. |to_qspace_button| image:: img/to_qspace_button.png
.. figure:: img/intensity_view.png
:width: 100%
:align: center
+------------------+
| Intensity view |
+==================+
| |intensity_view| |
+------------------+
Intensity view
This view display the total acquired intensity over the sample.
When an entry is selected in the list on the left, each point on the scatter
plot represents the total intensity of the acquired image at that point.
By default, this view displays the total intensity summed over all scans for each point of the sample.
When the root node (labeled ``Intensity``) is selected, the total intensity of
all entries is displayed.
Select displayed data
---------------------
The list on the left allows you to select the entries that will be used when
converting to q-space.
When an entry is selected in the list on the left, the scatter plot represents the intensity of the acquired images for the selected entry.
When the **Total** item is selected, the total intensity of selected entries is displayed.
You can select points on the central scatter plot to display an intensity
profile at that point at the bottom of the window.
It is possible to change the sorting of the scans by changing the selected positioner in the **Sort by** drop-down list (`eta` by default).
Roi selection
.............
It is possible to normalize the displayed data with the **Normalization** drop-down list.
Before starting the conversion, a ROI (Region Of Interest) must be selected
on the scatter plot. To draw a ROI, click on the |draw_roi_button| icon on the
right and then select an area on the scatter plot.
By clicking on a point of the scatter plot, you can display an intensity profile at that point in the lower part of the window.
When this is done, click on the |to_qspace_button| to open the :ref:`conversion window <conversion_window>`.
Shift scan sample positions
---------------------------
Conversion parameters
.....................
It is possible to set some translation of sample position between different scans with the `Shift Editor`_.
To open the `Shift Editor`_, click on the **Edit** button next to the **Apply Shift** checkbox.
Please see the :ref:`conversion window <conversion_window>` documentation.
The **Apply Shift** checkbox toggles the usage of the current scan shifts.
When the conversion is done, a new :ref:`qspace item <qspace_item>` is
added to the :ref:`project tree <project_tree>`. You can view the result in
the :ref:`qspace view <qspace_view>`.
Shift Editor
............
.. figure:: img/shift_editor.png
:width: 100%
:align: center
Shift Editor window
The **Selection** frame on the left enables the selection of the **Displayed data** and to select a region of interest to be
displayed in the central frame with the **ROI** selection tools and the the **View selection** button.
The central frame allows to edit the shift by selecting reference position on 2 different scans.
The right frame provides a **Preview** of the positions that are matched in the **Entries** tab and a preview of the integrated intensity in the **Intensity** tab.
Do not forget to save the shift before closing the `Shift Editor`_ window with the **Save** button at the bottom.
Prepare Q space conversion
--------------------------
Click on the |to_qspace_button| button in the **ROI** frame on the right side of the window to start a Q space conversion of all sample points.
This opens the :ref:`QSpace conversion dialog <conversion_window>`.
Select scans
............
To run a Q space conversion for a subset of the scans, select only the one that you want to use in the list on the left side of the window.
Sample ROI selection
....................
You can select a rectangular ROI to select the sample positions for which you want to run a QSpace conversion.
To draw a ROI, click on the |draw_roi_button| button in the right frame and then draw the selected area on the scatter plot.
Once this is done, click on the |to_qspace_button| button to open the :ref:`QSpace conversion dialog <conversion_window>`.
.. _merge_kmap:
Merging KMAP data
=================
.. |merge_window| image:: img/merge_window.png
When using X-SOCS to merge data the following window is displayed:
+---------------+
| Merge window |
+===============+
| |merge_window||
+---------------+
Fill the :ref:`input parameters <input_parse>` then start the parsing by clicking on ``parse file``.
The time it takes depends on the size of the spec file. It can take more than
a minute if the file is very large.
.. warning::
At the moment X-SOCS will merge all scans found in the SPEC file that
have a matching EDF file. So you will have to make sure that ONLY the
images corresponding to the scans you want to merge are in that folder.
When the parsing is done the other parameter fields become enabled
(:ref:`parse results <spec_and_edf>`,
:ref:`acquisition parameters <merge_acq_params>`
and :ref:`output parameters <merge_output_params>`).
After filling in those fields, click on ``Merge`` at the bottom of the window.
You will be presented with q summary of the files that will created. Presse
``Merge`` again to start the merge.
.. note::
When the merge starts you should see the progress of some files change from
0 to 2%, and then not change for a while. This is because the merge process
is opening the EDF files, this can take a while, and there is at the
moment no way to get the progress until its done.
When all is done, click on ``Close`` to close the window.
....
Fields description
------------------
.. _input_parse:
+-------------------+--------------------------------------------------------+
| Input |
+===================+========================================================+
| ``Spec file`` | path to the log (usualy a \*.spec file) |
| | generated by the |
| | KMAP command. It contains all the scans defails. |
+-------------------+--------------------------------------------------------+
| ``Img dir`` | path to the folder containing the EDF files that |
| | must be merged into the HDF5 file. |
+-------------------+--------------------------------------------------------+
| ``Version`` | allows you to select predefined values for the nextNr |
| | padding and offset. For data acquired before March 2016|
| | you will probably want to set the version to 0, |
| | otherwise version 1 should be OK. |
+-------------------+--------------------------------------------------------+
| ``nextNr padding``| controls the padding applied to the nextNr value found |
| | in the spec file which is used to generated the EDF |
| | file name. This setting is only used for data acquired |
| | before mid 2016. |
+-------------------+--------------------------------------------------------+
| ``nextNr offset`` | controls the offset applied to the nextNr value. |
+-------------------+--------------------------------------------------------+
....
.. _spec_and_edf:
+-------------------+--------------------------------------------------------+
| Spec + EDF |
+===================+========================================================+
| ``Matched scans`` | scans found in the spec file which have |
| | a matching EDF file. |
+-------------------+--------------------------------------------------------+
| ``Other scans`` | scans found in the spec file which do NOT have |
| | a matching EDF file. |
+-------------------+--------------------------------------------------------+
....
.. _merge_acq_params:
+-------------------+--------------------------------------------------------+
| Acq. Parameters |
+===================+========================================================+
| ``Beam energy`` | energy of the beam, in eV |
+-------------------+--------------------------------------------------------+
| ``Direct beam`` | position of the direct beam on the detector (in pixels)|
+-------------------+--------------------------------------------------------+
| ``Ch. per deg.`` | channels per degree |
+-------------------+--------------------------------------------------------+
....
.. _merge_output_params:
+---------------------+--------------------------------------------------------+
| Output |
+=====================+========================================================+
| ``Prefix`` | a string that will be prepended to all the hdf5 file |
| | names. Default is the "prefix" value found in the |
| | spec file |
+---------------------+--------------------------------------------------------+
| ``Ouput directory`` | folder into which the merged files will be written |
+---------------------+--------------------------------------------------------+
......@@ -2,66 +2,188 @@
Loading a project
=================
The first thing to do once you have started X-SOCS is to create a project. It will contain
all the information (input and processed data) about a single sample.
Once X-SOCS is started, you can either create a `New project`_ or open an `Existing project`_.
From there you can either :ref:`load an existing project <load_project>`
or :ref:`create a new project <create_project>`.
A project contains the information about a single K-map acquisition (scans and calibration) and stores data reduction results (conversion to QSpace and fit).
.. _load_project:
.. _create_project:
Existing project
----------------
New project
+++++++++++
.. |open_icon| image:: img/open_icon.png
.. |create_icon| image:: ../../../xsocs/resources/gui/icons/create_project.png
.. |open_dialog| image:: img/open_dialog.png
To start the project creation wizard, click on the |create_icon| button in the toolbar or select the **Create project** item in the **File** menu.
To load an existing project simply click on the `open project`
button |open_icon| in the toolbar or select the "`File/Open project`" menu
item. A dialog will open, which allows you to browse to the xsocs.proj file
you want to open and review its content.
Create project
--------------
+------------------------------+
| Open project dialog |
+==============================+
| |open_dialog| |
+------------------------------+
The first page will ask you for the folder path into which the new project will be created.
From there you can either choose a different file, or confirm your
choice and load the selected project. The :ref:`project tree <create_project>`
will then be populated with the project information.
.. warning::
There can be only one project per folder!
A confirmation dialog will pop up if a project file is already present in the chosen directory.
.. _create_project:
The next page gives you the choice between:
New project
-----------
#. **Load X-Socs data (HDF5)** to import K-map data already in HDF5 format.
.. |create_icon| image:: ../../../xsocs/resources/gui/icons/create_project.png
To load data that has already been merged, select the `master` file of the merged data (the file that contains links to all entries).
Once the project is created, the `Project summary`_ page is displayed.
#. **Import SPEC data** to import K-map data from SPEC and EDF files.
X-SOCS will merge the SPEC file and the EDF images into HDF5 files.
To start the project creation wizard simmply click on the `create project` button
|create_icon| in the toolbar or select the "`File/Create project`" menu item.
The first page will ask for the folder path into which the new project will be
created.
See `Merge SPEC and EDF to HDF5`_ below.
Merge SPEC and EDF to HDF5
--------------------------
When using X-SOCS to merge data the following window is displayed:
.. _figure_merge_window:
.. figure:: img/merge_window.png
:align: center
Merge window
Fill the `Input Parameters`_ before starting the parsing by clicking on the **Parse file** button.
The time it takes depends on the size of the spec file.
It can take more than a minute if the file is very large.
.. warning::
Please keep in mind that there can be only one project per folder! A
confirmation dialog will pop up if a project file is already present in the
chosen directory.
The next page gives you a choice between importing `SPEC` and `EDF` data
(and merge them into HDF5 files) or importing already merged HDF5 files.
At the moment X-SOCS merges all scans found in the SPEC file that have a matching EDF file.
So you will have to make sure that ONLY the images corresponding to the scans you want to merge are in the **Img. dir** folder.
Once SPEC parsing is done, the other parameter fields are enabled:
- `SPEC + EDF Parameters`_ for parsing results and scan/EDF file matching.
- `Image ROI Parameters`_ for only saving a region of interest from the input images.
- `Acq. Parameters`_ for calibration information.
In case this information is available in the SPEC file, those parameters are already set.
- `Output Parameters`_.
After filling those fields, click on the **Merge** button at the bottom of the window to start the file conversion.
.. note::
When the merge starts you should see the progress of some files changing from 0 to 2%, and then not change for a while.
This is because the merge process is opening the EDF files to get the number of frames and this can take a while.