Commit fac97c78 authored by Henri Payno's avatar Henri Payno
Browse files

doc: improve documentation regarding edf 2 nx tutorial + applications

parent da968510
Pipeline #73356 passed with stages
in 3 minutes and 15 seconds
.. _dxfile2nxtutorial:
dxfile2nx tutorial
==================
the `dxfile2nx` application is used to convert an acquisition stored with the `dxfile format <https://dxfile.readthedocs.io/en/latest/>`_ to the `NXTomo <https://manual.nexusformat.org/classes/applications/NXtomo.html>`_ format.
To call this application you can call directly
......
.. _edf2nxtutorial:
edf2nx tutorial
===============
......@@ -6,11 +8,11 @@ the `edf2nx` application is used to convert acquisition from edf standard tomogr
This format will also be stored in .h5 / .hdf5 / .nx file.
For comprehension we will use the nexus format (.nx) in this tutorial.
To call this application you can call directly
To access this application you can call directly
.. code-block:: bash
nxtomomill edf2nx [options]
nxtomomill edf2nx [-h] [--dataset-basename DATASET_BASENAME] [--info-file INFO_FILE] [--config CONFIG] [scan_path] [output_file]
if nxtomomill has been installed in the global scope.
Otherwise you can call
......@@ -18,20 +20,25 @@ Otherwise you can call
.. code-block:: bash
python -m nxtomomill edf2nx [options]
nxtomomill edf2nx [options]
simple convertion (no configuration file)
-----------------------------------------
The first two parameters should be:
To execute a conversion from EDF-Spec to NXtomo using the default parameters the first two parameters should be:
* input-file-directory: the root directory containing all the .edf frames and the .info file of the acquisition
* output-file: filename where the tomography scan will be stored
* input folder directory aka scan_path: root directory containing all the .edf frames and the .info file of the acquisition.
By default it expects this folder name to be the .edf file prefix (as `folder_name_0000.edf...`) and .info file to be named `folder_name.info`.
To have advanced option on this please have a look at :ref:`edf2nxtutorialConfigFile` and on `dataset_basename` and `dataset_info_file` fields.
* output_file: output filename which will contain the NXtomo created
Sor for example to convert an edf-like tomography dataset '/data/idxx/inhouse/myname/sample1_' to 'sample1_.nx' you should call:
.. code-block:: bash
python -m nxtomomill edf2nx /data/idxx/inhouse/myname/sample1_ /data/idxx/inhouse/myname/sample1_.nx
nxtomomill edf2nx /data/idxx/inhouse/myname/sample1_ /data/idxx/inhouse/myname/sample1_.nx
Normally the resulting file should have more or less the same size than the initial directory.
......@@ -42,7 +49,7 @@ You can also access the help of edf2nx by calling:
.. code-block:: bash
python -m nxtomomill edf2nx --help
nxtomomill edf2nx --help
The result can be displayed using any hdf5 display or using silx:
......@@ -55,14 +62,53 @@ All the .edf files in the origin directory are considered except those having '_
The algorithm selects raw dark fields - darkendxxxx.edf - and raw flat fields refxxxx.edf. However, if processed darks (dark.edf) and refs (refHST) exist,
they are stored in the destination file instead of the raw files.
The names of the motors are hard-coded: 'srot' for the rotation, 'sx', 'sy' and 'sz' for the positioning motors.
The names of the motors are defined to some defauls ('srot', 'somega') for the rotation, 'sx', 'sy' and 'sz' for the positioning motors.
You can defined different keys from the configuration file. If you have 'redundant' keys to be used you can also let us know so we can add those as default keys.
.. edf2nxtutorialConfigFile:
.. note:: The conversion is based on a set of key value that are contained in the EDF file headers.
Those values are set to a default value. Those values can be tune either by updating the nxtomomill.settings file
or defined on the fly when calling edf2nx. See edf2nx --help to access the different keys.
Example:
advanced convertion (using configuration file)
----------------------------------------------
The conversion is based on settings (defined in settings.py module and EDFTomoConfig class). In order to define:
1. which key of the EDF headers should be used to get rotation angle, translations
2. prefix to be used to deduce dark and flat files
3. rules to compute rotation angle if we cannot deduce them from edf headers
4. pattern to recognize some file to be ignored (like pyhst reconstruction files)
All of the settings used can be defined on a configuration file. A configuration file can be created from:
.. code-block:: bash
nxtomomill edf-quick-start [edf_2nx_config.cfg] [--level]
For now user can get configuration file with two level of details: `required` and `advanced`.
Sections and fields comments should be clear enought for you to understand the meaning of each elements (otherwise let us know).
We can notice that for the previous explained case:
1. define EDF header keys to be used: this will be defined in the *EDF_KEYS_SECTION* section
2. define prefix to be used for dark and flat: this will be defined in the *DARK_AND_FLAT_SECTION* section
3. rules to compute rotation angle if we cannot deduce them from edf headers: this will be defined in the *SAMPLE_SECTION* section
4. pattern to recognize some file to be ignored (like pyhst reconstruction files): this will be defined in the *GENERAL_SECTION* section
Regarding the *GENERAL_SECTION* we can also provide more hint on:
* `dataset_basename`: the `dataset_basename` will be used to deduce all the information required to build a NXtomo: get projections files (like dataset_basename_XXXX.edf) and retrieve infomration like energy, sample / detector distance from the >info file.
If not provided then the `dataset_basename` will be the name of the provided folder.
* `dataset_info_file`: In order to retrieve scan range, energy, distance, pixel size... we use a .info file with predefined keys.
If not provided the converted will look for a `dataset_basename.info` file. But you can provide provide path to another .info file to be used.
Once your configuration is edited you can use it from the `nxtomomill edf2nx` using the `--config` option like:
.. code-block:: bash
.. code-block:: bash
nxtomomill edf2nx --config edf_2nx_config.cfg
python -m nxtomomill tomoedf2nx /data/idxx/inhouse/myname/sample1_ /data/idxx/inhouse/myname/sample1_.nx --rot_angle_key=srot --ignore_file_containing=_slice_,test
.. note:: to ease usage the "dataset basename" and the "info file" can also be provided from command line (`--dataset-basename` and `--info-file` options).
If you provide one of those parameters from the command line option then it should not be provided from the configuration.
This is the same for `scan_path` aka folder path (containing raw data / .edf) and `output_file`
......@@ -58,6 +58,9 @@ Application to convert from dx file (HDF5) to NXTomo (HDF5) file
incident beam energy in keV
--overwrite Do not ask for user permission to overwrite output files
--data-copy Force data duplication of frames. This will permit to have an 'all-embed' file. Otherwise we will have link between the dxfile and the NXTomo.
For a complete tutorial you can have a look at :ref:`dxfile2nxtutorial`
"""
__authors__ = [
......
......@@ -28,37 +28,24 @@ Application to convert a tomo dataset written in edf into and hdf5/nexus file.
.. code-block:: bash
usage: nxtomomill edf2nx [-h] [--file_extension] [--motor_pos_key MOTOR_POS_KEY] [--motor_mne_key MOTOR_MNE_KEY] [--refs_name_keys REFS_NAME_KEYS] [--ignore_file_containing IGNORE_FILE_CONTAINING]
[--rot_angle_key ROT_ANGLE_KEY] [--dark_names DARK_NAMES] [--x_trans_key X_TRANS_KEY] [--y_trans_key Y_TRANS_KEY] [--z_trans_key Z_TRANS_KEY]
scan_path output_file
usage: nxtomomill edf2nx [-h] [--dataset-basename DATASET_BASENAME] [--info-file INFO_FILE] [--config CONFIG] [scan_path] [output_file]
convert data acquired as edf to hdf5 - nexus compliant file format.
positional arguments:
scan_path folder containing the edf files
output_file foutput .h5 file
scan_path folder containing the edf files
output_file foutput .h5 file
optional arguments:
-h, --help show this help message and exit
--file_extension extension of the output file. Valid values are .h5/.hdf5/.nx
--motor_pos_key MOTOR_POS_KEY
motor position key in EDF HEADER
--motor_mne_key MOTOR_MNE_KEY
motor mne key in EDF HEADER
--refs_name_keys REFS_NAME_KEYS
prefix of flat field file
--ignore_file_containing IGNORE_FILE_CONTAINING
substring that lead to ignoring the file if contained in the name
--rot_angle_key ROT_ANGLE_KEY
rotation angle key in EDF HEADER
--dark_names DARK_NAMES
prefix of the dark field file
--x_trans_key X_TRANS_KEY
x translation key in EDF HEADER
--y_trans_key Y_TRANS_KEY
y translation key in EDF HEADER
--z_trans_key Z_TRANS_KEY
z translation key in EDF HEADER
-h, --help show this help message and exit
--dataset-basename DATASET_BASENAME, --file-prefix DATASET_BASENAME
file prefix to be used to deduce projections
--info-file INFO_FILE
.info file containing acquisition information (ScanRange, Energy, TOMO_N...)
--config CONFIG, --configuration-file CONFIG, --configuration CONFIG
file containing the full configuration to convert from SPEC-EDF to bliss to nexus. Default configuration file can be created from `nxtomomill edf-quick-start` command
For a complete tutorial you can have a look at :ref:`edf2nxtutorial`
"""
__authors__ = ["C. Nemoz", "H. Payno", "A.Sole"]
......@@ -101,7 +88,8 @@ def main(argv):
"--configuration-file",
"--configuration",
default=None,
help="file containing the full configuration to convert from SPEC-EDF to bliss to nexus",
help="file containing the full configuration to convert from SPEC-EDF to bliss to nexus. "
"Default configuration file can be created from `nxtomomill edf-quick-start` command",
)
options = parser.parse_args(argv[1:])
......
......@@ -35,6 +35,7 @@ Application to create a default configuration file to be used by edf2nx applicat
positional arguments:
output_file output .cfg file
For a complete tutorial you can have a look at :ref:`edf2nxtutorial`
"""
__authors__ = [
......
......@@ -95,6 +95,8 @@ Application to convert a bliss-hdf5 tomography dataset to Nexus - NXtomo (hdf5)
--config CONFIG, --config-file CONFIG, --configuration CONFIG, --configuration-file CONFIG
file containing the full configuration to convert from h5 bliss to nexus
For a complete tutorial you can have a look at: :ref:`Tomoh52nx`
"""
__authors__ = ["C. Nemoz", "H. Payno", "A.Sole"]
......
......@@ -40,6 +40,7 @@ Application to create a default configuration file to be used by h52nx applicati
--from-title-names Provide minimalistic configuration to make a conversion from titles names. (FRAME TYPE section is ignored). Exclusive with `from-scan-urls` option
--from-scan-urls Provide minimalistic configuration to make a conversion from scan urls. (ENTRIES and TITLES section is ignored). Exclusive with `from-title-names` option
For a complete tutorial you can have a look at: :ref:`Tomoh52nx`
"""
__authors__ = [
......
......@@ -88,15 +88,15 @@ class TomoEDFConfig(ConfigBase):
COMMENTS_GENERAL_SECTION = {
GENERAL_SECTION_DK: "general information. \n",
INPUT_FOLDER_DK: "input file if not provided must be provided from the command line",
OUTPUT_FILE_DK: "output file name. If not provided will use the input file basename and the file extension",
INPUT_FOLDER_DK: "Folder containing .edf files. if not provided from the configuration file must be provided from the command line",
OUTPUT_FILE_DK: "output file name. If not provided from the configuration file must be provided from the command line",
OVERWRITE_DK: "overwrite output files if exists without asking",
FILE_EXTENSION_DK: "file extension. Ignored if the output file is provided and contains an extension",
DATASET_BASENAME_DK: "dataset file prefix. If not provided will take the folder basename",
DATASET_FILE_INFO_DK: f"path to .info file containing dataset information (Energy, ScanRange, TOMO_N...). If not will deduce it from {INPUT_FOLDER_DK} and {DATASET_BASENAME_DK}",
DATASET_BASENAME_DK: f"dataset file prefix. Usde to determine projections file and info file. If not provided will take the name of {INPUT_FOLDER_DK}",
DATASET_FILE_INFO_DK: f"path to .info file containing dataset information (Energy, ScanRange, TOMO_N...). If not will deduce it from {DATASET_BASENAME_DK}",
LOG_LEVEL_DK: 'Log level. Valid levels are "debug", "info", "warning" and "error"',
TITLE_DK: "NXtomo title",
IGNORE_FILE_PATTERN_DK: "some file pattern leading to ignoring the file",
IGNORE_FILE_PATTERN_DK: "some file pattern leading to ignoring the file. Like reconstructed slice files.",
}
LEVEL_GENERAL_SECTION = {
......@@ -128,9 +128,9 @@ class TomoEDFConfig(ConfigBase):
ROT_ANGLE_KEY_DK = "rot_angle_key"
COMMENTS_KEYS_SECTION = {
EDF_KEYS_SECTION_DK: "section to define EDF keys to pick in headers to deduce several information.\n",
EDF_KEYS_SECTION_DK: "section to define EDF keys to pick from headers to deduce information like rotation angle.\n",
MOTOR_POSITION_KEY_DK: "motor position key",
MOTOR_MNE_KEY_DK: "motor mne key",
MOTOR_MNE_KEY_DK: "key to retrieve indices of each motor in metadata",
X_TRANS_KEY_DK: "key to be used for x translation",
Y_TRANS_KEY_DK: "key to be used for y translation",
Z_TRANS_KEY_DK: "key to be used for z translation",
......@@ -156,8 +156,8 @@ class TomoEDFConfig(ConfigBase):
COMMENTS_DARK_FLAT_SECTION = {
FLAT_DARK_SECTION_DK: "section to define dark and flat detection. \n",
DARK_NAMES_DK: "prefix of the dark field file(s)",
FLAT_NAMES_DK: "prefix of the flat field file(s)",
DARK_NAMES_DK: "prefix of dark field file(s)",
FLAT_NAMES_DK: "prefix of flat field file(s)",
}
LEVEL_DARK_FLAT_SECTION = {
......@@ -217,8 +217,8 @@ class TomoEDFConfig(ConfigBase):
COMMENTS_SAMPLE_SECTION_DK = {
SAMPLE_SECTION_DK: "section dedicated to sample definition.\n",
SAMPLE_NAME_DK: "name of the sample",
FORCE_ANGLE_CALCULATION: "if set to False try first to deduce rotation angle from edf metadata. Else compute them from numpy.linspace",
FORCE_ANGLE_CALCULATION_ENDPOINT: "if rotation angles have to be calculated set numpy.linspace endpoint parameter to this value. If True then the rotation angle value of the last projection will be equal to the `ScanRange` value",
FORCE_ANGLE_CALCULATION: "Should the rotation angle be computed from scan range and numpy.linspace or should we try to load it from .edf header.",
FORCE_ANGLE_CALCULATION_ENDPOINT: "If rotation angles have to be calculated set numpy.linspace endpoint parameter to this value. If True then the rotation angle value of the last projection will be equal to the `ScanRange` value",
FORCE_ANGLE_CALCULATION_REVERT_NEG_SCAN_RANGE: "Invert rotation angle values in the case of negative `ScanRange` value",
}
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment