Add validator (!141) · Merge requests · tomotools / Nabu

payno requested to merge add_validator into master Aug 26, 2021

About

Add a validator application to check input data.

Proposed API:

nabu validator [file.h5/folder] [[entries]] [--no-phase-retrieval] [[--check-datasets]] [[--all-entries]]

To do

Add the application
Add documentation

Notes

Help returns:

usage: nabu-validator [-h] [--ignore-dark] [--ignore-flat] [--no-phase-retrieval] [--check-nan-and-vds] [--all-entries ALL_ENTRIES] [--summarize SUMMARIZE] file_ [entries [entries ...]]

Check if provided scan(s) seems valid to be reconstructed.

positional arguments:
  file_                 Data file to validate (h5 file, edf files)
  entries               Entries to be validated (in the case of a h5 file)

optional arguments:
  -h, --help            show this help message and exit
  --ignore-dark         Do not check for dark
  --ignore-flat         Do not check for flat
  --no-phase-retrieval  Check scan energy, distance and pixel size
  --check-nan-and-vds   Check frame datasets for nan or broken virtual dataset
  --all-entries ALL_ENTRIES
                        Check all entries of the files (for HDF5 only for now)
  --summarize SUMMARIZE
                        Only provide a general information and details of issues

Some example of execution and results

HDF5 folder

$: nabu-validator bambou_hercules_0001_1_1.nx

No issue found from hdf5 scan(master_file: bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx, entry: entry0000).

$: nabu-validator bambou_hercules_0001_1_1.nx --extend

No issue found from hdf5 scan(master_file: bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx, entry: entry0000).
   + projection(s): valid: True. Expected location: <bound method DataUrl.path of DataUrl(valid=True, scheme=None, file_path='/mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx', data_path='instrument/detector/data', data_slice=None)>
   + dark(s): valid: True. Expected location: <bound method DataUrl.path of DataUrl(valid=True, scheme=None, file_path='/mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx', data_path='instrument/detector/data', data_slice=None)>
   + flat(s): valid: True. Expected location: <bound method DataUrl.path of DataUrl(valid=True, scheme=None, file_path='/mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx', data_path='instrument/detector/data', data_slice=None)>
   + distance: valid: True. Expected location: /mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx?instrument/beam/incident_energy
   + energy: valid: True. Expected location: /mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx?instrument/beam/incident_energy
   + pixel size: valid: True. Expected location: /mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx?instrument/detector/x_pixel_size

And then if I remove folders containing frames:

$: nabu-validator bambou_hercules_0001_1_1.nx --check-nan-and-vds

3 issues found from hdf5 scan(master_file: bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx, entry: entry0000)
   - projection(s): valid: False. Expected location: <bound method DataUrl.path of DataUrl(valid=True, scheme=None, file_path='/mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx', data_path='instrument/detector/data', data_slice=None)>
   - dark(s): valid: False. Expected location: <bound method DataUrl.path of DataUrl(valid=True, scheme=None, file_path='/mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx', data_path='instrument/detector/data', data_slice=None)>
   - flat(s): valid: False. Expected location: <bound method DataUrl.path of DataUrl(valid=True, scheme=None, file_path='/mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx', data_path='instrument/detector/data', data_slice=None)>

Maybe the --check-nan-and-vds option should be set by default ?

if ignoring dark and flat on the same dataset:

$: nabu-validator bambou_hercules_0001_1_1.nx --check-nan-and-vds --ignore-dark --ignore-flat

1 issues found from hdf5 scan(master_file: bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx, entry: entry0000)
   - projection(s): valid: False. Expected location: <bound method DataUrl.path of DataUrl(valid=True, scheme=None, file_path='/mnt/multipath-shares/tmp_14_days/payno/bamboo_hercules/bambou_hercules_0001/bambou_hercules_0001_1_1.nx', data_path='instrument/detector/data', data_slice=None)>

EDF folder

$: nabu-validator 5.06_crayon_W150_60_Al2_W0.25_xc1000_

No issue found from  edf scan(5.06_crayon_W150_60_Al2_W0.25_xc1000_).

$: nabu-validator 5.06_crayon_W150_60_Al2_W0.25_xc1000_ --extend

No issue found from  edf scan(5.06_crayon_W150_60_Al2_W0.25_xc1000_).
   + projection(s): valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_/*.edf
   + dark(s): valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/dark[*].edf
   + flat(s): valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/refHST[*].edf
   + distance: valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_.info::Distance
   + energy: valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_.info::Energy
   + pixel size: valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_.info::PixelSize

when removing Energy and Distance from .info file:

$: nabu-validator 5.06_crayon_W150_60_Al2_W0.25_xc1000_ --ignore-dark --extend

   + projection(s): valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_/*.edf
   + dark(s): valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/dark[*].edf
   + flat(s): valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/refHST[*].edf
   - distance: valid: False. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_.info::Distance
   - energy: valid: False. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_.info::Energy
   + pixel size: valid: True. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_.info::PixelSize

$: nabu-validator 5.06_crayon_W150_60_Al2_W0.25_xc1000_ --ignore-dark

2 issues found from  edf scan(5.06_crayon_W150_60_Al2_W0.25_xc1000_)
   - distance: valid: False. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_.info::Distance
   - energy: valid: False. Expected location: 5.06_crayon_W150_60_Al2_W0.25_xc1000_/5.06_crayon_W150_60_Al2_W0.25_xc1000_.info::Energy

Edited Aug 27, 2021 by payno

Add validator