Skip to content

edf2nx: Add output check and delete source files.

payno requested to merge fix_135 into master

TODO

  • add an option to remove EDF source file

  • add a method to check the output volume. for this we have several ways to handle it

    • get a boolean check_output_volume -> for me this does not bring enough option to fine tune. Indeed I think in most of the cases we want to do a quick check (checking Virtual dataset) as in the tomoscan validators but sometimes (when removing source files - ID16A) we probably want to do more advanced stuff like comparing frame by frame the final vs raw volumes.
    • get an integer to define a level (0: no check, 1: check for broken/invalid links on the final volume or frame at zero, 2: compare frame by frame)
    • get a list of possible checks like -> this is the option selected
      • empty: no check done
      • check_vds_links: check for broken virtual source
      • compare_volumes: compare input volume and output volume
      • check_vds_links, compare_volumes: do both
  • add unit test

  • test on a real dataset

  • add CLI option for checking volume and removing input because I think ID16a script are using directly the CLI

  • dedicated handling for multi edf files

limitations

  • the .info will not be removed. This can be added but not so sure about the interest (especially in the case this is not the default one ("{basename}.info") but another one. This could lead to important technical mishap from users and with no ways to go back.
  • For now the two options to access it are only available on 'advanced' edf2nx configuration file. But looks like to be the usage. If not we can still add it later to the CLI directly.
  • I am not sure this test is enough ? But the core of the processing from my point if view is more on the algorithm behind edf2nx.

usage

from the configuration file

this PR adds two parameters to the edf2nx configuration file under the general section:

  • delete_edf_source_file: as a boolean. False is the default, if True then the .edf used during the conversion will be removed.
  • output_checks: as a tuple of str. For now the only str available is compare-output-volume which test frame by frame the url used and the final HDF5 volume

image

from the command line

nxtomomill edf2nx {} --output-checks compare-output-volume --delete-edf
Edited by payno

Merge request reports