Skip to content

Resolve "Check empty datasets in raw data synchronization"

Marjolaine Bodin requested to merge 27-check_empty_datasets_sync_raw into main

Closes #27 (closed)

Example:

$ icat-sync-raw --beamline id10 --proposal blc15164 --session 20240210 --save-dir sync-raw-results

--------------------------------
Directory: /data/visitor/blc15164/id10/20240210/RAW_DATA
 Start time: 2024-02-10

Investigation ID: 1511427772
 URL: https://data.esrf.fr/investigation/1511427772/datasets
 Search URL: https://data.esrf.fr/beamline/ID10?search=BLC-15164
 Start time: 2024-02-10 08:00:00+01:00
 End time: 2024-02-12 08:00:00+01:00
 Duration: 2 days, 0:00:00

Registered datasets :
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_345C/Pd_omegaMG_345C_0002
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_345C/Pd_omegaMG_345C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_glass/Pd_glass_0002
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_320C/Au_omegaMG_320C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_260C/Au_omegaMG_260C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/J2/J2_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_220C/Pd_omegaMG_220C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_290C/Pd_omegaMG_290C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_335C/Au_omegaMG_335C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_330C/Pd_omegaMG_330C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_330C/Pd_omegaMG_330C_0002
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_330C/Au_omegaMG_330C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_290C/Au_omegaMG_290C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_345C/Au_omegaMG_345C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_310C/Au_omegaMG_310C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_240C/Au_omegaMG_240C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omega/Au_omega_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_325C/Pd_omegaMG_325C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_325C/Pd_omegaMG_325C_0002
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_220C/Au_omegaMG_220C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/J6_cand2_rot90/J6_cand2_rot90_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_325C/Au_omegaMG_325C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_320C/Pd_omegaMG_320C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/eh2_exp/eh2_exp_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_RT/Pd_RT_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/J6_cnad2/J6_cnad2_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_240C/Pd_omegaMG_240C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_310C/Pd_omegaMG_310C_0002
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_310C/Pd_omegaMG_310C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_340C/Au_omegaMG_340C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/LaB6_21p67keV/LaB6_21p67keV_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/LaB6_21p67keV/LaB6_21p67keV_0002
  /data/visitor/blc15164/id10/20240210/RAW_DATA/eh2_cdi/eh2_cdi_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/J6/J6_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/J5_cand1_rot90/J5_cand1_rot90_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_260C/Pd_omegaMG_260C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_340C/Pd_omegaMG_340C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_340C/Pd_omegaMG_340C_0002
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_350C/Au_omegaMG_350C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Au_omegaMG_200C/Au_omegaMG_200C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/J5_cand1/J5_cand1_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_335C/Pd_omegaMG_335C_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_335C/Pd_omegaMG_335C_0002

Unregistered datasets :
  /data/visitor/blc15164/id10/20240210/RAW_DATA/LaB6_10022024/LaB6_10022024_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/eh2_optics/eh2_optics_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_350C/Pd_omegaMG_350C_0002

Incomplete Datasets (missing files) :
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_glass/Pd_glass_0001
  /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_350C/Pd_omegaMG_350C_0001

Datasets (TODO):
 43 registered
 0 invalid
 0 not uploaded
 3 unregistered
 2 missing files

--------------------------------
->  Register dataset : /data/visitor/blc15164/id10/20240210/RAW_DATA/LaB6_10022024/LaB6_10022024_0001
    Name: 0001
    Sample: LaB6_10022024
    Start time: 2024-02-10 17:31:15.063571+01:00
    End time: 2024-02-10 17:36:13.229806+01:00
    Duration: 0:04:58.166235
->  Register dataset : /data/visitor/blc15164/id10/20240210/RAW_DATA/eh2_optics/eh2_optics_0001
    Name: 0001
    Sample: eh2_optics
    Start time: 2024-02-10 09:34:42.119897+01:00
    End time: 2024-02-11 19:44:36.785518+01:00
    Duration: 1 day, 10:09:54.665621
->  Register dataset : /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_350C/Pd_omegaMG_350C_0002
    Name: 0002
    Sample: Pd_omegaMG_350C
    Start time: 2024-02-12 05:46:43.174435+01:00
    End time: 2024-02-12 07:00:36.371682+01:00
    Duration: 1:13:53.197247
->  Add missing files to dataset : /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_glass/Pd_glass_0001
    Name: 0001
->  Add missing files to dataset : /data/visitor/blc15164/id10/20240210/RAW_DATA/Pd_omegaMG_350C/Pd_omegaMG_350C_0001
    Name: 0001
Edited by Marjolaine Bodin

Merge request reports