Resolve "Bug linking processed datasets"
Closes #201 (closed)
This MR fixes the issue with linking processed datasets to existing processed datasets. I think it requires further consideration to develop a solution that works optimally in all scenarios, but for now, it resolves the problem for BM29.
For the records, once deployed I did few tests but sending unordered processed datasets with this script:
#!/bin/bash
# Define parameters
#BEAMLINE="bm29"
#PROPOSAL="BLC16410"
BEAMLINE="ID00"
PROPOSAL="ID002509"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
SAMPLE="sample_friday_${TIMESTAMP}"
BASE_PATH="/tmp/TEST1/$SAMPLE"
#QUEUE="dau-dm-04:61613"
QUEUE="bcu-mq-01:61613"
echo "Running for sample: $SAMPLE"
echo "Base path: $BASE_PATH"
# process_1_1
icat-store-processed --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
--dataset process_1_1 --sample "$SAMPLE" \
--path "$BASE_PATH/process_1_1" \
--raw "$BASE_PATH/process_1" \
--queue "$QUEUE"
sleep 5
# RAW BUFFER BEFORE
icat-store-raw --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
--dataset raw1 --sample "$SAMPLE" \
--path "$BASE_PATH/raw1" \
--queue "$QUEUE"
sleep 5
# process_3
icat-store-processed --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
--dataset process_3 --sample "$SAMPLE" \
--path "$BASE_PATH/process_3" \
--raw "$BASE_PATH/process_2 $BASE_PATH/process_1_1" \
--queue "$QUEUE"
sleep 20
# process_2
icat-store-processed --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
--dataset process_2 --sample "$SAMPLE" \
--path "$BASE_PATH/process_2" \
--raw "$BASE_PATH/process_1" \
--queue "$QUEUE"
sleep 20
# process_1
icat-store-processed --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
--dataset process_1 --sample "$SAMPLE" \
--path "$BASE_PATH/process_1" \
--raw "$BASE_PATH/raw1" \
--queue "$QUEUE"
sleep 5
Results seem fine to me: https://data.esrf.fr/investigation/2235105143/datasets/2250244476
Edited by Alejandro De Maria Antolinos