Skip to content

Resolve "Bug linking processed datasets"

Closes #201 (closed)

This MR fixes the issue with linking processed datasets to existing processed datasets. I think it requires further consideration to develop a solution that works optimally in all scenarios, but for now, it resolves the problem for BM29.

For the records, once deployed I did few tests but sending unordered processed datasets with this script:

#!/bin/bash

# Define parameters
#BEAMLINE="bm29"
#PROPOSAL="BLC16410"
BEAMLINE="ID00"
PROPOSAL="ID002509"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
SAMPLE="sample_friday_${TIMESTAMP}"
BASE_PATH="/tmp/TEST1/$SAMPLE"
#QUEUE="dau-dm-04:61613"
QUEUE="bcu-mq-01:61613"

echo "Running for sample: $SAMPLE"
echo "Base path: $BASE_PATH"

# process_1_1
icat-store-processed --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
  --dataset process_1_1 --sample "$SAMPLE" \
  --path "$BASE_PATH/process_1_1" \
  --raw "$BASE_PATH/process_1" \
  --queue "$QUEUE"
sleep 5

# RAW BUFFER BEFORE
icat-store-raw --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
  --dataset raw1 --sample "$SAMPLE" \
  --path "$BASE_PATH/raw1" \
  --queue "$QUEUE"

  
sleep 5


# process_3
icat-store-processed --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
  --dataset process_3 --sample "$SAMPLE" \
  --path "$BASE_PATH/process_3" \
  --raw "$BASE_PATH/process_2 $BASE_PATH/process_1_1"  \
  --queue "$QUEUE"
sleep 20

# process_2
icat-store-processed --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
  --dataset process_2 --sample "$SAMPLE" \
  --path "$BASE_PATH/process_2" \
  --raw "$BASE_PATH/process_1" \
  --queue "$QUEUE"
sleep 20

# process_1
icat-store-processed --beamline "$BEAMLINE" --proposal "$PROPOSAL" \
  --dataset process_1 --sample "$SAMPLE" \
  --path "$BASE_PATH/process_1" \
  --raw "$BASE_PATH/raw1" \
  --queue "$QUEUE"
sleep 5

Results seem fine to me: https://data.esrf.fr/investigation/2235105143/datasets/2250244476

image

Edited by Alejandro De Maria Antolinos

Merge request reports

Loading