Skip to content

"Offline tool for converting HDF5 to .dat"

Closes #1

This MR addresses 3 main points:

  • A generic way for launching a processing job. It is focused for the dataportal but not only
  • A demo task that allows to test the environment for administrator.
  • HDF5 to ASCII tool
  • Upload dataset to DRAC

A generic way for launching a processing job

This MR proposes a mechanism to abstract all common functionality and ensure it is implemented in a coherent and safe manner.

Thus, the tasks inherit from RawDatasetTask, an abstract class designed to encapsulate the logic related to status handling and determining the appropriate output location.

The main goal is not to reinvent the wheel for each new task as well as make things as safe as possible

A demo task that allows to test the environment for administrator

Thanks to the above, the tasks only need to define what they must do for processing

Example:

class SumTask(
    RawDatasetTask,
    input_names=["a", "callback", "raw_data_path"],
    optional_input_names=["b", "delay"],
    output_names=["result"],
):
    """
    Adds two numbers.
    This task is used to test that the environment is up and running.
    """
    def process(self, input_folder, output_folder):
        result = self.inputs.a + self.get_input_value("b", 0)
        time.sleep(self.get_input_value("delay", 0))
        self.outputs.result = result
        # Writing the string to a file called 'result'
        result_file_path = os.path.join(output_folder, "result.txt")
        with open(result_file_path, "w") as file:
            file.write(f"{self.inputs.a} + {self.inputs.b} = {result}")

HDF5 to ASCII tool

Given a dataset that contains integration done in HDF5 the task will look for such files and will extract them into individual .dat files


    def process(self, input_folder, output_folder):
        # Get the files within the raw dataset folder which file name has integrate
        files = self.list_files_with_pattern(input_folder)
        self._log(
            {
                "logs": {
                    "integrate files found": f"{str(files)}",
                    "input_folder": f"{str(input_folder)}",
                    "output_folder": f"{str(output_folder)}",
                }
            }
        )
        for src in files:
            dest = output_folder + "/%04i.dat"
            for idx, frame in enumerate(extract_all(src)):
                write_ascii(frame, dest % idx)
Edited by Alejandro De Maria Antolinos

Merge request reports

Loading