Skip to content

Single-machine, multi-process integration

Pierre Paleo requested to merge local_cluster into master

This MR adds the MultiprocIntegrator class for distributing the work over a set of processes on the local machine.

To simplify task distribution, each worker (process) handles one full dataset (finer granularity would involve IPC).

To do

  • Refactor the classes to use a base class DistributedIntegrator
  • Ensure that distributed integration (over SLURM) still works
  • Test MultiprocIntegrator
  • Automatic resources handling:
    • Determine automatically the best affinity mask
    • Pick the best opencl platform
  • Plug MultiprocIntegrator to the CLI/configuration file
  • Clean-up the test/debug code fragments (eg. os.environ, is_power9, etc)
Edited by Pierre Paleo

Merge request reports