Skip to content

GitLab

Last edited by Pierre Paleo
Page history

Nabu deployment on ESRF infrastructure

Introduction

On a recent-enough operating system, installing nabu boils down to pip install nabu (along with pip install pycuda && pip install scikit-cuda for the Cuda backend).

Unfortunately, ESRF infrastructure is far from this ideal setting, sometimes for understandable reasons (old hardware, dedicated beamlines machines where upgrading is risky, etc). In this case, conda is often needed to bring a decent version of Python.

However, conda environments sometimes cannot be used between hosts when GPU drivers differ. For example, one node might have Debian 8.11 and a Kepler GPU, while another has Debian 8.11 with a Fermi GPU. In this case, a conda environment must be created for each. See (1) in notes for more details.

Conda environments classification

The servers can be classified as follows:

  • "Ideal Case": recent operating system (Debian >= 10, Ubuntu >= 20.04). Plain venv is enough. Examples: scisoft 10,11,14,15
  • "venv + PyQt5 patch": venv is almost enough, but some tinkering with PyQt5 needs to be done to have GUI working. Examples: p9-gpu slurm partition
  • Conda
    • Fermi driver
    • Kepler driver
    • ...

ESRF machines

Last updated: 22/02/2021

Machine name Operating system GPU name/driver Notes
p9-XX (p9gpu partition) Ubuntu 20.04 Tesla V100 / 10.1.243-3 ppc64le architecture !
gpid16a-180X (id16a partition) Ubuntu 20.04 Tesla K20m / 10.1.243-3 Should work with plain venv
gpu2-1401 (deb9-gpu partition) Debian 9.11 Tesla K20m / 9.1.85-8~bpo9+1
gpu2-140XX Debian 8.11 Tesla K20m / 7.5.18-4~bpo8+1 OAR queue "gpu"
lbs191 Debian 8.11 GTX 1080 Ti / 8.0.44-4
lbs051 Debian 8.11 K20 / 6.5.19-1
gptomo-nice-0405 Debian 8.11 M2070-Q / 7.5.18-4~bpo8+1 Fermi generation !
gpid11-nice Debian 8.11 Titan X / 6.5.19-1
crunch Debian 8.11 Titan Black / 6.5.19-1 Kepler.
scisoft10,scisoft11 Ubuntu 20.04 Quadro P2000 / 10.1.243-3 Used for DAU CI
scisoft12 Debian 8.11 Tesla K20m / 6.5.19-1 Seems to be down
scisoft13 Debian 8.8 (!) GTX 1080 Ti / 9.1.85-8~bpo9+1
scisoft14 Debian 10 Titan V, Q P6000 / 9.2.148-7
scisoft15 Debian sid (!) Titan V / 11.1.1-4 ppc64le architecture !

Notes

(1)
It is sometimes possible to share the same conda environment across different Cuda driver versions. However, it seems that the cudatoolkit conda package is not the way to go. Instead, pycuda must be compiled against the system's cuda driver.

  • Works: using conda environment of gpu2-1303 (Cuda 7.5.18-4~bpo8+1) from gpid11-nice (Cuda6.5.19-1)
  • Works: using conda environment of gpu2-1303 (OAR) from gptomo-nice-0405 (both have the same cuda driver)
  • Does NOT work: gpu2-1303 and gptomo-nice-0405 using conda cudadriver (skcuda.cudart.cudaErrorInsufficientDriver error)

See also

http://www.esrf.eu/Infrastructure/Computing/NICE/Implementation