Skip to content
Snippets Groups Projects
Commit 03c34cd1 authored by payno's avatar payno
Browse files

Merge branch 'move_export_param' into 'main'

Move export param

See merge request !9
parents 4c36de30 3fa46ec6
No related branches found
Tags v0.3.1
1 merge request!9Move export param
Pipeline #144231 passed
%% Cell type:markdown id: tags:
# How to launch a shell script to slurm using sluurp
The class to launch a shell script to slurm is `SBatchScriptJob` and with it you must provide:
* `slurm_config`: configuration to slurl
* `script`: script to launch
* `script_path`: where to save the script (if clean_script is set to True) then this script will be removed once processing is done. It can be useful to keep it for debugging
* `working_directory`: working directory to provide to `SBATCH` when executing the script
Once the job is created you can submit it to slurm using the `submit` function from `sluurp.executor`
%% Cell type:markdown id: tags:
## slurm_config
`slurm_config` must be provided as a dictionary and can contain the following keys:
* `cpu-per-task` (int): define number of cpu to require for each task
* `n_tasks` (int): define th enumber of tasks
* `memory` (int): memory in GB ( <=> `--mem` option in slurm)
* `partition` (str): partition to be used
* `n_gpus` (int): number of gpu to use
* `job_name` (str): name of the job ( <=> `-J` option in slurm)
* `walltime` (str): walltime ( <=> `-t` option in slurm)
* `python_venv` (str): path to the python environment to source before executing script lines
* `modules` (tuple of str): module names to load before executing script lines
* `sbatch_extra_params`: dict: possible extra parameter to be used when cal the 'sbatch' command. For now handles 'export' only
%% Cell type:markdown id: tags:
## example
%% Cell type:code id: tags:
``` python
import os
import tempfile
from sluurp.executor import submit
from sluurp.job import SBatchScriptJob
sleep_time = 10
with tempfile.TemporaryDirectory() as folder:
job = SBatchScriptJob(
script=(f"sleep {sleep_time}s", "echo 'succeed'"),
slurm_config={
"partition": "nice",
"memory": 32,
"job_name": "hello_world",
},
script_path=os.path.join(folder, "script.sh"),
working_directory=folder,
)
submit(job)
```
%% Cell type:code id: tags:
``` python
```
......
......@@ -15,6 +15,7 @@ slurm_config = {
# "job_name": "test sluurp",
# "python_venv": "/scisoft/tomotools/activate dev", # optional python environement to source before script command
# "modules": ("tomotools", ), # optional module to source before script command
# "sbatch_extra_params": {"export": "NONE"}, # extra parameters to provide to sbatch like 'export'
}
# step 1: launch the jobs
......
from .executor import submit # noqa F401
__version__ = "0.3.0"
__version__ = "0.3.1"
......@@ -156,9 +156,11 @@ class SBatchScriptJob(ScriptJob):
def __init__(self, slurm_config: Optional[dict] = None, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
assert isinstance(slurm_config, (dict, type(None)))
self._job_id = None
self._status = "Under submission"
self._slurm_config = slurm_config if slurm_config is not None else {}
self._sbatch_extra_params = self._slurm_config.pop("sbatch_extra_params", {})
self._pycuda_cache_dir = os.path.join(
os.path.dirname(self.script_path), f".pycuda_cache_dir_{uuid4()}"
......@@ -188,22 +190,13 @@ class SBatchScriptJob(ScriptJob):
def get_status(self):
return self._status
def run(self, export="NONE"):
def run(self):
"""
run the sbatch command with the script created.
:param str export: value to provide to the 'export' function of sbatch.
valid values are:
* 'NIL':
* 'ALL':
* 'NONE':
* <environment variables>
see details there: https://slurm.schedmd.com/sbatch.html#OPT_export
Default value is set to NONE as we can get some code run from conda. Else it might mess up
when executing (like different architectures ...)
"""
self._status = "Running"
script_path = self._save_script()
export = self._sbatch_extra_params.get("export", "NONE")
if not os.path.exists(script_path):
raise OSError(
f"{script_path} doesn't exists. Unable to write script file. Can be write rights error."
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment