NOTES B – GPUs, tasks, and what the scheduler is telling us

Making GPUs requestable per job turned out to expose three assumptions in this plug-in that had held only because nothing had ever asked for a GPU explicitly. They are recorded together because they share a root: an allocation describes what a job was given, not what the calculation needs.

A GPU in the allocation is not a reason to use gpu-code

NGPUS comes from the scheduler. seamm_exec reads SLURM_JOB_GPUS:

if "JOB_GPUS" in ce:
    ce["NGPUS"] = len(ce["JOB_GPUS"].split(","))

and this plug-in used its presence alone to select gpu-code:

if "NGPUS" in ce:
    cmd = ["{gpu-code}"]

So the first classical job submitted with --gres=gpu:1 – an OPLS-AA water equilibration – was handed to the MACE engine, which has no model to serve. SEAMM_FF is "Unknown" for a non-PyTorch forcefield, and the engine died with FileNotFoundError: 'Unknown' before LAMMPS ran at all.

gpu-code serves two unrelated purposes: a Kokkos command is how a classical forcefield uses a GPU, and an MDI command serves a machine-learned one. The test is therefore not “are there GPUs” but “does this command suit this forcefield”, which is decided by whether it drives MDI. A classical forcefield with an MDI gpu-code now runs on the CPU and says why.

GPU indices are not the scheduler’s indices

GPU selection went through GPUtil, which reads nvidia-smi and therefore reports physical indices, knowing nothing about CUDA_VISIBLE_DEVICES. A scheduler that allocates a subset of a node’s GPUs sets that variable, and doing so renumbers the devices: given physical GPU 1, the job’s own device 0 is that GPU.

The step reported 1. The calculation would have run on a card it had not been allocated, beside whichever job actually held it, while the reserved one sat idle. Selection now happens in the numbering CUDA itself uses.

mdi_bind.sh had the same bug from the other end: it exported CUDA_VISIBLE_DEVICES unconditionally. That variable is not composable – setting it again is interpreted against the machine’s full set of devices, not against the allocation – so writing an index there was itself a way to land on the wrong GPU. It now leaves a scheduler’s value alone, and maps back to a physical index only for the nvidia-smi monitor, which ignores the variable.

Neither bug appears where every job sees every GPU, which is why they went unnoticed for so long: they need a scheduler handing out individual GPUs.

An MDI run needs two tasks

An MDI gpu-code launches the engine and the LAMMPS driver as separate MPI ranks, and those counts are written into the command rather than derived from the allocation. A job given fewer tasks than the command asks for dies inside mpirun with:

All nodes which are allocated for this job are already filled.

which mentions neither MDI, nor tasks, nor anything to change – and arrives wrapped in ERROR conda.cli.main_run:execute(127).

This is easy to hit precisely because the obvious reading is wrong: asking for one GPU and one task looks correct and cannot work. The step now counts the ranks the command asks for, compares them with SLURM_NTASKS, and says so before launching.

Configuration consequences

On a machine with GPUs, gres should be overridable rather than defaulted. A default makes every job request a GPU, including CPU-only work, which serialises the queue to the number of cards regardless of any concurrency limit. Jobs that need one ask for it.

Nothing requests a GPU automatically. A machine-learned forcefield run gets one only if the submission includes it. Whether that should be inferred is left open: the forcefield is often only known at run time, from a variable, so the information is not reliably available at submission – which is the same reason a step cannot declare its own resource needs.