seamm_scheduler package

Submodules

seamm_scheduler.backend module

QueueBackend: submit, poll and cancel jobs on a queueing system.

A backend pairs a Scheduler (which commands, which syntax) with a transport (how to run them: on this host, or over ssh). Command construction and output parsing live in the scheduler, so every transport behaves identically.

class seamm_scheduler.backend.QueueBackend(scheduler, transport=None)

Bases: object

Talks to one queueing system through one transport.

Parameters:

• scheduler (seamm_scheduler.scheduler.Scheduler)

• transport (LocalTransport or SshTransport, optional) – How commands run. A subclass may instead override _run().

cancel(job_id)

Cancel a submitted job.

cancel_many(job_ids)

Cancel several jobs in one command.

count_jobs()

How many jobs the user has in the queue now, or None if unknown.

find_jobs(job_name)

The ids of the user’s queued or running jobs called job_name, or None if that cannot be known now (the queue could not be asked).

property host

poll_many(job_ids)

The current status of a batch of jobs, in as few commands as the scheduler allows.

Returns:

{str – Keyed by job id. A job the scheduler has no record of at all (e.g. purged from accounting) is absent; callers should treat “missing” as “can’t confirm this job still exists”.

Return type:

JobStatus}

submit(script, *, job_name=None)

Submit a script (its full text, directives and shebang included – see seamm_scheduler.script.build_script), fed on the submit command’s standard input, so the script never needs to exist as a file the target host can see.

Returns:

The job id.

Return type:

str

seamm_scheduler.config module

Load a JobServer’s <root>/<jobserver-name>.ini: its targets.

System/machine config, not a user preference – lives at <root> (~/SEAMM by default), alongside orca.ini/lammps.ini/ dashboards.ini, not in ~/.seamm.d/seamm.ini. Named after the JobServer instance (its --name, default hostname), one section per target.

A target says where a job’s flowchart evaluator runs (type: local for a subprocess of the JobServer, slurm for a batch job of its own) and, with the keys added for the task layer, where the evaluator’s tasks run (tasks = pool | taskserver | queue). Every key the task layer added is optional: a section without tasks = means exactly what it meant before (whole-flowchart submission, the codes running inside the evaluator).

A section is copied verbatim into each job’s target.json (setup text included), which the Dashboard can show, so a section must never hold secrets.

Lives here, not in seamm_jobserver, so any dependency-light consumer – seamm_jobserver, seamm_webui’s queue list, seamm_exec’s task layer – can read and validate the file without pulling in the rest of the SEAMM stack. See seamm_exec’s docs/developer_guide/campaigns/2026-10-02/.

The keys of a section

Where the flowchart runs and how to reach the queue:

type

local: a subprocess of the JobServer, no scheduler. queue: a batch job on the queueing system named by scheduler. slurm: the original spelling of queue with SLURM. Default slurm.

scheduler

slurm (default), pbs, or seamm (the machine’s TaskServer). Also the scheduler of the tasks when tasks = queue.

transport

local: the scheduler’s commands run here. ssh: they run on host over passwordless ssh.

host

The ssh host, for transport = ssh.

remote_root

ssh only: the directory on the remote host under which each job (and each task, for tasks = queue) gets its own directory, staged there and back with rsync. Without it a shared filesystem is assumed.

remote_run_from_jobserver, remote_conda_env

ssh only: how to start SEAMM on the remote host – the absolute path of its run_from_jobserver (preferred), or a conda environment name.

setup

Shell lines run at the top of the batch script, before SEAMM; indented continuation lines for several. Trusted text, copied into the job’s target.json.

default

In [DEFAULT] only: the section a job uses when it names none.

The JobServer’s behaviour for the section:

max_concurrent_jobs

How many of this section’s jobs are kept submitted at once (default 20).

max_resubmits

How many times a job the queue lost is submitted again (default 3).

poll_interval

Seconds between polls of the queue (default 60).

Submission directives – every other key. Each becomes a directive of the scheduler in its own spelling (SLURM: #SBATCH --<key>=<value> with underscores as dashes), so any option the scheduler takes works. The portable spellings, which PBS maps onto its own, are account, partition (PBS queue), qos, constraint, nodes, ntasks, cpus_per_task, mem, mem_per_cpu, time (PBS walltime), gres and export. A blank value passes no directive. Set mem or mem_per_cpu: a site that reserves the whole node per job by default will otherwise run one job at a time. export = NONE makes SLURM rebuild the job’s environment as from a fresh login, which an ssh submission needs when the cluster loads codes as modules.

The task layer – where the flowchart’s calculations run (all optional; a section without tasks behaves as before the task layer existed):

tasks

pool: in the flowchart’s own allocation (the default). taskserver: through the machine’s TaskServer at url. queue: as batch jobs submitted by the task layer to scheduler.

shared_filesystem

Whether the flowchart and the cluster see the same storage, so task directories need no staging. Default yes for transport = local or a flowchart that is itself a batch job on the cluster; no for ssh.

bundle_tasks, bundle_walltime

How tasks are packed into one batch job: a fixed number, or as many as fit the walltime (seconds, or a SLURM time) from their estimates. Without either, by estimate within the section’s time.

max_walltime

The queue’s longest walltime; a timed-out task’s retry asks for more time but never more than this.

estimate_scale, min_task_walltime

Guards on the walltime of a bundle when it comes from the tasks’ estimates: each estimate is multiplied by estimate_scale (default 1), and each task gets at least min_task_walltime (seconds, or a SLURM time). Still within bundle_walltime and max_walltime.

max_queued_tasks

The most queued and running bundle jobs of this user at once (sites cap a user’s jobs; every array element counts). Shared by every task set a job runs at once.

inline_below

Tasks estimated below this many seconds (default 60) run in the flowchart’s pool when their program is installed there.

remote_python, remote_seamm_root

ssh only: a Python with seamm_exec on the cluster, which runs the task worker and configures each program from the cluster’s own <root>/<program>.ini, and that root (default: the venv’s).

url

tasks = taskserver: the TaskServer’s URL.

Per-job overrides: the companion section [<name>.limits] says which directives a job may override (overridable = a, b, c) and, per directive, <key>.choices, <key>.min and <key>.max. Without the section nothing is overridable.

class seamm_scheduler.config.FieldLimits(choices: list | None = None, minimum: str | None = None, maximum: str | None = None)

Bases: object

Constraints on a per-job override of one directive.

All fields optional: a directive can be listed as overridable with no constraint at all, meaning “any value the scheduler itself accepts.”

choices: list | None = None

maximum: str | None = None

minimum: str | None = None

seamm_scheduler.config.SlurmSection

The historical name.

class seamm_scheduler.config.TargetSection(name: str, transport: str, host: str | None, type: str = 'slurm', directives: dict = <factory>, max_concurrent_jobs: int = 20, max_resubmits: int = 3, limits: dict = <factory>, remote_root: str | None = None, remote_conda_env: str | None = None, remote_run_from_jobserver: str | None = None, setup: str | None = None, tasks: str | None = None, scheduler: str | None = None, shared_filesystem: bool | None = None, bundle_tasks: int | None = None, bundle_walltime: float | None = None, max_walltime: float | None = None, estimate_scale: float | None = None, min_task_walltime: float | None = None, max_queued_tasks: int | None = None, inline_below: float | None = None, url: str | None = None, remote_python: str | None = None, remote_seamm_root: str | None = None, poll_interval: float | None = None)

Bases: object

One target from a <root>/<jobserver-name>.ini file.

property batch_scheduler

The queueing system the evaluator jobs are submitted to.

build_backend()

Construct the backend that submits this section’s evaluator jobs.

build_stager()

Construct the stager for this section’s evaluator jobs – paired with the transport the same way build_backend() picks a backend.

build_task_backend()

The QueueBackend the evaluator submits its tasks through.

build_task_stager()

The stager for task directories: none on a shared filesystem.

bundle_tasks: int | None = None

bundle_walltime: float | None = None

directives: dict

estimate_scale: float | None = None

classmethod from_settings(data)

The inverse of task_settings().

host: str | None

inline_below: float | None = None

property is_batch

Whether the JobServer runs this section’s evaluators as batch jobs.

limits: dict

max_concurrent_jobs: int = 20

max_queued_tasks: int | None = None

max_resubmits: int = 3

max_walltime: float | None = None

merge_overrides(requested)

Merge a job’s requested per-directive overrides on top of this section’s defaults, validating each against self.limits.

Parameters:

requested (dict(str, any)) – Directive overrides a job asked for, e.g. {"ntasks": 4} – typically a job’s parameters["slurm"].

Returns:

The final directives dict (this section’s defaults with the validated overrides applied), ready for seamm_scheduler.script.build_script.

Return type:

dict

Raises:

ValueError – If a requested field isn’t overridable for this section, or its value is outside the configured choices/bounds. Never trust a caller (e.g. a web UI) to have already enforced this – validate here regardless of what constrained the request’s origin.

min_task_walltime: float | None = None

name: str

poll_interval: float | None = None

remote_conda_env: str | None = None

remote_python: str | None = None

remote_root: str | None = None

remote_run_from_jobserver: str | None = None

remote_seamm_root: str | None = None

remote_wdir_for(local_wdir)

The remote scratch path a transport = ssh section’s job should run in (or, for an already-running job, where its files currently live), derived from its local working directory’s name and this section’s remote_root.

Deterministic and side-effect-free – callers besides the one that originally submitted the job (e.g. a Dashboard pulling a running job’s files back on demand, rather than waiting for the job to finish) can recompute the same path independently, as long as they agree on local_wdir and this section’s config. Job directory names (Job_NNNNNN) are unique across the whole datastore, not just within a project, so no collision risk.

Raises:

RuntimeError – If this section has no remote_root set – only meaningful for transport = ssh.

scheduler: str | None = None

setup: str | None = None

shared_filesystem: bool | None = None

property task_scheduler

The name of the queueing system for tasks.

task_settings()

This section as a JSON-serializable dict, for <job>/target.json.

The JobServer writes it into each job’s directory so that an evaluator can find its target wherever it runs (on the JobServer’s host, or as a batch job on a cluster that cannot read this ini file).

property task_transport

How the evaluator reaches the queue for its tasks.

An evaluator that is itself a batch job (type=slurm or queue) runs inside the cluster and submits its tasks there with local commands, whatever transport the JobServer uses to reach the cluster.

tasks: str | None = None

property tasks_share_filesystem

Whether task directories need no staging.

transport: str

type: str = 'slurm'

url: str | None = None

seamm_scheduler.config.list_sections(root, jobserver_name)

Load every routable section from <root>/<jobserver_name>.ini, not just the one load_target() resolves via default=/single-section fallback.

For per-job routing (a job asks for a named queue) and for surfacing “what queues exist” to a submission UI (e.g. seamm_webui’s GET /api/queues).

Returns:

Empty if the config file doesn’t exist.

Return type:

dict(str, TargetSection)

seamm_scheduler.config.load_slurm_config(root, jobserver_name, section=None)

The historical name.

seamm_scheduler.config.load_target(root, jobserver_name, section=None)

Load one target from <root>/<jobserver_name>.ini, if it exists.

Parameters:

• root (str or Path) – The SEAMM root directory (e.g. ~/SEAMM), same as every other per-code .ini file.

• jobserver_name (str) – This JobServer instance’s --name (default: hostname).

• section (str or None) – Which section to use. If None, uses [DEFAULT]’s default = key, falling back to the sole section if there is exactly one and no default is set.

Returns:

None means “no such file” – the caller should fall back to running jobs as local subprocesses, exactly as if this feature did not exist.

Return type:

TargetSection or None

seamm_scheduler.local module

Run queueing-system commands directly on the current host.

class seamm_scheduler.local.LocalTransport(*, drop_env_prefixes=())

Bases: object

Runs commands on this host – the case where the caller (a JobServer, or an evaluator inside an allocation) is itself on a submit host with sbatch/qsub and friends on the PATH.

host = None

name = 'local'

run(argv, input_text=None)

Run a command.

Returns:

(returncode, stdout, stderr).

Return type:

(int, str, str)

seamm_scheduler.pbs module

PBS (PBS Professional and OpenPBS): #PBS directives, qsub/qstat/qdel.

This module implements the Scheduler interface for a second queueing system. It was validated on a real OpenPBS 23.06 site (MolSSI10, 2026-10-03), whose recorded output is replayed by the tests, alongside mocked qsub/qstat output.

A PBS job starts in the user’s home directory, so chdir becomes a cd at the top of the script. Dependencies and other job attributes are -W options (SLURM’s dependency is accepted). Without -V only the PBS_O_* variables reach the job, as with SLURM’s export=NONE.

Resources become one select statement: nodes chunks, each with its share of the MPI ranks (mpiprocs), cores (ncpus), threads, memory and GPUs. partition is the PBS queue (-q). PBS has no QOS, so qos is ignored.

Status prefers qstat -x -f -F json (PBS Pro 18 and later) and falls back to the text table of qstat -x. -x includes finished jobs, whose Exit_status decides between completed and failed.

class seamm_scheduler.pbs.Pbs

Bases: Scheduler

PBS Professional / OpenPBS.

cancel_cmd(ids)

The command that cancels ids.

classify(raw_state)

This scheduler’s state string -> one of the categories above.

count_cmd()

A command listing the user’s own jobs, one per line, or None if the scheduler cannot. Used to respect per-user queued-job limits.

directive_lines(directives)

The script’s directive lines, e.g. ["#SBATCH --ntasks=4"].

directive_prefix = '#PBS'

“#SBATCH”, “#PBS”, …

directives(resources, extra=None)

PBS directives from resources and a section’s/job’s directives.

extra may use PBS names (queue, walltime, select, …) or the portable/SLURM spellings a target section uses (partition, time, ntasks, mem, …). The resources become one select chunk, merged into any select the section gives: the section’s settings (e.g. its mem) are kept unless the resources set them.

env_names = {'job_id': 'PBS_JOBID', 'ncpus': 'NCPUS', 'ngpus': 'NGPUS', 'nodefile': 'PBS_NODEFILE', 'submit_dir': 'PBS_O_WORKDIR', 'threads': 'OMP_NUM_THREADS'}

The scheduler-neutral names -> the environment variables a job sees, e.g. {"job_id": "SLURM_JOB_ID", "ntasks": "SLURM_NTASKS"}. Used by seamm_exec.computational_environment() to recognize and read an allocation.

env_prefixes = ('PBS_',)

find_cmd(job_name)

A command printing the ids of the user’s queued or running jobs called job_name, one per line, or None if the scheduler cannot.

log_directives(directory)

Directives that put the job’s own output in directory.

name = 'pbs'

“slurm”, “pbs”, …

parse_status(stdout, ids)

The table of qstat -x:

Job id            Name             User              Time Use S Queue
----------------  ---------------- ----------------  -------- - -----
1234.pbs01        seamm-task       psaxe             00:00:01 F workq

The table has no exit status, so a finished job’s category comes from qstat -x -f in poll().

parse_submit(stdout)

The job id from the submit command’s output.

poll(run, ids)

The status of ids, using run(argv) -> (rc, out, err).

Returns:

{str – An id the scheduler has no record of is absent.

Return type:

JobStatus}

prologue_lines(directives)

Commands the script runs first, for directives the scheduler has no option for (e.g. PBS has no working-directory option: cd instead).

status_cmd(ids)

The command that reports the state of ids.

submit_cmd(script_path=None, *, job_name=None)

The command that submits a script. With script_path None the script is fed on the command’s standard input.

seamm_scheduler.pbs.classify(raw_state, exit_status=None)

A PBS job_state letter -> one of this library’s categories.

seamm_scheduler.scheduler module

The interface every queueing system implements, and the job status it reports.

A Scheduler knows one queueing system’s syntax and nothing else: how to write its directives, which commands submit, poll and cancel, how to read their output, and the names of the environment variables it sets inside a job. It never runs anything itself. A QueueBackend pairs a scheduler with a transport (local commands or ssh) to actually talk to a cluster.

Adding a queueing system is one module with a Scheduler subclass, plus an entry in SCHEDULERS.

class seamm_scheduler.scheduler.JobStatus(job_id: str, state: str, category: str, exit_code: str | None = None, reason: str | None = None, raw: dict = <factory>)

Bases: object

The status of one queued job, as last polled.

category: str

exit_code: str | None = None

property is_terminal

Whether this job has finished (successfully or not) and is no longer pending or running.

job_id: str

raw: dict

reason: str | None = None

state: str

property task_state

The state in the task layer’s vocabulary (see TASK_STATES).

property timed_out

Whether the queue stopped the job for running past its time limit.

SLURM says TIMEOUT; PBS gives exit status -29 (JOB_EXEC_KILL_WALLTIME) or a comment such as “job killed: walltime 610 exceeded limit 600”.

seamm_scheduler.scheduler.SCHEDULERS = {'pbs': ('seamm_scheduler.pbs', 'Pbs'), 'seamm': ('seamm_scheduler.seamm', 'Seamm'), 'slurm': ('seamm_scheduler.slurm', 'Slurm')}

name -> (module, class)

class seamm_scheduler.scheduler.Scheduler

Bases: object

One queueing system’s syntax: directives, commands and states.

Subclasses set name, directive_prefix and env_names, and implement the *_cmd/parse_* methods. poll() composes them; a scheduler whose status needs more than one command (SLURM: squeue then sacct) overrides it.

cancel_cmd(ids)

The command that cancels ids.

classify(raw_state)

This scheduler’s state string -> one of the categories above.

count_cmd()

A command listing the user’s own jobs, one per line, or None if the scheduler cannot. Used to respect per-user queued-job limits.

directive_lines(directives)

The script’s directive lines, e.g. ["#SBATCH --ntasks=4"].

directive_prefix = None

“#SBATCH”, “#PBS”, …

directives(resources, extra=None)

Translate scheduler-neutral resources into this scheduler’s directive dict (the form directive_lines() and seamm_scheduler.script.build_script take).

Parameters:

• resources (object or dict) – Anything with the attributes (or keys) ntasks, cpus_per_task, mem_per_cpu (bytes), ngpus, walltime (seconds), partition, account, qos and nodes, e.g. seamm_exec.Resources. Missing or None values are left out.

• extra (dict, optional) – Directives in this scheduler’s own spelling (a target section’s site defaults), applied first; the resources override them.

Return type:

dict

env_names = {}

The scheduler-neutral names -> the environment variables a job sees, e.g. {"job_id": "SLURM_JOB_ID", "ntasks": "SLURM_NTASKS"}. Used by seamm_exec.computational_environment() to recognize and read an allocation.

find(run, job_name)

The ids of the user’s jobs called job_name, queued, running or (where the scheduler remembers them) finished; None if that cannot be known now.

find_cmd(job_name)

A command printing the ids of the user’s queued or running jobs called job_name, one per line, or None if the scheduler cannot.

log_directives(directory)

Directives that put the job’s own output in directory.

name = None

“slurm”, “pbs”, …

parse_status(stdout, ids)

{id: JobStatus} from the status command’s output. An id the output does not mention is left out.

parse_submit(stdout)

The job id from the submit command’s output.

poll(run, ids)

The status of ids, using run(argv) -> (rc, out, err).

Returns:

{str – An id the scheduler has no record of is absent.

Return type:

JobStatus}

poll_failed = False

True when the queue could not be asked (a failed command, e.g. ssh could not connect), so a job missing from the answer may still exist. Callers must not take “missing” as “gone” then.

Type:

Set by poll()

prologue_lines(directives)

Commands the script runs first, for directives the scheduler has no option for (e.g. PBS has no working-directory option: cd instead).

status_cmd(ids)

The command that reports the state of ids.

submit_cmd(script_path=None, *, job_name=None)

The command that submits a script. With script_path None the script is fed on the command’s standard input.

exception seamm_scheduler.scheduler.SchedulerError

Bases: RuntimeError

Raised when a queueing-system command fails unexpectedly.

exception seamm_scheduler.scheduler.SubmitError

Bases: SchedulerError

Raised when a job cannot be submitted.

seamm_scheduler.scheduler.TASK_STATES = {'cancelled': 'lost', 'completed': 'finished', 'failed': 'failed', 'pending': 'queued', 'running': 'running', 'unknown': 'running'}

The category of a job -> the task layer’s vocabulary (queued | running | finished | failed | lost). A cancelled job is “lost” to the task layer: something outside it removed the job, so the tasks in it are worth resubmitting.

seamm_scheduler.scheduler.format_memory_mb(nbytes)

Bytes -> whole megabytes with an M suffix, rounded up.

seamm_scheduler.scheduler.format_walltime(seconds)

Seconds -> HH:MM:SS (hours may exceed 24), the form SLURM and PBS both accept.

seamm_scheduler.scheduler.get_scheduler(name)

The scheduler called name (“slurm”, “pbs”, “seamm”).

seamm_scheduler.script module

Build a full batch script from a directives dict and a payload.

Matches the <root>/<jobserver-name>.ini config shape used by seamm_jobserver: each target section’s directive keys (partition, account, qos, nodes, ntasks, time, mem, gpus, …) become the scheduler’s directive lines. A blank/None value means “don’t pass that directive” – let the scheduler’s own defaults apply.

seamm_scheduler.script.build_script(directives, payload, *, shell='/bin/bash', scheduler='slurm')

Build a full batch script: shebang, directive lines, then payload.

Parameters:

• directives (dict(str, any)) – Submission options in the scheduler’s own vocabulary (for SLURM, the plain names such as "partition" – see seamm_scheduler.slurm.Slurm.directive_lines()).

• payload (str) – The body of the script – the commands to run. Used verbatim, after the directive block.

• shell (str = "/bin/bash") – The shebang interpreter.

• scheduler (str or Scheduler = "slurm") – Whose directive syntax to write.

Returns:

The full script text, ready to hand to QueueBackend.submit().

Return type:

str

seamm_scheduler.seamm module

The seamm scheduler: the TaskServer (seamm_scheduler.taskserver).

Driven like SLURM or PBS – locally, or over ssh with rsync staging – by the task layer’s SchedulerBackend and by the JobServer. Its commands run the queue with the target machine’s own Python (python -m seamm_scheduler.taskserver), never a command found on the PATH.

class seamm_scheduler.seamm.Seamm(python=None, root=None)

Bases: Scheduler

The TaskServer on a machine without a queueing system.

cancel_cmd(ids)

The command that cancels ids.

classify(raw_state)

This scheduler’s state string -> one of the categories above.

count_cmd()

A command listing the user’s own jobs, one per line, or None if the scheduler cannot. Used to respect per-user queued-job limits.

directive_lines(directives)

The script’s directive lines, e.g. ["#SBATCH --ntasks=4"].

directive_prefix = '#SEAMM'

“#SBATCH”, “#PBS”, …

directives(resources, extra=None)

cores, memory (bytes), time (seconds) and the rest.

extra may use the portable spellings of a target section (ntasks, cpus_per_task, mem, mem_per_cpu, time, job_name, chdir, output); queue settings the TaskServer does not have (partition, account, qos, export, …) are dropped.

env_names = {'job_id': 'SEAMM_TASKSERVER_JOB_ID', 'memory': 'SEAMM_TASKSERVER_MEMORY', 'ntasks': 'SEAMM_TASKSERVER_NTASKS'}

The scheduler-neutral names -> the environment variables a job sees, e.g. {"job_id": "SLURM_JOB_ID", "ntasks": "SLURM_NTASKS"}. Used by seamm_exec.computational_environment() to recognize and read an allocation.

env_prefixes = ('SEAMM_TASKSERVER_',)

find_cmd(job_name)

A command printing the ids of the user’s queued or running jobs called job_name, one per line, or None if the scheduler cannot.

log_directives(directory)

Directives that put the job’s own output in directory.

name = 'seamm'

“slurm”, “pbs”, …

parse_status(stdout, ids)

{id: JobStatus} from the status command’s output. An id the output does not mention is left out.

parse_submit(stdout)

The job id from the submit command’s output.

python

The Python that runs the queue on the target machine

root

its default, ~/SEAMM)

Type:

The SEAMM root there (None

status_cmd(ids)

The command that reports the state of ids.

submit_cmd(script_path=None, *, job_name=None)

The command that submits a script. With script_path None the script is fed on the command’s standard input.

seamm_scheduler.seamm.classify(state)

seamm_scheduler.slurm module

SLURM: #SBATCH directives, sbatch/squeue/sacct/scancel.

Not every SLURM version supports squeue --json/sacct --json: SLURM 21.08 has it, 20.11 (MolSSI10) does not. Slurm probes for it once per instance and falls back to the text formats. SLURM 25.11 (TinkerCliffs) nests the exit code of sacct --json one level deeper; both shapes are read.

class seamm_scheduler.slurm.LocalSlurm(*, drop_env_prefixes=())

Bases: SlurmBackend

SLURM’s CLI directly on the current host – the case where the caller runs on a SLURM submit host with those commands on PATH.

class seamm_scheduler.slurm.Slurm

Bases: Scheduler

The SLURM queueing system.

cancel_cmd(ids)

The command that cancels ids.

classify(raw_state)

This scheduler’s state string -> one of the categories above.

count_cmd()

A command listing the user’s own jobs, one per line, or None if the scheduler cannot. Used to respect per-user queued-job limits.

directive_lines(directives)

#SBATCH --flag=value lines.

Keys are the plain names in _DIRECTIVE_FLAGS (e.g. "partition", not "--partition"). A blank or None value is skipped. Other keys pass through, underscores becoming dashes ({"gres": "gpu:1"} -> #SBATCH --gres=gpu:1), so a target section can carry options this module does not special-case.

directive_prefix = '#SBATCH'

“#SBATCH”, “#PBS”, …

directives(resources, extra=None)

Translate scheduler-neutral resources into this scheduler’s directive dict (the form directive_lines() and seamm_scheduler.script.build_script take).

Parameters:

• resources (object or dict) – Anything with the attributes (or keys) ntasks, cpus_per_task, mem_per_cpu (bytes), ngpus, walltime (seconds), partition, account, qos and nodes, e.g. seamm_exec.Resources. Missing or None values are left out.

• extra (dict, optional) – Directives in this scheduler’s own spelling (a target section’s site defaults), applied first; the resources override them.

Return type:

dict

env_names = {'cpus_per_task': 'SLURM_CPUS_PER_TASK', 'gpus': 'SLURM_JOB_GPUS', 'job_id': 'SLURM_JOB_ID', 'mem_per_cpu': 'SLURM_MEM_PER_CPU', 'mem_per_node': 'SLURM_MEM_PER_NODE', 'nnodes': 'SLURM_NNODES', 'nodelist': 'SLURM_NODELIST', 'ntasks': 'SLURM_NTASKS', 'ntasks_per_node': 'SLURM_NTASKS_PER_NODE', 'submit_dir': 'SLURM_SUBMIT_DIR'}

The scheduler-neutral names -> the environment variables a job sees, e.g. {"job_id": "SLURM_JOB_ID", "ntasks": "SLURM_NTASKS"}. Used by seamm_exec.computational_environment() to recognize and read an allocation.

env_prefixes = ('SLURM_', 'SBATCH_')

The prefixes of every variable SLURM sets in a job.

find(run, job_name)

squeue for a queued or running job, then sacct for a finished one.

find_cmd(job_name)

A command printing the ids of the user’s queued or running jobs called job_name, one per line, or None if the scheduler cannot.

log_directives(directory)

Directives that put the job’s own output in directory.

name = 'slurm'

“slurm”, “pbs”, …

parse_status(stdout, ids)

{id: JobStatus} from the status command’s output. An id the output does not mention is left out.

parse_submit(stdout)

The job id from the submit command’s output.

poll(run, ids)

The status of ids, using run(argv) -> (rc, out, err).

Returns:

{str – An id the scheduler has no record of is absent.

Return type:

JobStatus}

status_cmd(ids)

The command that reports the state of ids.

submit_cmd(script_path=None, *, job_name=None)

The command that submits a script. With script_path None the script is fed on the command’s standard input.

class seamm_scheduler.slurm.SlurmBackend(transport=None)

Bases: QueueBackend

A QueueBackend for SLURM. Subclasses either pass a transport or implement _run themselves.

exception seamm_scheduler.slurm.SlurmError

Bases: SchedulerError

Raised when a SLURM CLI command fails unexpectedly.

exception seamm_scheduler.slurm.SlurmSubmitError

Bases: SlurmError, SubmitError

Raised when sbatch fails to submit a job.

class seamm_scheduler.slurm.SshSlurm(host, *, ssh_command='ssh', ssh_options=(), timeout=None)

Bases: SlurmBackend

SLURM’s CLI on a remote host over passwordless ssh.

property host

property ssh_command

seamm_scheduler.slurm.classify(raw_state)

Classify a raw SLURM state string into one of this library’s categories.

Parameters:

raw_state (str) – A SLURM job state, e.g. "RUNNING", "COMPLETED", or (as sacct sometimes reports for a cancelled job) "CANCELLED by 1234".

Returns:

One of "pending", "running", "completed", "cancelled", "failed", or "unknown".

Return type:

str

seamm_scheduler.ssh module

Run queueing-system commands on a remote host over ssh.

class seamm_scheduler.ssh.SshTransport(host, *, ssh_command='ssh', ssh_options=(), timeout=None)

Bases: object

Runs commands on a remote host over passwordless ssh – the case where the caller is not on a submit host and reaches the cluster’s login node over ssh instead.

Assumes key-based, passwordless auth is already set up for host (e.g. an ~/.ssh/config alias), the same way a user would run ssh <host> sbatch ... by hand.

name = 'ssh'

run(argv, input_text=None)

Run a command on the remote host.

Returns:

(returncode, stdout, stderr).

Return type:

(int, str, str)

seamm_scheduler.ssh.TASK_SSH_OPTIONS = ('-o', 'BatchMode=yes', '-o', 'ConnectTimeout=30', '-o', 'ServerAliveInterval=15', '-o', 'ServerAliveCountMax=4', '-o', 'ClearAllForwardings=yes')

ssh options for the task layer’s many short connections from a machine whose network comes and goes (a laptop that sleeps or changes networks): fail fast instead of hanging, never prompt, and do not set up the host alias’s port forwards on every command.

seamm_scheduler.stage module

Stage a job’s working directory to/from a remote host.

Needed when a JobServer’s transport = ssh points at a host that shares no filesystem with the JobServer itself – unlike every case validated so far, where the JobServer runs on (or already shares storage with) the SLURM submit host. Paired with the transport the same way LocalSlurm/SshSlurm are: LocalStager for the shared-storage case (a no-op) and RsyncStager for the no-shared-filesystem case.

Lives here, not in seamm_jobserver, for the same reason config.py does: a lightweight consumer should be able to use the transport without pulling in the rest of seamm_jobserver (psutil, GUI code, job-running machinery). See seamm_jobserver’s docs/developer_guide/campaigns/ 2026-08-05/ (Phase 8) for the full design this implements.

The task layer (seamm_exec’s scheduler back end) moves many small task directories at once: JobStager.push() and JobStager.pull() copy a list of paths relative to a base directory in one rsync.

class seamm_scheduler.stage.JobStager

Bases: ABC

Moves a job’s working directory to/from wherever it actually needs to run.

pull(remote_base, local_base, paths, *, exclude=())

Copy paths (relative to remote_base) back under local_base. No-op for LocalStager.

push(local_base, remote_base, paths, *, delete=False)

Copy paths (relative to local_base) to the same places under remote_base. No-op for LocalStager.

abstractmethod stage_in(local_wdir, remote_wdir)

Make remote_wdir ready to run the job, from local_wdir.

Returns the working directory the job should actually use (for the #SBATCH --chdir directive, and for the job’s own command line) – local_wdir itself for LocalStager, remote_wdir for RsyncStager.

abstractmethod stage_out(remote_wdir, local_wdir)

Pull a finished job’s files back, so job_data.json and the flowchart’s results land in local_wdir where the JobServer (and the dashboard) can see them. No-op for LocalStager.

class seamm_scheduler.stage.LocalStager

Bases: JobStager

No-op: the JobServer already shares storage with the SLURM submit host (every case validated so far), so there’s nothing to move.

stage_in(local_wdir, remote_wdir)

Make remote_wdir ready to run the job, from local_wdir.

Returns the working directory the job should actually use (for the #SBATCH --chdir directive, and for the job’s own command line) – local_wdir itself for LocalStager, remote_wdir for RsyncStager.

stage_out(remote_wdir, local_wdir)

Pull a finished job’s files back, so job_data.json and the flowchart’s results land in local_wdir where the JobServer (and the dashboard) can see them. No-op for LocalStager.

class seamm_scheduler.stage.RsyncStager(host, *, ssh_command='ssh', rsync_command='rsync', ssh_options=(), timeout=None)

Bases: JobStager

Pushes/pulls a job’s working directory to/from a remote host with no shared filesystem, over rsync -e ssh. Assumes the same passwordless SSH access SshSlurm does.

A job’s working directory is already self-contained by the time a JobServer picks it up: flowchart.flow, the job_data.json stub, and a data/ directory holding every file the flowchart references – SEAMM’s existing type: "file" control-parameter mechanism (seamm_dashboard_client.dashboard.Dashboard.submit() + safe_filename()) already stages those into data/ and rewrites the command line to job:data/... at submission time, before this ever runs. So staging is just “copy the whole directory,” not a general arbitrary-path resolver.

Known limitation, not handled here: a job://<n>/... cross-job file reference (e.g. a restart checkpoint from another job) points outside the referencing job’s own working directory, so it isn’t staged – such a reference in a job routed through this stager will fail to resolve on the remote host.

pull(remote_base, local_base, paths, *, exclude=())

Copy paths (relative to remote_base) back under local_base. No-op for LocalStager.

push(local_base, remote_base, paths, *, delete=False)

One rsync --files-from for many paths: a bundle’s task directories, say, rather than one ssh connection each.

stage_in(local_wdir, remote_wdir)

Make remote_wdir ready to run the job, from local_wdir.

Returns the working directory the job should actually use (for the #SBATCH --chdir directive, and for the job’s own command line) – local_wdir itself for LocalStager, remote_wdir for RsyncStager.

stage_out(remote_wdir, local_wdir)

Pull a finished job’s files back, so job_data.json and the flowchart’s results land in local_wdir where the JobServer (and the dashboard) can see them. No-op for LocalStager.

exception seamm_scheduler.stage.StageError

Bases: RuntimeError

Raised when staging a job’s files to or from a remote host fails.

seamm_scheduler.taskserver module

The TaskServer: a small queueing system for a machine that has none.

It shares one machine’s cores and memory among the jobs of everything on it – the tasks of several flowcharts’ evaluators, and the evaluators themselves when the JobServer submits them here – so that two jobs on a laptop no longer oversubscribe it. It is driven like SLURM or PBS, by commands, locally or over ssh, through the seamm scheduler (seamm_scheduler.seamm):

python -m seamm_scheduler.taskserver [--root R] submit [--name N] [SCRIPT]
python -m seamm_scheduler.taskserver [--root R] status [--json] [ID ...]
python -m seamm_scheduler.taskserver [--root R] cancel ID ...
python -m seamm_scheduler.taskserver [--root R] find NAME
python -m seamm_scheduler.taskserver [--root R] count
python -m seamm_scheduler.taskserver [--root R] queue
python -m seamm_scheduler.taskserver [--root R] config

A script carries its request in #SEAMM lines, as a SLURM script does in #SBATCH lines: --cores, --memory (bytes, or with a unit), --time (seconds, or HH:MM:SS), --chdir, --output, --name and --kind (task or evaluator). An evaluator – a flowchart’s own process, light and mostly waiting on its tasks – is charged no cores and 1 GB, and tasks are admitted against the tasks running alone, so evaluators can never hold the cores or memory their tasks wait for; an evaluator itself starts only when everything still fits. A job named seamm-<number> (the JobServer’s name for an evaluator) is an evaluator unless it says otherwise.

There is no daemon. The queue is one SQLite file, <root>/taskserver/queue.db; every submit and status call, and every job’s runner when its job ends, makes a scheduling pass that starts the queued jobs that fit, in order. A queued job is claimed by one guarded UPDATE in a BEGIN IMMEDIATE transaction, so concurrent passes never start a job twice. Each job runs under its own runner (run-job) in a session of its own, so nothing that restarts or is upgraded touches it. The runner enforces the job’s time limit – measured with time.monotonic, which stops while the machine sleeps – and its memory: the job is stopped when its processes use more than 125 % of its request for 30 s, and the newest job is stopped when the machine itself runs low on memory.

The machine’s capacity is in <root>/taskserver.ini:

[taskserver]
cores = 8
memory = 8 GB

by default its physical cores and half its memory (the rest is for the desktop, the evaluators and everything else).

seamm_scheduler.taskserver.DEFAULT_MEMORY_PER_CORE = 2147483648

Charged to a job that asks for no memory, per core

seamm_scheduler.taskserver.EVALUATOR_MEMORY = 1073741824

Charged to an evaluator

seamm_scheduler.taskserver.KEEP_ENVIRONMENT = {'HOME', 'LANG', 'LOGNAME', 'PATH', 'SHELL', 'TMPDIR', 'TZ', 'USER'}

The variables a job’s script starts with (and LC_*); the script sets up the rest itself, as a batch job does.

seamm_scheduler.taskserver.KEEP_FINISHED = 2592000

Terminal jobs older than this are removed from the queue (seconds)

seamm_scheduler.taskserver.KILL_GRACE = 10.0

Seconds between SIGTERM and SIGKILL

seamm_scheduler.taskserver.MACHINE_MEMORY_FLOOR = 0.1

Below this fraction of the machine’s memory available, the newest job stops

seamm_scheduler.taskserver.MEMORY_GRACE = 30.0

… for this long (seconds) is stopped

seamm_scheduler.taskserver.MEMORY_LIMIT_FACTOR = 1.25

A job using more than this fraction of its memory request …

seamm_scheduler.taskserver.QUEUED = 'queued'

claimed, its runner not yet running the script.

Type:

Job states. starting

class seamm_scheduler.taskserver.Queue(root=None)

Bases: object

The queue in <root>/taskserver/queue.db.

cancel(job_id)

Cancel a job: a queued one at once, a running one through its runner.

charge(row)

What a job takes from the machine while it runs: (cores, memory).

An evaluator: no cores, and 1 GB (a quarter of the memory of a queue with less than 4 GB, so evaluators can still run there).

check()

Mark jobs whose runner has vanished as lost (a sleeping one is not).

close()

finish(job_id, state, exit_code=None, reason=None)

Record a job’s end, unless it has already ended.

get(job_id)

jobs(ids=None)

schedule(start=True)

Claim the queued jobs that fit, in order, and start their runners.

Returns the ids started.

submit(script, name=None, cores=None, memory=None, time_limit=None, chdir=None, output=None, kind=None)

Add a job; return its id. The request is checked against the machine.

seamm_scheduler.taskserver.RESERVE_AFTER = 1800

The oldest queued job reserves room after waiting this long (seconds)

seamm_scheduler.taskserver.SCHEMA_VERSION = 2

the job’s own pid)

Type:

The version of the queue’s database layout (2

seamm_scheduler.taskserver.START_TIMEOUT = 120.0

A claimed job whose runner has not started after this long is lost

seamm_scheduler.taskserver.capacity(root)

The machine’s {"cores": int, "memory": bytes} for the queue.

seamm_scheduler.taskserver.default_root()

seamm_scheduler.taskserver.format_memory(nbytes)

seamm_scheduler.taskserver.main(argv=None)

seamm_scheduler.taskserver.parse_directives(script)

The request in a script’s #SEAMM --key value lines.

seamm_scheduler.taskserver.parse_memory(value)

Bytes from "8 GB", "512M", "2gb", or a plain number of bytes.

seamm_scheduler.taskserver.parse_time(value)

Seconds from 3600, "1:00:00", "1-00:00:00" or "90s".

seamm_scheduler.taskserver.run_job(root, job_id, poll=2.0)

Run job job_id to its end: time and memory limits, cancellation.

seamm_scheduler.taskserver.settings(root)

The runner’s limits from <root>/taskserver.ini, with the defaults.

Module contents

seamm_scheduler: queueing systems (SLURM, PBS) for SEAMM, with local and ssh transports.