2026-08-06 – Initial scaffold#

Status (2026-08-06, later)#

Wired into seamm_jobserver’s whole-flowchart SLURM submission mode and validated end-to-end against a real cluster (MolSSI10), isolated from its live production JobServer: real sbatch-to-completion runs, a genuine resubmit-and-give-up sequence under repeated real SLURM failures, and a deterministic kill/restart test confirming reattachment resumes a still-live SLURM job without duplicate submission. One correction to the design below: no conda activation turned out to be needed in the sbatch payload after all (see “Not in this package”).

Scope#

seamm_slurm is Phase 1 of the seamm_jobserver SLURM-integration plan (full plan: ~/Work/SEAMM/jobserver-slurm-plan.md, a workspace-root living doc, not packaged). That plan calls for two long-term submission models (“Option 1”: JobServer submits a whole flowchart run as one SLURM job; “Option 2”, future: seamm_exec submits individual heavy codes per step) sharing one SLURM-talking backend, rather than duplicating sbatch/ squeue/sacct handling in each consumer. This package is that shared backend.

Design points#

  • Two transports behind one interface. LocalSlurm runs the SLURM CLI directly (the process is on a SLURM submit host, e.g. a cluster head/login node). SshSlurm runs the identical commands on a remote host over passwordless SSH. Both subclass SlurmBackend and only need to implement _run(argv, input_text) -> (returncode, stdout, stderr) – all submit/poll/cancel logic and JSON-vs-text parsing lives once, in the base class.

  • JSON when available, text parsing otherwise, decided per-backend at runtime. Confirmed during Phase 0 groundwork that this is a real, not hypothetical, need: ChemAI runs SLURM 21.08.5 (squeue --json/ sacct --json both work, OpenAPI v0.0.37); MolSSI10 runs SLURM 20.11.4 (--json unrecognized on both commands). SlurmBackend probes once per command per backend instance and remembers the result.

  • ``sacct``’s and ``squeue``’s JSON schemas are not analogous – confirmed against real completed jobs on ChemAI. squeue --json’s job_state is a flat string ("COMPLETED"). sacct --json’s state is a nested object ({"current": "COMPLETED", "reason": ...}), and its exit_code is similarly nested ({"status": "SUCCESS", "return_code": 0}). The two JSON parsers are separate for this reason, not just for the different field sets.

  • Submission is filesystem-agnostic by construction. submit() takes the full script text and pipes it to sbatch --parsable on stdin – never needs the script to exist as a file the target host can read. This sidesteps a real Phase-0 finding: there is no shared filesystem between this Mac and either cluster host, so a submitted job’s script can’t simply be “the same file” the caller wrote locally, whether that be the future SshSlurm cross-host case or a caller (like a JobServer) that just wants to submit without worrying about where the target host will look for a file.

  • No SEAMM-status vocabulary here. JobStatus.category classifies a SLURM job as one of pending/running/completed/cancelled/ failed/unknown – a SLURM-domain concept, not a SEAMM one. Mapping that to seamm_jobserver’s own jobs.status column values (submitted/running/finished/…) is Phase 2’s job, done in seamm_jobserver itself, keeping this package genuinely reusable (including, later, by seamm_exec, per Option 2).

  • ``poll_many()`` batches. One squeue call plus (for anything squeue no longer lists) one sacct call per polling cycle, not one call per tracked job – matters once a JobServer instance is tracking many outstanding jobs.

Not in this package#

  • Building the actual sbatch payload for a SEAMM flowchart run (the run_from_jobserver invocation – no conda activation needed, since it’s invoked by its full absolute path and SLURM inherits the submitting environment) – that’s seamm_jobserver’s job (Phase 2), using script.build_script() from here for the #SBATCH boilerplate.

  • The <root>/<jobserver-name>.ini config file and its multi-section (multi-cluster) routing – also Phase 2/seamm_jobserver.

  • A generic multi-scheduler abstraction (PBS/LSF/etc.) – the config format this package’s caller uses leaves room for a type = key per cluster section, but no non-SLURM backend is implemented or planned here yet.