2026-10-03 – MolSSI10: from SLURM to OpenPBS#

Status: design note for Paul and the design session; nothing on MolSSI10 has been changed. Phase 7 of the parallel-execution campaign (seamm_exec docs/developer_guide/campaigns/2026-10-02/) includes “PBS validated on a real PBS site”. seamm_scheduler’s PBS back end has only been tested against mocked qsub/qstat output. The plan is to make MolSSI10 that site by replacing its SLURM with a permanent single-node OpenPBS. SLURM stays covered by ChemAI (local) and TinkerCliffs (ssh).

MolSSI10 today (checked read-only, 2026-10-03)#

  • Debian 11.11, kernel 5.10, 6 cores, 125 GB, 689 GB free on /. Passwordless sudo for psaxe.

  • SLURM 20.11.4 (Debian slurm-wlm, slurmctld, slurmd, slurmdbd) with munge. One node and one partition, batch; select/cons_tres with CR_Core_Memory. Accounting goes through slurmdbd into MariaDB 10.5, whose only database besides the system ones is slurm_acct_db.

  • PostgreSQL 13 runs a pubchemqc cluster on port 5437 (/mnt/ssd1). It is unrelated to SEAMM and must not be touched.

  • SEAMM runs as psaxe: seamm-jobserver and seamm-webui from ~/SEAMM (venv → /mnt/hdd2/psaxe/SEAMM/venv).

  • ~/SEAMM/molssi10.ini has two queues: [molssi10] (this host’s SLURM, transport = local) and [tinkercliffs] (ssh). The default queue is ``molssi10``. So every production job submitted without a queue is an sbatch on this host’s SLURM: 37 jobs since 2026-09-19.

  • ~/SEAMM_DEV/PaulVT.local.ini on the Mac also has a [molssi10] queue (ssh to this host’s SLURM). Its remote_run_from_jobserver still points to the old conda env /home/psaxe/miniconda3/envs/seamm.

What shapes the plan#

  1. The JobServer cannot run a flowchart as a PBS job. A target section’s type is slurm (the whole flowchart is one batch job) or local (a subprocess). PBS is accepted only for a flowchart’s tasks (tasks = queue, scheduler = pbs). The comment in seamm_scheduler/config.py says so: a PBS evaluator needs a new type and a JobServer change. Because MolSSI10’s default queue is its local SLURM, removing SLURM without that change breaks production submissions that don’t name a queue. See decision A.

  2. OpenPBS has no Debian packages, and its last release is v23.06.06 (June 2023). We would build it from source: autotools, with hwloc (2.4.1 is installed), libical, libedit, Tcl/Tk, swig, expat, OpenSSL and the PostgreSQL development files. Its data service runs its own PostgreSQL instance (by default on port 15007, data under PBS_HOME/datastore) using the system’s PostgreSQL server binaries, so it sits beside the pubchemqc cluster without touching it. If the build turns out to need much patching on Debian 11, the fallback is a container (decision C).

  3. Gaps in the PBS back end that only a real site can settle:

    • A PBS job starts in $HOME, not the submit directory as with SLURM. The batch script must cd "$PBS_O_WORKDIR" or -d/-w the directory. build_script adds nothing today.

    • qstat -x needs the server attribute job_history_enable = True (and a job_history_duration), or finished jobs vanish and adoption after a restart cannot tell completed from lost.

    • The environment: qsub without -V passes only PBS_O_*. That is the behaviour we want (SLURM’s export=NONE lesson: ORCA’s clobbered LD_LIBRARY_PATH), but the payload must then set up its own environment, as the bundle worker already does for SLURM.

    • The real qstat -f -F json and qsub output (ids like 12.molssi10), the Exit_status of a qdel-ed or killed job, and qselect, to replace the mocks with recorded fixtures.

    • mem in a select chunk and the mpiprocs/ncpus accounting under cgroups (or none) on one node.

Plan#

Each step that changes the host is done only with Paul’s OK, in a window with no SEAMM jobs running there.

  1. Decisions (below).

  2. Freeze SLURM 20.11 (read-only, no host change). Record real sinfo, squeue, sacct (text, since 20.11 has no JSON) and scontrol show job output for running, finished, failed and cancelled test jobs, as seamm_scheduler fixtures, so the old-SLURM text path stays tested without a host. Snapshot /etc/slurm, the munge key, both [molssi10] ini sections and mysqldump slurm_acct_db.

  3. Build OpenPBS in a scratch directory on MolSSI10 (apt install of the build dependencies only, nothing started). This proves the build before SLURM is touched. Install prefix /opt/pbs, PBS_HOME /var/spool/pbs.

  4. Swap (the window): set the JobServer’s default queue so new jobs don’t go to SLURM (decision A). Wait for running jobs to finish. Stop and disable slurmctld, slurmd, slurmdbd and munge, then apt remove the SLURM and munge packages (configuration kept with remove, not purge). Install OpenPBS, with the server, scheduler, communication daemon and MoM on this host, through its systemd unit. Set up one queue workq with 6 CPUs and the node’s memory as resources, job_history_enable = True (7 days), and no sudo needed for submission.

  5. Make the back end real (seamm_scheduler): the working directory, the environment, recorded fixtures for qsub/qstat/qdel/qselect and job-history states, adoption of queued bundles after an evaluator restart, and a pbs variant in seamm-manager’s ini templates if needed. Keep the mocks for unit tests.

  6. Validate:

    1. the scheduler suite from the Mac over ssh;

    2. a table-writing MOPAC flowchart from MolSSI10’s own JobServer with its tasks on the local PBS queue;

    3. with decision A, the same flowchart as a PBS job;

    4. a MolSSI10 → TinkerCliffs job, proving the production ssh path still works;

    5. the Mac’s SEAMM_DEV [molssi10] queue as PBS over ssh.

  7. Record in these notes, the design doc’s phase 7 and memory.

Rollback: stop and disable PBS, then apt install the same SLURM and munge packages. The configuration is kept by remove, and the snapshot covers /etc/slurm and the munge key; restore slurm_acct_db from the dump. Then restore the ini sections and restart the JobServer.

Budget: about a day for the build and swap, and a day for the back-end work and validation.

Decisions for Paul#

  1. PBS evaluators in the JobServer. Today MolSSI10’s default queue runs each flowchart as a SLURM job. Either:

    1. extend the JobServer in this campaign so a section can run the evaluator as a PBS job: a scheduler-neutral type = queue with scheduler = pbs. This is the honest production test, but it is a JobServer change. Recommended.

    2. make MolSSI10’s default queue local (flowcharts run as subprocesses, as on the Mac) with tasks = queue, scheduler = pbs, so only tasks go through PBS, and leave (1) for later.

    Either way, [molssi10]’s default changes during the window.

  2. MariaDB is used only for SLURM’s accounting. Remove it with SLURM (after the dump), or leave it installed and stopped?

  3. On the host or in a container? OpenPBS in a container on MolSSI10 (images exist for development) would leave the host untouched, but a container’s PBS submitting the JobServer’s jobs on the host is not the “would it work in production” check we want. Recommended: on the host, with the container only if the build fails.

  4. The window: when MolSSI10 can have no jobs running (its production JobServer submits to TinkerCliffs as well as locally).

Paul’s decisions (2026-10-03)#

  • MolSSI10 is no longer production; it is used only for testing this work. Remove SLURM and change its queues freely.

  • Remove MariaDB (after the dump). Install OpenPBS on the host.

  • Debian 11 is end of life (2026-08-31): point apt at the archives.

  • A (implied by the test-only role): the JobServer gets a scheduler-neutral type = queue with scheduler = pbs|slurm.

Done (2026-10-03)#

  1. SLURM 20.11 frozen: devtools/capture_fixtures.py drives the back end over ssh against a real site and records every command; tests/fixtures/slurm-20.11/lifecycle.json (completed, failed 3:0, cancelled, pending on a dependency, unknown id) plus sinfo/scontrol text, replayed by tests/test_recorded_sites.py. Snapshot on MolSSI10 in ~/slurm_snapshot_2026-10-03 (/etc/slurm, munge key, the slurm_acct_db dump, sacct history, the ini); a copy without the munge key in ~/Work/SEAMM/molssi10_slurm_snapshot_2026-10-03.

  2. apt: /etc/apt backed up to ~/apt_backup_2026-10-03; Debian sources now archive.debian.org (bullseye, -updates, -security), pgdg apt-archive.postgresql.org, pgadmin4 commented out, Acquire::Check-Valid-Until "false".

  3. Removed: slurmctld, slurmd, slurmdbd, slurm-client, slurm-wlm, munge and MariaDB (remove, configuration kept; libmunge2 remains).

  4. OpenPBS 23.06.06 built from source in ~/build (build dependencies from Debian; libpq-dev 17 already installed), installed in /opt/pbs, PBS_HOME /var/spool/pbs, systemd unit pbs enabled; commands linked into /usr/local/bin for non-interactive ssh. Its data service runs on port 15007 with the PostgreSQL 15 binaries, beside the pubchemqc cluster on 5437. Two gotchas:

    • /etc/hosts mapped the hostname to 127.0.1.1 (Debian’s default), and PBS insists on an interface address: now 198.82.19.68 (hosts.orig in the apt backup). If the address changes, PBS stops.

    • A first start needs pbs_server -t create; without it the server exits silently after connecting to its database.

    Configured with qmgr: node molssi10 (6 CPUs, 125 GB), queue workq (default, enabled, started), job_history_enable = True (7 days).

  5. Back end made real (seamm_scheduler d3d16c0): dependencies are -W depend= (real qsub refuses -l depend); chdir becomes a cd (Scheduler.prologue_lines); export=ALL becomes -V; find uses qselect -x so finished jobs are found after a restart. Recorded OpenPBS fixtures (tests/fixtures/openpbs-23.06): running, queued, held, completed, failed (3), cancelled (271), unknown id.

  6. JobServer (seamm_scheduler 542ca51, seamm_jobserver 67073be): type = queue + scheduler; is_batch/batch_scheduler; a PBS evaluator job gets -V (like SLURM’s default) unless export = NONE. Web UI (seamm_webui dc587ee): imports from seamm_scheduler instead of the seamm_slurm shim (an old seamm_slurm refused type = queue), and treats type = queue as a batch queue.

  7. Validated from the Mac’s SEAMM_DEV (ini [molssi10] and [molssi10-tasks]; the SLURM version saved as PaulVT.local.ini.slurm-2026-10-03) to MolSSI10’s PBS over ssh, with a test environment ~/SEAMM_PBS there (released phase 4 + the two checkouts):

    • job 4006: the flowchart as a PBS job; results and the seamm.db table staged back, values identical to local runs;

    • job 4007: the same, with each MOPAC calculation a PBS job submitted from inside the evaluator job (tasks = queue, bundle 1, nothing inline); three bundle jobs, exit 0, identical table.

Follow-ups from the design session (2026-10-03)#

Remaining checks against “slurm” (workspace grep):

  • seamm_webui routers/jobs.py: on-demand sync of a remote job’s files tested type != "slurm", so it skipped PBS-over-ssh jobs. Fixed (seamm_webui 6be2499, with a test).

  • seamm_exec computational_environment.py: scheduler == "slurm" / "pbs" picks how to read the allocation (_slurm()/_pbs()); correct as is.

  • seamm_jobserver: mode == "slurm" is the internal name of a job running as a batch job on any scheduler (in job_data-like state and the reattach path); harmless, left as is to keep the state format.

  • seamm_scheduler config.py: the "slurm" alias and the historical LocalSlurm/SshSlurm classes; intended.

  • seamm-manager’s ini templates and seamm_dashboard_client have none.

SLURM 20.11 coverage: the replayed calls are exactly those the back end makes: sbatch --parsable, squeue --json (refused by 20.11) and the text squeue --format=%i|%T|%R, sacct --json (refused) and the text sacct --parsable2, the sacct --allocations and squeue --me finds, squeue -r count, and scancel. The back end does not call sinfo or scontrol; their recorded output (sinfo.txt, sinfo-Nl.txt, scontrol-show-node.txt, scontrol-show-partition.txt) is kept beside the fixture as reference, not replayed.

Draft HISTORY lines (for the release PRs):

  • seamm_scheduler: “PBS validated on a real OpenPBS 23.06 site: dependencies as -W depend, the job starts with a cd into its directory, export=ALL becomes -V, and finished jobs are found again (qselect -x). A section can run the flowchart itself as a batch job on any scheduler: type = queue with scheduler = pbs or slurm (type = slurm still works). Recorded output from real SLURM 20.11 and OpenPBS 23.06 sites is replayed by the tests.”

  • seamm_jobserver: “Flowcharts can run as PBS jobs: a queue with type = queue and scheduler = pbs. Like SLURM’s default, the job gets the JobServer’s environment unless the queue sets export = NONE. Requires seamm-scheduler >= the new version.”

  • seamm_webui: “Queues are read with seamm_scheduler (no longer the seamm_slurm shim); type = queue queues, such as PBS, are listed, and remote ones sync their files on demand. Requires seamm-scheduler >= the new version.”

Review (2026-10-03)#

The design session’s review (a subagent read the eight commits and checked the findings with scripts and molssi10’s qstat history) found four must-fixes, all fixed (seamm_scheduler 19d11c0, seamm_jobserver 0606455, seamm_webui 6be2499 and b177c6b):

  1. Pbs.directives() replaced the section’s select when the resources set ntasks (every bundle does), losing the site’s memory: the live bundles of job 4007 ran without mem. The resources are now merged into the chunk.

  2. The JobServer’s PBS path dropped SLURM-spelled keys (ntasks, mem) from the section and from a job’s overrides; they now become the select. The flowchart job’s output went to $HOME (where qsub ran) and was never staged back; it is now pbs.out in the job directory.

  3. A garbled qstat -F json reply looked like “no jobs” (the JobServer would finalize or resubmit everything); it is now a failed poll.

  4. The pins named the released seamm_scheduler, which lacks type = queue; now >= 2026.10.3. A remote venv needs the new seamm_scheduler too (documented); ~/SEAMM_PBS was updated.

Also fixed: SLURM-form times (1-00:00:00) and walltime limits; “Unknown Job Id” on the text path means gone; cd ... || exit 1; -W block dropped; type lower-cased; qstat -f continuation lines; replay tests check exit codes and poll_failed; the web UI uses section.is_batch; user docs for type = queue/PBS in seamm_scheduler and the JobServer.

Re-validated live (job 4008): the bundles ran with select=1:ncpus=1:mem=4gb:mpiprocs=1, the flowchart job’s pbs.out came back in the job directory, same table, all exit 0. (The output files of jobs 4006/4007, ~/seamm-4006.o17 and ~/seamm-4007.o18 on molssi10, are from before the fix.)

Released and MolSSI10 rebuilt (2026-10-03)#

Released (Paul merged; Releases created after checking each main head): seamm_scheduler 2026.10.3 (#2), seamm_jobserver 2026.10.3 (#25), seamm_webui 2026.10.3 (#11). The downstream CI first failed with “only seamm-scheduler 2026.10.2 is available”: PyPI’s CDN served uv a stale index for a few minutes after the upload, although the JSON API and a plain curl already listed 2026.10.3. Rerun once uv pip compile --no-cache resolves it locally.

MolSSI10’s own ~/SEAMM rebuilt (Paul’s decision): seamm-manager upgraded to 2026.10.2.3; environment migrate (old environment kept as venvs/2026-10-03T17-48-17); environment recreate (it installs the nightly list, so yesterday’s versions); update --latest --all (venv → venvs/2026-10-03T17-50-02: seamm, molsystem, table_step, loop_step, seamm_scheduler, seamm_jobserver 2026.10.3, seamm_exec 2026.10.3.1, mopac_step 2026.10.3.2; web UI 2026.10.3). molssi10.ini: [molssi10] is type = queue, scheduler = pbs, transport = local, workq, select = 1:ncpus=1:mem=20gb, limits ntasks ≤ 6, mem ≤ 100G, time ≤ 24 h; still the default; [tinkercliffs] unchanged (SLURM version saved as molssi10.ini.slurm-2026-10-03). JobServer and web UI (port 55060) running; the legacy dashboard left stopped, as before.

Found while doing it (seamm-manager): environment migrate restarted a JobServer that had been deliberately stopped (stopped again at once; it dispatched nothing), and environment recreate has no --latest, so it cannot pick up a release made the same day. The JobServer’s start-up log still says “SLURM via transport” for every batch queue.

Validated with released packages on both ends:

  • MolSSI10 job 684: default queue, the flowchart as a PBS job on MolSSI10 (PBS job 26), pbs.out in the job directory;

  • MolSSI10 job 685: tinkercliffs queue, the ssh path to TinkerCliffs’ SLURM unchanged;

  • SEAMM_DEV job 4009: the Mac’s SEAMM_DEV (released packages, no editable checkouts of these) to MolSSI10’s PBS, run in MolSSI10’s ~/SEAMM environment.

All three gave the same table as every earlier run, with no caught exceptions. SEAMM_DEV’s [molssi10] and [molssi10-tasks] now use /home/psaxe/SEAMM/venv/bin/run_from_jobserver. The ~/SEAMM_PBS test environment and ~/pbs_dev checkouts on MolSSI10 are no longer used.