Phase 7 notes ============= 2026-10-05 -- design note (draft for Paul and the design session) Phase 7 of the parallel-execution campaign: the **TaskServer** (the machine-wide queue for a computer without a queueing system), the **Dashboard task view**, the triage of the items deferred in phases 1-6, what the PBS test site needs, and the rollout of phases 6 and 7 together. PBS itself was done in phase 4/7 (2026-10-03, see the campaign index). Nothing here is coded yet. Scope ----- From the campaign plan ("The TaskServer"; decision Q4: loopback only, reached through an ssh port-forward) and the design session's list (2026-10-05): 1. A long-lived broker that owns a machine's cores **and memory** and takes the tasks of *every* evaluator on that machine, so two jobs on one Mac no longer oversubscribe it (the problem ``OMPI_MCA_hwloc_base_binding_policy=none`` only papered over), with memory a first-class resource (the 16 GB laptop incidents); reachable from a remote evaluator; surviving its own and the evaluators' restarts; and its relation to the JobServer's local queue. 2. A read-only Dashboard (web UI) task view from the files: per-step task counts and states, a parallel loop's iterations as rows, the "as of the last finished step" wording for a running job. 3. Triage of the deferred items. 4. PBS: what MolSSI10 needs to stay a useful test site. 5. A rollout plan for phases 6 and 7 together. Survey (2026-10-05) ------------------- - **The task layer already has the slot**: a target with ``tasks = taskserver`` reaches ``TaskSet.backend``, which raises "not yet" (``tasks.py``). Any ``TaskBackend`` (``submit``/``status``/``cancel``/``fetch``, optional ``wait``/``adopt``/``reattach``/ ``capacity``) plugs in there. - **A queueing system is one module** in seamm_scheduler (``slurm.py``, ``pbs.py``, about 500 lines each): directives, the submit/poll/cancel/find commands and their parsing, the environment a job sees. ``QueueBackend`` pairs it with a transport (local or ssh) and a stager (rsync, or none on a shared filesystem). The task layer's ``SchedulerBackend`` (bundles, the worker, adoption from manifest records, retries with ``2**n`` time) and the JobServer (``type = queue``) both drive any scheduler module through it, and both were validated over ssh against SLURM (TinkerCliffs) and OpenPBS (MolSSI10), including laptop sleep and lost connections. - **The JobServer has no private state**: it reads and writes ``jobs`` in the datastore's ``seamm.db`` and rebuilds its in-memory view on restart from the pids and queue ids stored there; no HTTP or other IPC. Services are created by seamm-manager from a fixed list (``known_services`` in ``services.py``); launchd and systemd units are generic. - **The web UI** (FastAPI + React) finds a job's directory from the datastore row, lists its files with an unbounded ``rglob`` (large for a parallel loop with many iterations), never reads ``job_data.json`` or any manifest, has no tabs on the job page and no auto-refresh (only a Refresh button, which first syncs a remote job's files back). Backend tests use ``TestClient`` with fake job directories; the frontend has no tests (CI builds and lints it). The TaskServer -------------- Two designs. **(A)** is the one decided in the campaign plan; **(B)** is proposed here because the survey shows how much of it already exists. Paul chooses (D1). **(A) As decided: an HTTP service.** ``POST /tasks``, ``GET /tasks/{id}``, ``GET /tasks/{id}/files/{name}``, ``DELETE /tasks/{id}``, ``GET /programs``; files in the request, results fetched by GET; a pool and an SQLite queue inside the service; bound to ``127.0.0.1``; a remote evaluator reaches it through ``ssh -N -L`` opened by a new ``TaskServerClient`` back end. New: the service, the protocol, its client, file transfer in the protocol, the tunnel's life cycle (a laptop that sleeps or changes network drops it), adoption for the new back end. **(B) Proposed: the TaskServer as a queueing system.** A small queue with a command line in the spirit of ``sbatch``/``squeue``/``scancel``, and a ``seamm`` scheduler module in seamm_scheduler that speaks it: .. code-block:: text seamm-taskserver submit [--cores N --memory BYTES --time SECONDS --name NAME --chdir DIR --output FILE] SCRIPT -> prints the job id seamm-taskserver status [ID ...] (JSON: state, exit code, reason, times) seamm-taskserver cancel ID ... seamm-taskserver find NAME (ids of jobs with that name) seamm-taskserver queue (what is queued and running, for people) seamm-taskserver config (the machine's capacity) Everything that drives SLURM and PBS then drives it unchanged, locally or over ssh: bundling, the task worker, rsync staging, ``adopt`` after an evaluator restart, the retry rules, the queue-full and transient-failure handling, and the JobServer (``type = queue``, ``scheduler = seamm``) for whole jobs. Trust is the user's ssh key, as Q4 intended, and stronger: nothing listens on any port, not even loopback, and there is no tunnel to keep alive across sleep and network changes. The client side is one scheduler module (~300 lines, simpler than PBS since we define the output format), plus an entry in ``computational_environment()`` so a job sees its allocation. The queue itself (both shapes need one): - **State** in one SQLite file per machine, ``/taskserver/queue.db`` (WAL; one short transaction per change), with each job's script, request, state, pid, process start time, exit code and times. A versioned schema, so an upgrade of the package never loses or misreads queued and running jobs. - **Resources**: cores and memory, both first class. The machine's capacity comes from ``/taskserver.ini`` (``cores``, ``memory``), defaulting to the physical cores and *half* the RAM (the machine-wide memory rule; the desktop applications and the evaluators need the rest). A job without a memory request is charged a default per core (2 GB). A request larger than the capacity is refused at submission, never run "alone". - **Scheduling**: first fit in submission order (a small job may start beside a large one waiting for room), with the oldest waiting job reserving room after 30 minutes so a large job cannot starve. - **Running a job**: in its own session, as ``seamm-taskserver run-job ID``, a small runner that sets up the environment (``SEAMM_TASKSERVER_JOB_ID``, ``..._NTASKS``, ``..._MEM``: ``computational_environment()`` reads them as it reads SLURM's), runs the script with the time limit (SIGTERM, then SIGKILL), records the exit code, and runs a scheduling pass so the next queued job starts. - **No daemon** (D2). Every ``submit`` and ``status`` call also runs a scheduling pass, and so does every runner when it ends; a ``status`` pass also checks that each running job's runner is alive (pid and start time) and marks a vanished one ``lost`` (after a reboot, say). There is therefore nothing to keep running and no service to manage: the queue works whether or not anything is watching it, an upgrade or restart of anything never touches running jobs, and a laptop that sleeps simply resumes. *(Alternative: a ``taskserver`` service created by seamm-manager, KeepAlive, polling every second, which can enforce limits and start jobs without waiting for a call; the survey lists the six places in seamm-manager that would learn its name.)* - **Where it lives**: in seamm_scheduler (console script ``seamm-taskserver``), which is already in every SEAMM venv, locally and on a remote workstation; no new package. It depends on nothing but the standard library and psutil, and runs scripts, not SEAMM code: it changes rarely. **Targets** (evaluator side, in ``.ini`` / ``target.json``):: [workstation] ; a second machine without a queue type = local tasks = queue scheduler = seamm transport = ssh host = workstation.local remote_root = /home/paul/seamm_tasks remote_python = /home/paul/SEAMM/venv/bin/python [local] ; this Mac: every job's tasks share its queue type = queue ; the JobServer submits the evaluators to it too scheduler = seamm transport = local tasks = queue On the Mac, the JobServer's ``[local]`` section can stay ``type = local`` (evaluators as subprocesses, tasks in each evaluator's own pool -- today's behaviour) or become ``type = queue, scheduler = seamm``, so the evaluators and all their tasks share one accounting of the machine (D3). The JobServer then *is* a TaskServer client, through the same scheduler module it already uses for SLURM and PBS; no new code in it. **Adoption and restarts**: the evaluator's manifest holds the queue's job ids, so ``SchedulerBackend.adopt`` works as for SLURM. The queue survives its own "restart" trivially (there is no process); runners are independent of evaluators. **Validation** (D4): unit tests of the queue (scheduling, limits, lost runners, a concurrent-submit soak against SQLite locking); the scheduler module against recorded output, as for PBS; on the Mac, two jobs of concurrent MOPAC tasks through the JobServer with ``[local] type = queue`` showing the core and memory limits held; one task run over ssh to a second machine (paul.local, with Paul's OK) with staging; a kill soak of evaluators and runners. All small (MOPAC, a few tasks at once). Dashboard task view ------------------- Read-only, from files, no new database (D5): - **Backend**: ``GET /api/jobs/{id}/tasks`` walks the job directory for ``tasks/manifest.json`` (skipping ``_bundles``, bounded depth) and returns, per step directory, the counts by state and per task its key, state, attempts, back end and id, bundle and the reason of a failure; and for each parallel loop (a step directory whose iterations have ``_evaluator/``), one row per iteration: its directory, its state (from ``_evaluator/job_data.json``, read with the JobServer's header tolerance, or the checkpoint), whether it has been merged or failed (from the parent's checkpoint frame: ``next``, ``failed``, ``directories``), with links to its ``job.out`` and ``iteration.out``. Path safety through the existing ``_resolve_job_file``. A cheaper file list for big jobs (a depth or a subtree parameter on ``/files``) comes with it. - **Frontend**: a Files | Tasks switch on the job page; the Tasks panel a table grouped by step, iterations as rows (phase 6's deferred "child rows" -- inside the job, not as datastore rows); refreshed every 15 s while the job is running. For a remote job the panel says "as of the last sync" and the existing throttled sync runs first; the database-derived panes say "as of the last finished step" (a running job's database shows only finished steps). - **Tests**: backend tests with fake job directories (manifests, iterations, a parallel checkpoint); the frontend is checked live in a browser with a real parallel-loop job (Playwright, as the web UI's earlier features were). Deferred items -------------- ===================================== ===== ============================================= item what why / how ===================================== ===== ============================================= Dimer builder wall-walk per-point defer Dimer-builder work, not task-layer; with its TaskSets on a queue next campaign. MOPAC ``success.dat`` skip-rerun vs do A stale ``success.dat`` must not skip a task the task fingerprint whose inputs changed: the fingerprint decides, ``success.dat`` only for legacy directories. Small, mopac_step. Remote staging cleanup (stage-back do Pruned directories come back on the next of pruned directories) stage-out; prune on the remote too. The TaskServer over ssh uses the same staging. ``conda run`` PATH fragility do Launch the env's own interpreter (mopac_step#161) (``/bin/python``) instead of ``conda run ... python``; check ORCA's MDI launch the same way (consider each code). Warm evaluator per bundle (5 s defer Phase 8, if real loops need it: measure the start-up per iteration) start-up on ChemAI/TC first. TIMEOUT seen on reattach not doubled; do Small task-layer fixes: a timeout found when ``2**n`` capped only by adopting counts like one seen live; cap the ``bundle_walltime`` doubling at the queue's maximum walltime too. Flaky do Make it measure overlap of the tasks' ``test_tasks_run_concurrently`` running intervals, not total elapsed < 2.5 s. Timing-file helper (vasp-step#18) do A locked, slim-row append helper in seamm_exec (``fcntl`` lock, rotation by size); vasp-step adopts it in its own release. seamm_manager#26 (migrate starts do Before the rollout: these change shared stopped services), #27 (recreate environments, #28/#29 already damaged one. --latest), #28 (installer applies its yml to a hand-built env), #29 (update runs refused packages' installers) Release vs docs ``gh-pages`` push do In devops: one concurrency group for every collision (properties_step, twice) workflow that pushes ``gh-pages``, so the docs push of CI-on-main and of Release queue instead of failing (and skipping PyPI). ===================================== ===== ============================================= PBS test site (MolSSI10) ------------------------ - ``job_history_enable`` on with ``job_history_duration`` of at least 7 days, so ``qstat -x`` still reports a job that finished while the laptop was asleep for a long weekend (the JobServer and ``SchedulerBackend`` both rely on it to find terminal states). - Remove the orphan test jobs and their remote directories left by the 2026-10-03 validation (list them first; MolSSI10 is test-only, so with a one-line OK). - ``max_resubmits``: PBS loses jobs the same ways SLURM does (node failure, ``qdel`` by an admin, walltime); the same guidance holds -- 3 for production sections, more for a deliberately short-walltime test section -- and the JobServer's user guide says so for both schedulers. Rollout of phases 6 and 7 ------------------------- Paul decides when, timed with the EC pilot (D6). Phase 6 is opt-in and phase 7's queue is opt-in, so the rollout only needs the packages, not a configuration change: - Packages: molsystem, seamm, seamm-exec, loop-step, read-structure-step, table-step, properties-step, lammps-step, orca-step, geometry-analysis-step (phase 6, 2026.10.5); seamm-scheduler, seamm-exec, seamm-webui, seamm-manager and mopac-step at their phase 7 versions. - TinkerCliffs (``/projects/seamm/SEAMM``): ``update`` of exactly that list with ``--dry-run`` first, never ``--all`` (the xnn/lammps environment caveat); no services. - ChemAI: hands-off except on Paul's explicit ask, for this list only; restart its JobServer and web UI after, between jobs. - Memory rule: nothing on the laptop beyond the small validation runs above. Decisions for Paul ------------------ - **D1 -- the TaskServer's shape.** Recommended: **(B)**, the queue with a command line and a ``seamm`` scheduler module -- the same trust as Q4 (ssh only) without a listener or a tunnel, and nearly all of the client already built and tested. (A) is the plan as written. - **D2 -- daemon or none.** Recommended: no daemon (passes on every call and when a job ends); the alternative is a seamm-manager service. - **D3 -- the Mac's own jobs.** Recommended: offer ``[local] type = queue, scheduler = seamm`` as an option, switched on for SEAMM_DEV first; the default stays as today. - **D4 -- validation** as above, including one ssh run to paul.local. - **D5 -- the task view** as above, read-only from files, iterations as rows within the job. - **D6 -- the deferred items and the rollout** as triaged above; the rollout when Paul says. Packages and order ------------------ seamm_scheduler (the queue, ``seamm-taskserver``, the ``seamm`` scheduler module) → seamm_exec (``computational_environment``, the retry fixes, the timing helper, the flaky test) → seamm_webui (task view) → mopac_step (``success.dat``, #161) → seamm_manager (#26-#29) → devops (the ``gh-pages`` concurrency group). The PBS site items are configuration on MolSSI10, not releases. Review (design session, 2026-10-05) ----------------------------------- The design session recommends (B). Folded in: 1. **No deadlock with D3.** With the Mac's evaluators in the same queue as their tasks, N one-core evaluators on N cores would hold every core while their tasks wait. Rule: a job submitted by the JobServer as an evaluator is charged **no cores** and a small memory (1 GB) -- evaluators are light and mostly waiting; only tasks are charged their cores. Tested with capacity 2 and three jobs each with a 2-core task. 2. **Atomic claim without a daemon.** Passes run from concurrent callers (several evaluators' status polls, runners ending), so a pass claims a queued job with one ``UPDATE ... SET state = 'starting' WHERE id = ? AND state = 'queued'`` in a short ``BEGIN IMMEDIATE`` transaction and starts it only if one row changed; the soak runs concurrent passes as well as concurrent submissions. 3. **Time limits across sleep.** A job's time is measured with ``time.monotonic()``, which stops while the machine sleeps on both macOS and Linux (Linux's ``CLOCK_MONOTONIC`` excludes suspend; ``CLOCK_BOOTTIME`` is the one that does not), so a laptop closed overnight does not time out its jobs on waking. A sleeping runner is never "lost": only a vanished pid, or a pid whose start time differs, is. 4. **Memory enforced, not just accounted.** The runner watches its job's process tree (the job's session, so detached MPI ranks too) and stops the job -- SIGTERM, then SIGKILL -- when it uses more than 125 % of its request for 30 s, recording ``memory`` as the reason. On macOS RSS misses compressed and swapped pages (the ``mac-memory-guard-rss-blind`` lesson), so the measure is the process footprint where psutil offers it (``memory_full_info().uss``), and a second guard watches the machine: below 10 % available memory the newest job is stopped. seamm-manager writes the capacity into ``/taskserver.ini`` on install, explicitly (physical cores, half the RAM), rather than leaving a silent default; the 2-4 GB rule for the Claude sessions' own local work is separate and unchanged. Nits taken: over ssh the queue is invoked as `` -m seamm_scheduler.taskserver ...``, never a command on the PATH (the ``conda run`` lesson); a versioned ``queue.db`` schema with a migration test; ``seamm-taskserver queue`` shows memory as well as cores; the task view refreshes only while the job is running, and the ``/files`` depth parameter defaults to 2; seamm-mbe and mbe-step join the ChemAI rollout list only if Paul wants them there; the PBS orphan-job removal waits for Paul's one-line OK. Decisions (Paul, 2026-10-05) ---------------------------- D1 (B): the TaskServer is a queue with a command line and a ``seamm`` scheduler module. D2: no daemon, with the atomic claim and sleep-safe timing of the review. D3: the Mac's own jobs through the queue as an option, switched on in SEAMM_DEV first; evaluators charged no cores. D4-D6 as proposed, except that the ssh validation uses the Mac mini at work (``macmini`` in ``~/.ssh/config``: 8 cores, 16 GB, its own ``~/SEAMM``) rather than paul.local. The rollout of phases 6 and 7 waits for Paul's word, timed with the EC pilot. Implementation (2026-10-05) --------------------------- Local commits on ``dev``, not pushed; test venvs ``~/SEAMM_DEV/venvs/phase7-B`` (released packages + editable checkouts) and ``phase7-webui``. - **seamm_scheduler**: ``taskserver.py`` -- the queue (``queue.db``, schema 2 with a migration), scheduling passes with the atomic claim, first fit with the 30-minute reservation, the runner (``time.monotonic`` limits, memory by the session's unique set size with a 30 s grace, the machine-wide floor), the command line, and, found in testing, two rules: evaluators charged no cores *and* never counted against tasks (three 1-GB evaluators deadlocked a 2-GB queue through memory, not cores), and a job whose runner vanished is stopped before it is marked lost (its script runs in a session of its own and a rerun would have run beside it). ``seamm.py`` -- the ``seamm`` scheduler (``#SEAMM`` directives, ``python -m seamm_scheduler.taskserver`` with the section's ``remote_python``/``remote_seamm_root``). ``max_walltime`` in target sections. Staging: SQLite side files and ``loop_entry.db`` are mirrored by listing them, not with rsync filters -- openrsync (macOS) protected the directories' contents under ``P */``, so the phase 5 fix never removed a *nested* stale log (found in the web UI: job 4014 kept a stale ``loop_entry.db``). - **seamm_exec**: ``computational_environment`` reads a TaskServer job; the retry doubling capped by ``max_walltime``; a reattach-seen timeout is already doubled (regression test; a job the queue has forgotten cannot be known to have timed out); remote copies of a bundle removed once staged back; ``seamm_exec.timing``; ``test_tasks_run_concurrently`` by overlap. - **mopac_step**: a stale ``success.dat`` no longer skips changed input; the MDI engine by the environment's Python path (#161, not reproducible with conda 24.11.3 here). - **seamm_webui**: ``/api/jobs/{id}/tasks`` and the Files | Tasks switch; ``/files`` takes an optional ``depth`` (no default, so the file browser keeps its full tree). Its ``test_sync`` tests expect seamm_scheduler's one-call stage-out and fail against 2026.10.4: the web UI must pin the new seamm_scheduler. Validation: - Unit tests: seamm_scheduler 212 (TaskServer 16, staging 17), seamm_exec 152, the web UI 56, mopac_step 43. The web UI's Tasks tab checked in headless Chromium on job 4014 (24 iterations, 25 steps with tasks, jump to an iteration's ``job.out``). - **Two jobs sharing the Mac** (SEAMM_DEV ``[local] type = queue, scheduler = seamm``, capacity 4 cores / 4 GB): jobs 4015 and 4016, each a parallel loop of 8 MOPAC iterations at 1 core / 1 GB: 16 iteration tasks and 2 evaluators; at most 4 tasks (4 cores, 4 GB) ever ran at once across both jobs, both evaluators alongside; both finished, identical to each other (and within 0.07 kcal/mol of the TinkerCliffs serial reference, a Linux MOPAC). - **Over ssh to macmini** (no shared filesystem): four MOPAC tasks in two bundles queued and run by macmini's TaskServer (private venv ``~/phase7_test``), staged both ways, results back, nothing left under ``remote_root``. - **Kill soak** on SEAMM_DEV: the evaluator of job 4018 killed mid-loop -- its queue job failed (-9), the JobServer resubmitted it (``resubmit_count`` 1), the new evaluator adopted the iteration tasks, identical results; the runner of an iteration task of job 4019 killed -- the task lost and its job stopped, the iteration rerun and resumed, identical results. The rest (2026-10-05): - **seamm_manager** (#26-#29 and the TaskServer's capacity): a service recreated by an environment change is no longer started when it was stopped (``create_service(..., start=...)``), and updates restart the JobServer and web UI only if they ran (``restart_if_running``); ``environment recreate --latest``; an installer leaves alone a conda environment named in its .ini that SEAMM did not make (no ``seamm-*.sha256`` record and not SEAMM's name -- note: the TinkerCliffs ``seamm-lammps-xnndev`` got such records in the 2026-10-04 incident; remove them by hand before relying on this); ``update`` runs no installer for a refused package; ``install`` writes ``/taskserver.ini`` if missing. - **devops** (local branch ``release-docs-nonblocking``, not pushed): the Release workflow's docs deploy is ``continue-on-error``. The collision was with the package's own Docs workflow (``on: push``, its ``buildDocs.sh`` pushing ``gh-pages``) for the merge to main and ``make update``'s push to dev; a concurrency group would not do, since GitHub cancels all but one *pending* job in a group. - **seamm_jobserver**: user guide -- ``max_resubmits`` for SLURM, PBS and the TaskServer, and PBS job history. - **seamm_scheduler** docs: a TaskServer page. - **PBS (MolSSI10)**: ``job_history_enable = True``, ``job_history_duration = 168:00:00`` already; no jobs queued or running; the 22 leftover staged directories (``~/seamm_dev_remote_jobs``, 38 MB, 2026-08-09 to 2026-10-03) removed with Paul's OK (2026-10-05). The devops change is devops#3, merged 2026-10-05. State left for the soak: SEAMM_DEV runs from ``venvs/phase7-B`` with ``[local] type = queue, scheduler = seamm`` (backup ``PaulVT.local.ini.bak-2026-10-05-phase7``) and ``~/SEAMM_DEV/taskserver.ini`` (4 cores, 4 GB); macmini has ``~/phase7_test`` (a private venv and a queue); ``~/SEAMM_DEV/venvs/phase7-webui``. SEAMM_DEV jobs 4015-4019 are the validation record. Answers from the design session (2026-10-05, to the three questions with the code): 1. *The evaluator rule* (tasks never count evaluators) is right; its failure mode is recorded here: N evaluators can use more than their 1 GB charge each (one holding a large job database or a parallel loop's snapshots), and the machine-wide floor then stops the *newest* job, which may be a task rather than the evaluator that caused it. The stopped job's reason says why ("the machine was low on memory ... the newest job was stopped"). The real protection is the JobServer's ``max_concurrent_jobs``, which caps the evaluators running at once. 2. ``/files`` without a default depth is fine (the browser needs the full tree); the walk for ``/tasks`` stays bounded, and ``depth`` is documented for API users (the route's docstring, in the API's OpenAPI page). 3. No TaskServer service: the runners check the floor while anything runs, and when nothing runs there is nothing to protect; memory eaten by *other* processes while TaskServer jobs run is covered by the same check. Code review (design session, 2026-10-05) and fixes -------------------------------------------------- The review found the queue's core sound (atomic claim, migration, lost detection, cancel, staging) and three blocking defects, all fixed (seamm_scheduler 95d86ee): 1. **The 30-minute reservation could deadlock the queue**: the oldest queued job could be an evaluator, whose ``break`` then held up every task -- including those of the running evaluators, which therefore never finished. Now only a task reserves, and only against later tasks; a task larger than a lowered capacity fails instead of reserving for ever. Tested (four evaluators fill a 2 GB queue, a fifth waits an hour, the running ones' task still starts). 2. **A slow runner could be marked lost while starting its script** (check's SELECT, then finish): now one guarded ``UPDATE ... AND pid IS NULL``. 3. **A job inherited the environment of whoever made the scheduling pass** (another job's ``SEAMM_CE``, threads, ids): a job's script now starts from a minimal environment, as ``export=NONE``; the local transport drops ``SEAMM_TASKSERVER_`` and ``SEAMM_CE``. Tested. Also: the machine-wide floor waits its grace and stops one job per low-memory episode (state in ``meta``; read-only while memory is fine); a job's processes include descendants with sessions of their own (a pool's codes) for stopping and for memory; the runner always records its job's end (try/finally); over ssh the queue needs ``remote_python``; the evaluator charge is at most a quarter of a small queue's memory. The timing helper's rotation race (a writer waiting on a set-aside file rotating the new one onto the same name) is fixed with an inode check and unique names (seamm_exec d779a10). mopac_step: ``reuse_previous_run`` with a test, and ``_conda_python`` takes a path or asks conda (8a4d8d8) -- checked with a real MOPAC flowchart after lint caught a ``success`` variable left behind, which no unit test exercises. Web UI (0291f63): nested parallel loops find their frame in the iteration's own checkpoint; ``/files?depth`` prunes its walk and rejects ``depth < 1``. Kept, with reasons: the queue's root stays machine-wide (the default installation's root unless ``remote_seamm_root``), so that installations on one machine share its capacity rather than each using all of it; documented on the TaskServer page. Pins at release preparation: seamm_exec and the web UI on the new seamm_scheduler. Re-check (design session, 2026-10-05): approved for release preparation; agreed on the machine-wide queue root (the TaskServer page now says that an installation with ``SEAMM_ROOT`` set gets its own queue). Nit for the next devops change: a summary line when the Release's docs deploy fails (it is now a red step in a green job).