Phase 8 notes ============= 2026-10-05 -- candidate list for Paul (no design yet) Phases 0-7 of the parallel-execution campaign are released. This page gathers what was deferred along the way. The items were scattered through the phase 6 and 7 notes, their reviews and the design session's messages. For each one: what it is, why it was deferred, the evidence so far, a rough effort, and a recommendation. Paul picks; a design note follows for whatever is chosen. Effort: *S* is under a day, including the release; *M* is a few days; *L* is a week or more, or needs live runs on the clusters. Summary ------- == ========================================================== ======== ========= # item effort recommend == ========================================================== ======== ========= 1 Warm evaluator per bundle (parallel-loop start-up) L measure first 2 Dimer builder wall-walk as per-point TaskSets on a queue M defer 3 Packaging: sdists that cannot be built (8 packages) S decided 4 Dashboard: child iterations as datastore rows M-L defer 5 devops: a summary line when the docs deploy fails S decided 6 Run-path tests of the code steps (mopac, ORCA first) S-M decided 7 Legacy structure-handling wording (strain_step) S done 8 seamm_exec test flakiness: the timing-sensitive tests S done 9 PBS site: job history and ``max_resubmits`` S done; verify 10 Task view at 500 iterations S measure 11 TaskServer as the default local queue S defer until soaked 12 Separate placement of a parallel loop's codes, live M do with EC pilot 13 Timing helper adoption by the plug-ins (vasp-step#18) S each timing campaign 14 molsystem#121: periodicity 0 on an empty configuration S done 15 orca_step: cap the cores for small molecules S not needed == ========================================================== ======== ========= The items --------- 1. Warm evaluator per bundle ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* A bundle of parallel-loop iterations would run in one long-lived evaluator, instead of starting a fresh ``run_from_jobserver`` for each iteration. *Why deferred.* Phases 6 and 7 aimed at correctness and placement, and the start-up cost had not been measured where it matters. *Evidence.* About 5 s of evaluator start-up per iteration on the Mac. In the tiny MOPAC loop, p1 took 11 s serial against 21 s in parallel with 2 at once. On TinkerCliffs (``p8_tc``, about 10 s of MOPAC per iteration) start-up hid any speed-up: 1:48 serial, 1:50 inline with 4 at once, and 2:27 bundled as queue jobs. Nothing has been measured on ChemAI, nor TC's start-up on its own (imports over GPFS, the plug-in scan). *Effort.* L. The iteration's independence contract (its own database snapshot, checkpoint and merge) has to hold for several iterations in one process, with no state leaking between them. The kill and resume soaks would need repeating. *Recommendation.* Measure first (S): time ``run_from_jobserver`` start-up alone on ChemAI and on a TC compute node, and set it against the EC pilot's real iteration times. Build it only if start-up is a meaningful share of a real loop. 2. Dimer builder wall-walk as per-point TaskSets ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* The wall walk's points would be TaskSets on a queue, rather than run in turn inside the step. *Why deferred.* This is dimer-builder work, not task-layer work; triaged in phase 7 as "with its next campaign". *Evidence.* None new. *Effort.* M. *Recommendation.* Defer to the dimer builder's next campaign. 3. Packaging: sdists that cannot be built ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* A source distribution from which no wheel can be built. Measured 2026-10-05 by building every package in the workspace with ``uv build``, which builds the wheel from the sdist as ``pip`` does: 67 of 83 build. The released packages that fail have two causes: - **The sdist misses files ``setup.py`` reads** (``versioneer.py``, ``requirements.txt``): custom_step, forcefield_step, loop_step, supercell_step, table_step. The fix is two ``MANIFEST.in`` lines (``include requirements*``, ``include versioneer.py``). seamm_jobserver had the same gap, fixed in 2026.10.5. - **An old vendored ``versioneer.py`` calling ``configparser.SafeConfigParser``**, removed in Python 3.12, so nothing builds at all: strain_step (last released 2022), crystal_builder_step (2022), set_cell_step (2021). Their next release would fail in CI. The fix is the current ``versioneer.py`` from the cookiecutter template. The other failures are local prototypes with no repository (conformer_search, data_sources, pyscf, query, trajectory_analysis) or stale side copies (``forcefield_step_experimental``, ``strain_step_sv``). A ``pyproject.toml`` is *not* needed: the 67 that build have none. The first version of this item said otherwise. *Impact.* None for users today. Every release ships a wheel, which ``pip``/``uv`` install, so the sdist is only used by something building from source. *Decision (Paul, 2026-10-05).* Fix each package as part of its next release, not in releases of its own; strain_step together with item 7. The release skill's pre-flight now runs ``uv build && twine check`` and says to fix a failing build, never to fall back to ``python -m build --sdist --wheel``, which builds both from the source tree and hid seamm_jobserver's gap. 4. Dashboard: child iterations as datastore rows ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* Each parallel-loop iteration would be a row in the jobs database (``parent_id``), listed, filtered and linked like a job. *Why deferred.* Phase 6 left it for "later". Phase 7 delivered the in-job task view instead: iterations and tasks read from the job's files, with no schema change. *Evidence.* The task view works on job 4014 (24 iterations, 25 steps with tasks). Nobody has asked for cross-job queries over iterations. *Effort.* M-L. It needs: - a datastore migration; - the JobServer and evaluator writing the rows; - the web UI showing them; - the rows kept consistent with resume and merge. *Recommendation.* Defer until a concrete need appears, such as searching iterations across jobs or showing per-iteration status for remote jobs without fetching files. 5. devops: a summary line when the docs deploy fails ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* The Release workflow's docs deploy is ``continue-on-error``. That came from devops#3 and the same change in seamm_webui's own copy. A failure is therefore only a red step in a green job. A ``$GITHUB_STEP_SUMMARY`` line or a warning annotation would make it visible. *Why deferred.* A nit from the phase 7 re-check. *Evidence.* properties_step's Release hit the ``gh-pages`` push collision twice in phase 6. *Effort.* S. *Recommendation.* Do it with the next devops change. The phase 7 triage first proposed one concurrency group for every workflow that pushes ``gh-pages``, but the release notes rejected it: GitHub cancels all but one *pending* job in a group, so a Release's deploy could be dropped. A retry with a ``git pull --rebase`` of ``gh-pages`` before the push would remove the collision itself. *Measured (2026-10-05).* The docs deploy succeeded in the latest Release run of all 15 packages released in phases 6 and 7, properties_step's included. The only failures on record are properties_step's two in phase 6, before devops#3. A failed deploy costs nothing visible, since the package's Docs workflow publishes the same docs when the merge lands on main. *Decision (Paul, 2026-10-05).* No change of its own. Fold the ~10 lines (an ``id`` on the deploy step, and a step that runs on failure, writing a ``::warning::`` and a ``$GITHUB_STEP_SUMMARY`` line) into the next devops change, and the same into seamm_webui's own ``Release.yaml``. 6. Run-path tests of the code steps ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* An end-to-end test of each code step's run path: - writing the input; - finding the program through ``.ini``; - running the task; - reading and analyzing the output; - reporting the results. No code step had one. In phase 7, lint alone caught a leftover ``success`` variable in ``MOPAC.run()`` that would have crashed every MOPAC run. *Decision and plan (Paul, 2026-10-05).* - **Two tests per code.** A *fake* program replays a real run's recorded output, so the whole path runs in CI. The *real* program is found as SEAMM finds it (``$_EXE``, the PATH, the installation's ``.ini``), and its test is skipped where the code is not installed. - **A shared harness, ``seamm_exec.testing``.** ``run_spec`` builds a flowchart from a spec and runs it with ``run_flowchart``, with its own ``HOME`` and ``SEAMM_ROOT``, so the user's timing files and ini files are never touched, plus a tiny ``Water`` step. ``fake_program``, ``find_program`` and ``table_value`` complete it. Each code's test is then about 20 lines plus its recorded output. - **Done (2026-10-05):** - mopac_step: ``tests/test_run_path.py`` (fe3c248, on dev). It was checked to fail on the old ``success`` bug, and it moves to the harness once seamm_exec is released. - The harness, in seamm_exec 6a8914a. - The ORCA test (B3LYP/def2-SVP water, 1 core). It found two problems: - **A sub-step's tasks lost the SEAMM root** (fixed, seamm_exec afb3e6d). ``TaskSet`` and ``Evaluator`` read ``node.global_options``, which is empty on a sub-step, so ORCA's Energy, Optimization and BSSE tasks never read ``/orca.ini`` and took ``orca`` from the PATH. ORCA must be started by its full path. On ChemAI and MolSSI10, ``/usr/bin/orca`` is the Debian screen reader, and TinkerCliffs has no ``orca`` on the PATH. No ChemAI job since the 2026-10-04 rollout used the ORCA step, so production had not hit it yet. The orca.ini files: ChemAI and TinkerCliffs name the full ORCA 6.1.1 path and the OpenMPI library path; the Mac's is the same ORCA as its PATH; MolSSI10's ``[local]`` has no ``code``. - **ORCA 6.1.1 aborts water on 11 processes** (the grid error "the number of points read from the grid does not match the expectation"). orca_step gives even a 3-atom molecule every core it sees. A cap for small molecules is a separate decision. - **Next, each when its package is next released:** Gaussian (g09 on the Mac), then VASP. VASP is not on the Mac; it is on ChemAI, MolSSI10 and ARC, so its real-code test runs there. - **MDI codes (LAMMPS, xnn) separately.** They need a fake MDI engine, a design question of its own. - **Rarely used codes** (dftbplus, packmol, torchani, fhi_aims, atomic_charges, psi4, xtb): only when someone works on them. - **Release order:** seamm_exec (harness and root fix, pin ``seamm-util>=2026.9.27.1``) first, then orca_step and mopac_step pinned to it. 7. Legacy structure-handling wording (strain_step) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* seamm#221 (closed) translates legacy ``structure handling`` spellings such as "be put in a new configuration" when parameters are read, so old flowcharts run. But strain_step still *defines* its own parameter with the old wording: its default and enumeration in ``strain_step/__init__.py``. *Why deferred.* Outside the campaign's packages. *Evidence.* ``strain_step/__init__.py:24``: ``"default": "be put in a new configuration"``. *Effort.* S: switch to seamm's standard ``structure_handling_parameters`` and the current choices, plus a release. *Recommendation.* Do it, and grep the other steps for local copies of the old wording. *Done (Paul, 2026-10-05/06).* strain_step 2026.10.5 has SEAMM's standard choices (overwrite, new configuration -- still the default -- or new system; not discard), names a new configuration after the strains, and fixes the cell table when overwriting and a non-periodic system ending the flowchart. It also carries its item 3 fix (the cookiecutter's ``versioneer.py``; ``test_env.yaml`` removed). strain_step 2026.10.6 keeps the current names for a new system or configuration. The same choices went into supercell_step 2026.10.6 (overwrite stays the default), which also fixes supercells of bonded structures (bonds looked up by row position; each copy's bonds mapped from the previous copy's) and its item 3 ``MANIFEST.in``. Doing so found three bugs in molsystem's ``lower_symmetry`` (bonds by position, ``other=`` putting the coordinates on the source, Cartesian cells), fixed in molsystem 2026.10.6, which supercell_step requires. 8. seamm_exec test flakiness ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* Timing-sensitive tests that fail on loaded CI runners. *History.* - ``test_tasks_run_concurrently`` asserted a total elapsed time under 2.5 s. Phase 7 changed it to measure the overlap of the tasks' running intervals. - ``test_bundles_through_the_taskserver`` failed on Ubuntu with Python 3.12 in the 2026.10.5.1 PR. The backend has a bundle's results a moment before the runner marks the TaskServer job completed. It now polls for up to 30 s (02b59d2, test-only). *Evidence.* Two cases in three releases, each fixed in the test, not the code. *Effort.* S. *Recommendation.* Do a sweep: grep the tests for ``time.sleep`` and elapsed-time assertions, and give each one a poll with a deadline. *Done (2026-10-06).* A sweep of seamm_exec's and seamm_scheduler's tests for elapsed-time assertions and fixed sleeps. Most are safe: they wait for a condition with a deadline, clear a fault after a delay without asserting on time, or bound time in the direction a slow runner only makes safer. Two raced a slow runner and now wait for the state they need: seamm_exec's ``test_cancel_failure_leaves_tasks_adoptable`` (slept 1 s and assumed the task was submitted; 56b5a31) and seamm_scheduler's floor test (slept 1.5 s so that the second job would be the newest; 1127e5c). seamm_exec's last 40 CI runs showed no other flaky test: the outage, the PyPI cache lag and the TaskServer bundle race fixed in 02b59d2. Both go out with each package's next release. 9. PBS site: job history and ``max_resubmits`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* MolSSI10 is the PBS test site. ``qstat -x`` must keep finished jobs long enough for a laptop asleep over a long weekend; the phase 7 note asks for ``job_history_enable`` and at least 7 days of ``job_history_duration``. The ``max_resubmits`` guidance is the same as for SLURM. *Status.* - The jobserver user guide now covers ``max_resubmits`` for SLURM, PBS and the TaskServer, and how long PBS must keep job history (seamm_jobserver 2026.10.5). - The phase 7 notes record ``job_history_duration`` at ``168:00:00`` on MolSSI10. - The 22 stale staged directories were removed on 2026-10-05, and ``~/seamm_dev_remote_jobs`` was empty when checked again later that day. *Effort.* None, unless the check fails. *Recommendation.* Done. Re-verify ``qmgr -c 'p s'`` once at the next MolSSI10 visit. 10. Task view at 500 iterations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* ``/api/jobs/{id}/tasks`` walks the job's files and reads every iteration's ``job_data.json`` and manifests on each 15 s refresh. *Why deferred.* It was only exercised at 24 iterations. *Evidence.* No measurement at scale. *Effort.* S to measure, with a synthetic job of 500 iteration directories in the test fixture's layout. M if it needs caching, for example by the files' modification times. *Recommendation.* Measure, and act only if a refresh costs more than about 1 s. 11. TaskServer as the default local queue ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* Make ``type = queue, scheduler = seamm`` the default ``[local]`` section of a new installation, instead of ``type = local``. *Why deferred.* Paul's D3 made it opt-in, SEAMM_DEV first. *Evidence.* The soak on SEAMM_DEV ran jobs 4015-4019, two jobs sharing 4 cores and 4 GB, and the kill soak. The soak setting was removed on 2026-10-05, when SEAMM_DEV went back to ``type = local`` on the released packages. *Effort.* S in seamm_manager's install. The real cost is the soak time. *Recommendation.* Defer. Opt in on one everyday installation, the Mac's ``~/SEAMM`` or paul.local, for a few weeks of real use after the 6+7 rollout, then decide. 12. Separate placement, live ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* A parallel loop with ``placement = separate tasks``, where the iterations' codes are tasks on the target rather than run inline. *Why deferred.* Phase 6 validated inline placement and bundled iterations live, but not separate placement. The Loop's user guide marks it experimental. *Evidence.* Unit tests and local runs only. *Effort.* M, with live TC runs. *Recommendation.* Validate with the EC pilot, which is the first real loop that needs it, before removing the "experimental" label. 13. Timing helper adoption ~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* ``seamm_exec.timing`` (2026.10.5.1) gives a locked, rotating append for ``~/.seamm.d/timing/.csv``. No plug-in uses it yet; mopac_step, lammps_step, gaussian_step and vasp-step write their own files. *Why deferred.* Each code adopts it in its own release (vasp-step#18, still open). *Effort.* S per plug-in. *Recommendation.* Do it as each code step is next released, starting with vasp-step, whose cluster runs share a file over NFS. *Handed over (Paul, 2026-10-06).* The timing campaign (``campaigns/2026-10-05``, the design session) owns it: MOPAC, Gaussian and VASP move to ``record_task_timing`` in its Phase 4, LAMMPS and the rest in Phase 5. A survey for it: mopac_step and vasp_step run through the task layer; gaussian, lammps, psi4 and dftbplus still use ``executor.run``. On Paul's Mac ``lammps.csv`` is 707 MB, and mopac's, dftbplus's and gaussian's 47-63 MB. 14. molsystem#121 ~~~~~~~~~~~~~~~~~ *What.* Setting ``periodicity = 0`` on an empty periodic configuration raises. *Why deferred.* Filed during the phase 4/5 work; not on the campaign's path. *Effort.* S. *Recommendation.* Do it with the next molsystem release. *Done (2026-10-06).* molsystem 2026.10.6.1: the cell's ``to_fractionals`` and ``to_cartesians`` convert an empty list of coordinates to an empty one, so the periodicity setter (and any other caller) works with no atoms. read_structure_step's 2026.10.4 workaround is harmless and stays. 15. orca_step: cap the cores for small molecules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *What.* orca_step gives a calculation every core it sees: ``%pal nprocs`` is the available cores, capped only by the step's or the global ``ncores``. ORCA 6.1.1 then aborts small molecules in its start-up. *Evidence (2026-10-05).* Water at B3LYP/def2-SVP on the Mac with 11 processes aborted in ORCA's start-up: "the number of points read from the grid does not match the expectation" (``orca_startup_mpi``). On 1 core it runs in 3.7 s. Found while writing the run-path test (item 6), which runs on one core for that reason. *Effort.* S. Cap ``n_cores`` by the size of the calculation, for example by atoms or basis functions, using the same descriptors the timing records now hold. The cap belongs in ``orca_base`` where ``%pal`` is set. The threshold needs a few measured points: at what size does ORCA accept 2, 4 or 11 processes? *Recommendation.* Do it (Paul, 2026-10-05: its own item). Measure the threshold on the Mac first, then check it on a cluster node with more cores. *Result (2026-10-06): not needed.* ORCA 6.1.1 ran water, methane, ethanol, benzene, hexane and decane (3-32 atoms) on 2, 4, 6, 8 and 11 processes without a single failure, and so did the failing run's exact input and the same flowchart through SEAMM. The abort reproduces only with Homebrew's OpenMPI 5.0.8 first on the PATH: ORCA 6.1.1 needs OpenMPI 4.1.x on macOS, which orca_step puts on the PATH from ``library-path`` in ``orca.ini``. The failing run predates the sub-step root fix (seamm-exec 2026.10.5.2), so ``orca.ini`` was not read and ORCA used Homebrew's ``mpirun``. Follow-up done (Paul, 2026-10-06): orca_step 2026.10.6 checks, before a parallel run, the ``mpirun`` ORCA will use and stops with a clear message if it is OpenMPI 5 or missing (skipped on one core and for ``installation = modules``). Not phase 8, recorded here for completeness ------------------------------------------- - **The phase 6+7 rollout:** Paul decides when, timed with the EC pilot. ChemAI only on an explicit ask; TinkerCliffs with an explicit package list and a dry run, never ``--all``. - **Cleanup (2026-10-05), all done:** - The idle test venvs ``~/SEAMM_DEV/venvs/phase4-A``, ``phase4-B`` and ``phase7-B``, and the empty test queue ``~/SEAMM_DEV/taskserver`` with its test-sized ``taskserver.ini``: deleted by Paul (2026-10-05). - The local harness runs in ``Testing/phase4/runs`` and ``Testing/phase2`` (the ``Job_9000xx`` runs and logs) were deleted on Paul's word (2026-10-05); the harnesses and ``targets.ini`` stay.