2026-10-16 Labelling and training MLFFs entirely in SEAMM#

Goal#

Run the whole cycle that makes a machine-learned force field – generating structures, choosing which to label, labelling them, assembling the training sets, training and validating the models – as SEAMM flowcharts submitted to a JobServer, without the hand-made batch scripts, watchers and one-off corrections the campaign sessions (mbe, mlff, science) use today.

This document is the checklist: for each stage, what SEAMM already does, what is still done outside it, and what is needed. It is a living document; the science plans stay in the ~/Sites lab notebook (the carbonate plan is mlff-training/2026-10-08_carbonate-electrolyte-training-plan, the water plan mlff-training/2026-07-15_water-mlff-training-plan), and each step’s own campaign notes hold its design.

Status of this list (2026-10-16): the labelling requirements (section 2) are mbe’s, sent after carbonate wave 2; structures, selection, collection and validation (section 3) are science’s, in its order of priority; training and models, and mlff’s parts of collection and validation, are mlff’s. Section 4 puts the sessions’ priorities side by side.

1. The stages#

Stage	In SEAMM now	Outside SEAMM now	Owner
Structures	Packmol, LAMMPS (classical and MLFF MD via xnn), Normal Mode Sampling, Dimer Builder, Extract Clusters; the OPLS reference cells and Li-salt cell builds are SEAMM jobs	MD frame pools as plain LAMMPS Slurm jobs; seeded starts, cluster carving and n-mer sampling scripts; reference-cell density analysis	science
Selection	active_learning_step (QBC committee scoring, 2026.10.14.1) and `seamm-al`	`seamm-al score/select` run as a CLI from a frozen copy in the MLFF env on ChemAI; group quotas, budget and frame decorrelation by scripts; no contact check	science, worker
Labelling	mbe_step + seamm_mbe (VASP monomers/cell, ORCA pairs/triples), the task layer and bundles, the cost model and `seamm-timing`	Evaluator jobs submitted by hand on ARC; watchers on the Mac; restart and manifest repair scripts (section 2)	mbe
Collection	`mbe_labels.extxyz` per cell	Every correction as a script editing extxyz text: Pulay stress offset, minus-D4 twins, ΔfE and `total_charge` headers, the CCSD(T) Δ-model patch, merging, the force filter; energy caps undecided	mbe, science
Training	The Active Learning step’s Train Committee sub-step (wraps `seamm-al train` as a job task in the MLFF environment; can save the committee as an xnn model that Model Chemistry offers), Check Committee, Score Pool	Every production committee so far: Sina’s `train_cmte013.py` with a Python data module per experiment, Slurm scripts on ChemAI’s A100s, a hand-built conda env; scoring by script. The step has run only in smoke tests	mlff
Validation	LAMMPS MD with a committee as the model chemistry (works, but 120 ms/step against 76 direct); the Density, Diffusivity, Viscosity and Thermal Conductivity steps; `seamm_md_analysis`	Production MD as hand-launched MDI pairs with hand-written LAMMPS inputs; liquid analysis scripts: ρ, g(r) and ΔH_vap; dielectric and dipoles from the LES latent charges; CIP/SSIP; same-resolution g_OO against Skinner; molecular vs atomic pressure	mlff, science
Models	xnn_step model chemistries; Train Committee’s “save as xnn model”	Checkpoints in experiment folders, provenance from folder names, Slurm logs and journals; the production MLFF env switched by hand in `lammps.ini`/`xnn.ini`; each xnn release validated by hand; distribution waits on Sina	mlff

2. Labelling through the JobServer (mbe, 2026-10-16)#

The path already works: ChemAI’s [tinkercliffs_mbe] queue passed a smoke test (ChemAI job 5172, 2026-10-06), and the JobServer resubmits a lost evaluator in place. Throughput is the same either way (about 550 starts per hour per cluster; MaxSubmitPU 1000 on tc_normal_base). Most important first; 2.1-2.3 are the minimum to run as well as now, plus 2.4 if Owl is used.

2.1 Changing the settings of running jobs. target.json is copied from the

queue’s section when the job is submitted (seamm_scheduler config.py). In wave 2, estimate_scale, min_task_walltime and constraint were changed five times, each followed by a restart of every evaluator; through ChemAI that would need ChemAI.ini edited (Paul only), the JobServer restarted, and every running job’s target.json edited by hand. Needed: an action (REST/WebUI) that pushes the queue section’s current settings to running jobs, and evaluators that re-read the scheduling settings (estimate scale, minimum task time, constraint, nice) from target.json as they go. The connection settings (root, transport) stay fixed for a job’s life.

Also needed: who sets the bundle size. mbe_step passes its own tasks per bundle (“molecular bundle”, 4) and bundle walltime (“bundle walltime”, 8 h) from the flowchart (mbe_step/levels.py), and these override the target’s: a target bundle_walltime had no effect on 2026-10-16. Decide which wins, target or step, and make tasks per bundle adjustable on a running job. It is the real throughput lever: with starts capped per cluster, running bundles ≈ starts per hour × bundle hours, so longer bundles are the only way to keep more running.

2.2 A deliberate “restart job”. Only _finalize_or_resubmit_slurm_job

exists, and it resubmits only after the evaluator is lost. Needed: cancel the evaluator and resubmit it in place, adopting the queued and running bundles. Prerequisite, done: the manifests lost archive references when an evaluator was stopped just after a bundle was archived, so restarts reran finished tasks (up to 646 missing references in one cell after the timing-lock stall, 0-6 per cell on ordinary cancels; 55 rerun on TinkerCliffs at 04:2x after an incomplete repair); mbe ran carb_production/reconcile_manifests.py --apply before every resubmit. Fixed in seamm-exec 2026.10.16.3 (the archive is recorded before the directories are removed; damaged manifests heal on restore), on ARC since 2026-10-16 13:34; the script retires once no evaluator runs the old version.

2.3 Scheduler housekeeping now done by watchers on the Mac.

Starvation. A new evaluator queues FIFO behind its own job’s pending bundles at the same priority. mbe nices the bundles to 10 during restarts and back to 0 afterwards; a niced backlog left in place is starved by newer bundles (it happened on 2026-10-16). Needed: a target option such as bundle_nice (adjustable on running jobs, 2.1), or a higher priority for evaluators.

TinkerCliffs “env retrieval failed” holds are released automatically by the watcher. Needed: the JobServer’s poller releases them.

2.4 Owl. ChemAI has no Owl section (configuration). Owl matters for two

reasons: the start ceiling is per cluster, so two clusters double the throughput; and Owl is faster for the expensive tasks (revDSD TZ pairs: median 763 s on Owl against 1,348 s on TinkerCliffs, 1.77x, 2026-10-16). Needed later: a “move job to another queue” action, so a cell can be restarted on another cluster (/projects is shared, so nothing is restaged).

A queue’s constraint matters to the time estimates as well as to where bundles run: estimates are keyed by cluster:partition:cpu_model, so TinkerCliffs’ MBE queue pins constraint = amd until the estimates follow the bundle’s node type (seamm_exec campaign 2026-10-05, phase 5 as revised 2026-10-16).

2.5 Queue settings. [tinkercliffs_mbe] uses tc_normal_short (2x cost)

with a 1-day limit; wave-2 evaluators run longer than a day. Production needs _base with 7 days: an ini change, to be the documented default for campaign queues.

2.6 Progress view. Results stage back to ChemAI only at the end of a job

(the dashboard’s on-demand pull exists). mbe watches per-level task counts (done, pending), OOM/TIMEOUT/NODE_FAIL bundles and seamm-timing check. Needed: a per-job task progress view in the WebUI, including estimate ÷ wall per group (seamm-timing check for the job), the first thing to look at after every update.

2.7 Checks before a pilot.

Which ARC user ChemAI’s ssh runs as. Answered: psaxe (the JobServer runs as seamm only on ChemAI), so jobs from ChemAI share ~/.seamm.d/timing and the timing model with the direct runs. The queue’s remote_root is /projects/seamm/seamm_jobserver_remote_chemai (/home is slow from compute nodes).

Each cell’s job-directory size when it stages back to ChemAI. Open.

Proposed pilot (mbe): one or two cells of the next wave through ChemAI, the rest as now. It needs ChemAI.ini edits (an Owl section, _base), which are Paul’s call.

Also open for labelling: a k-point spacing option in mbe_step for cells smaller than 10 Å (vasp_step takes options["k_spacing"]; the 2026-10-06 smoke cell failed on it; deferred by Paul 2026-10-09).

3. The other stages#

Science’s order of priority: collection, selection, structures, validation. Paths are on ChemAI unless marked.

3.1 Collection (science, with mbe and mlff) – priority 1.

Outside SEAMM today, all as scripts that edit the extxyz text (not through ASE, which drops the DDEC6 charge column and the info keys xnn needs):

the Pulay stress offset: fitted offsets (water: quadratic in mass density; NaCl uses water’s; EC a constant +2030 atm; FEC 2740 + 1680(ρ_rel − 1)), applied by mlff’s make_stressfix.py / make_stressfix_full.py into *_pulayfix copies with a stress_pulay_shift_atm header (/D3/sina/xnns/research/experiments/arc/2026-10-13_1400_pulay_stressfix_labels_psaxe/; notebook mlff-training/2026-10-13_vasp-pulay-stress);

minus-D4 twins: ion_d4/d4_twins.py (NaCl), dmc_twins.py (DMC), subtracting xnn’s D4, checked against ORCA’s D4 (≤ 1e-5 meV/atom);

the ΔfE convention and the total_charge info key for ion sets: header edits, by hand or per-set converters (qz_tools/convert_qz.py for the EC/FEC/DMC QZ sets);

the CCSD(T) Δ-model patch (2026-10-16): /D3/psaxe/ccsdt_dm2/patch_dm2.py adds ΔE/ΔF/Δσ from dmodel2.py and its coefficients, and verify_patch.py checks everything else is byte-identical; 13 water sets in patched/ with a README and SHA256SUMS (notebook mlff-training/2026-10-16_water-ccsdt-delta-model);

merging the cells’ labels into sets passes through two hands for EC/FEC. mbe concatenates each cell’s 3/mbe_labels.extxyz into /projects/seamm/psaxe/{ec,fec}_production/wave<N>_labels.extxyz, adding frame=<cell name> to each frame (wave 1: a one-off command, no saved script). mlff then makes the route-B D4 twins (/projects/seamm/psaxe/ecfec_mlff/tools/subtract_d4_cells.py and d4_twins.slurm, importing science’s periodic/tools/route_b_d4.py), the stress-shifted variants (wave 1: opt-in ECFEC_CELLS_VARIANT=_stressfix, mbe’s first finite-difference estimate, +2100/+2700 atm), and the splits together with the molecular QZ no-CP sets in ecfec_data.py. Those sets are copies made on 2026-10-12 in the experiment folder (/D3/sina/xnns/research/experiments/arc/2026-10-12_0930_mace_ecfec_wave1_psaxe/Molecular/), byte-identical to Dropbox GM/TrainingData/Molecular/{EC,FEC} but not linked to it, so an update there does not reach a fit unless copied again; they come from the TinkerCliffs QZ relabel (/projects/seamm/psaxe/qz_relabel, README_QZ.md) through qz_tools/convert_qz.py and route-B D4 twins. For wave 2 mbe applies the Pulay offset at collection (science’s fits: EC +2030 atm; FEC +2656/+2740/+2824 atm at ρ_rel 0.95/1.00/1.05), marked in each frame’s info (pulay_offset_atm) and in the file name (wave2_labels_pulayfix.extxyz). Risk: the offset applied twice if mlff’s stress-shift path ran on those files too. Guarded for the wave-2 refit (mlff, 2026-10-16): the files are read as delivered, the data module refuses a frame carrying pulay_offset_atm when a shift is asked for and checks every wave-2 frame carries the expected offset, and the wave-1 cells get the same fitted offsets rather than the first estimates (mlff to settle with science which way). The case for recording every correction in the data, which the training-set step would make structural;

the force filter (|F| > 20 eV/Å, applied to the control forces in both arms of an A/B test): in mlff’s data modules;

energy caps for high-energy tails (DMC normal-mode monomers: p95 5.3 eV, maximum 12 eV above the minimum): Paul’s decision, not implemented.

Needed: a training-set step that gathers the labels across many cell jobs (three hand-made pieces owned by two sessions today) and applies an ordered list of recorded corrections, each with its parameters and a version – the Pulay offset per code, cutoff and element set; the ΔfE reference atoms; D4 removal; a Δ-model as a plug-in callable; an energy cap or weight; the force filter – writes extxyz losslessly (every per-atom column and info key) and records the provenance chain (source sets, corrections, hashes) in the set’s header or a sidecar.

Blockers: a lossless extxyz writer in SEAMM (ASE’s is not); a home for the correction definitions (offset fits, Δ-model coefficients) as versioned data; the A/B pattern (patched and control sets with identical frames and filters) as a first-class option.

mlff’s part: each experiment’s Python data module (h2o_data.py, nacl_data.py) combines the label files per source (folders of symlinks to the label variants: the *_pulayfix copies, the CCSD(T)-Δ patched sets, the *_minusD4 labels), applies the force cap (FMAX_CUT, filtered on the control labels in both arms), checks the monomer energy reference, makes the per-source splits, and picks the label variant through an environment variable (XNN_ARM=ctl|ccsdt). D4 is subtracted and recorded as the checkpoint’s dispersion record. Needed: the training-set step writes the set and a manifest that Train Committee reads – files, the D4 record, the Pulay offset, any Δ correction (named, with its coefficients file), the force cap; seamm-al train’s manifest.json/frames.csv already hold the md5s, so it is mostly an extension of that.

3.2 Selection (science, with worker) – priority 2.

Outside SEAMM today: seamm-al score/select as a CLI in the MLFF environment, from a frozen copy (/D3/psaxe/active_learning, frozen cc078f3, then 5d55e69); the wave-2 selection by select_w2.sh (stage 1: the top cell per liquid, density and temperature group; stage 2: the rest by rank within a core-hour budget) and greedy_w2.py (no adjacent frames of a run: they share about 70 % of their neighbour shells), result TinkerCliffs carb_production/wave2_final_AL.txt; ranking by ρ_P + ρ_F from a final-committee check (ChemAI 12035).

Needed: seamm-al select as a step; group quotas, a core-hour budget (from the seamm-timing estimates) and a frame-decorrelation rule as its options; a contact check beside the committee spreads – MD frames with unlabelled tight O···O or ion contacts score like OPLS frames (2026-10-13).

Blocker for wave 3: the contact check (worker).

3.3 Structures (science) – priority 3.

The OPLS reference cells and the Li-salt cell builds are SEAMM jobs (EC/FEC references ChemAI 5122-5131; Li-salt builds 5258-5262; NaCl step-4 frames 5177-5188).

Outside SEAMM today: MD frame pools as plain LAMMPS Slurm jobs (frames_li.slurm in /home/psaxe/li_frames, 11965-11970: six conditions × ρ {0.95, 1, 1.05} × T {298, 358} K → 336 frames; the scaled-charge runs, make_scaled.py, 11954/11955); seeded starts (make_liec4.py, Li(EC)₄⁺ seeds for Packmol); carving (carve_li_cluster.py, Li⁺-centred neutral clusters within 4.5 Å; sample_dm.py in /D3/psaxe/ccsdt_dm, connected n-mers from MD dumps; gen_ion_frame.py, clusters in boxes for the cell pipeline – Extract Clusters covers part of this); reference-cell analysis (the density from ⟨P⟩ through −P/K_T, block-averaged from nvt_state.trj, never the stored scalars).

Needed: an MD frame-pool step on top of the LAMMPS step (a grid of conditions, frames per condition, decorrelation spacing); Extract Clusters options for ion-centred, charge-neutral carving and for sampling connected n-mers with a stated distribution, recording the sampler’s distribution against the liquid’s (2026-10-16: the motif-balanced sampler put 33 % of triples in closed triangles, against 5 % in the liquid).

Packmol losing the P-F bonds of a second configuration source (ChemAI 5260, 2026-10-10) was fixed in packmol-step 2026.10.10 and molsystem 2026.10.10.

3.4 Validation (science, with mlff) – priority 4.

Outside SEAMM today: analyze_liquid.py (ρ, g(r), ΔH_vap through model.json’s e_monomer_min; generalized for nper/msite); mlff_dipoles.py, eps_both.py, ct_check.py (the dielectric constant from the LES latent dipoles, the 4×4 channel matrix, a charge-transfer check; Falcon /projects/seamm/psaxe/liqval_mlff and ChemAI); analyze_nacl_md.py (CIP/SSIP with a fixed cutoff, coordination numbers); the same-resolution g_OO against Skinner (~/Sites/mlff-training/2026-09-16_trimer-mlff-liquid-density/skinner_same_resolution.py, Q_max 21.8, variable Lorch; mlff’s skinner_cmp.py is a copy); mlff’s frozen-frame committee comparison of molecular and atomic pressure (pmol.py, /D3/psaxe/mlff_scores/pmol: whole molecules about their centres of mass, and the virial-theorem check ⟨P_mol − P_atom⟩ = 2 N_mol kT / V, N_mol the number of molecules).

Needed, as options of seamm_md_analysis or the Density step: g(r) at the resolution of a stated experimental reference; block-averaged density with block-size errors; molecular against atomic pressure; evaluating committee A on trajectory B.

Blocker: the LES dipole/BEC question (2026-10-14), which decides whether a dielectric constant or IR spectrum from the latent charges is usable at all.

mlff’s part, outside SEAMM today: the MD itself – the committee-mean MDI pair launched by hand from Slurm (ccsdt_md.slurm, cmte_md.slurm: mpirun -np 1 mdi_bind.sh seamm-al engine --members … --spreads : -np 1 mdi_bind.sh lmp -mdi … -in input.dat) with the LAMMPS input (relax, NVT, NPT equilibration, production) and water512.data written by hand; and the scripts /D3/psaxe/mlff_rho_prod.py (block-averaged ρ with its standard error), goo.py and skinner_cmp.py (O-O peaks and n_OO; same resolution as Skinner 2013), analyze_nacl_md.py + cip_fixed.py, and pmol.py, pmol2.py, geom.py in /D3/psaxe/mlff_scores/pmol* (atomic against molecular virial on fixed frames, committees cross-evaluated, O-H/HOH/H-bond counts).

mlff’s needs:

Close the GPU-path gap through SEAMM (120 against 76 ms/step; ChemAI 12250 against 12253): OMP_NUM_THREADS=1 is the suspect (lammps.py near line 1928, mdi_bind.sh line 309), a 15-minute test. Then validation MD can go through the LAMMPS step.

Batch the members in the engine: seamm-al engine uses about 40 % of the GPU (py-spy, 2026-10-16), launching the five members one by one; evaluating them in one batch helps both paths (two MDs per GPU is the cheap stopgap).

The analyses above as options of seamm_md_analysis or the MD steps: block-averaged density with its error, g(r) peaks and n_OO with the same-resolution comparison, ion coordination and CIP with fixed cutoffs; later molecular pressure and monomer geometry.

Cross-evaluation – committee A on committee B’s trajectory, P_atom and P_mol per frame – as a Score Pool option.

3.5 Training (mlff).

Outside SEAMM today: every production committee (g7_cmte, nacl_cmte, nacl_les1_cmte, g7_ccsdt/g7_ctl) was trained with Sina’s research script train_cmte013.py and his samplers (homogeneous.py), copied into each experiment folder /D3/sina/xnns/research/experiments/arc/<date>_<name>_psaxe/ (latest 2026-10-15_2130_ccsdt_committees_psaxe), with the training set as the experiment’s Python data module (3.1). Slurm: chemai_cmte.slurm.g7_template, chemai_ccsdt_cmte.slurm (one A100, 12 CPUs, 64 GB, five members in sequence, --nfold 5 --fold-seed 0 --seed 1000+M, label md5s logged) and dryrun.sh (a CPU --dry-run comparing manifests across arms). The environment is the hand-built conda clone seamm-lammps-xnn0141, without SEAMM. Scoring: /D3/psaxe/mlff_score_cmte.py and its Slurm scripts in /D3/psaxe/mlff_scores/ (CPU, float64; per-group E/F/S MAE on the data module’s validation and test splits). The Train Committee step and seamm-al train have run only in smoke tests (/D3/psaxe/active_learning/smoke_p2, smoke_013).

Needed:

A gate: Train Committee (or seamm-al train) with the same recipe, data and folds reproduces train_cmte013: identical folds and batch order (checkable in the manifests and the dry run), and test scores within the seed-to-seed spread of train_cmte013 itself (GPU training is not deterministic, so not bitwise). The samplers are ported (sampler_equivalence.txt) but the end-to-end match was never run; whether Sina’s per-source splits and --stratify give the same folds needs checking.

The training set comes from collection (3.1) as files and a manifest, not a Python module; the force cap, per-source weights and label variants become recorded options there.

Members as parallel GPU tasks through the task layer (ChemAI’s two A100s, ARC’s A100s); the step now trains them in one task, and “members to train” only splits them by hand.

The test-split score table (what mlff_score_cmte.py prints) as a Check Committee option.

Blockers: none from Sina (the xnn Trainer’s sampler hook is in xnn 0.12 and later).

3.6 Models (mlff).

Outside SEAMM today: the checkpoints stay in the experiment folders (runs/<name>/m0-4/best.pt and fold_manifest.json); the provenance is the folder name, the Slurm logs (md5s) and the journals. The production environment is switched by hand: lammps.ini and xnn.ini on ChemAI and ARC point at a conda clone (now seamm-lammps-xnn0141; .bak files; xnn0130 is the rollback), built by hand (clone and pip) and kept out of seamm-manager’s way. Each xnn release is validated by hand (float64 bitwise against the previous version, a GPU PME check, an MDI-pair check).

Needed:

Train Committee’s “save as xnn model” used for real committees, with the model record carrying the recipe, the training-set manifest (md5s), the folds, the xnn and torch versions and the dispersion record (what it stores today is unchecked).

MLFF environments managed by seamm-manager (the driver-aware torch installer, seamm_manager#31, exists) instead of hand clones, with a versioned switch and rollback; LAMMPS-MDI, cuequivariance and xnn in one environment (memory: mlff-pair-needs-combined-env).

Distribution (.mlff.json sidecar and catalog): waits on Sina.

3.7 Labelling, from science. Every label pipeline records VeryTightSCF and: the NoCOSX element list as checked parameters: TightSCF’s error on isolated monomers (−0.05 kJ/mol each, not additive) cost a day on 2026-10-16.
3.8 Infrastructure shared by all stages. The time estimates and their check: (seamm-exec 2026.10.16-2026.10.16.2, seamm-timing check); estimating for the target’s node type (seamm_exec campaign 2026-10-05, phase 5 as revised 2026-10-16); the GPU MLFF path through SEAMM (3.4 a); xnns 0.14.1 on ARC at 475 ms/step against 0.13.0’s 424 (graph build 35 against 18 ms), not investigated; job-start validation that catches undefined flowchart variables.

4. Priorities#

Each session’s order, most important first.

Session	Priorities
mbe	Labelling through the JobServer: live settings for running jobs and who sets the bundle size (2.1), a restart action (2.2), bundle priority and hold release (2.3); then Owl (2.4); proposed pilot of one or two cells
science	Collection: the training-set step (3.1); selection as a step with the contact check, which blocks wave 3 (3.2); structures: the frame-pool step and carving options (3.3); validation analyses (3.4)
mlff	The training gate (3.5 a), then the EC/FEC refit after wave 2 through the step; the GPU-path speed test (3.4 a); the collection manifest with science (3.1); members as parallel GPU tasks and the score table (3.5 c, d); the analyses as steps and batched members in the engine (3.4 b, c); model provenance, managed environments, distribution (3.6)

Common threads: the training-set step and its manifest (science’s first, mlff’s third) is what lets both selection and training leave their scripts; the training gate decides whether the existing Train Committee step can take the next production committee; and the GPU-path test is the cheapest item on the list.

5. Log#

2026-10-16: document started (Paul). Labelling list from mbe; manifest archive-reference bug fixed (seamm-exec 2026.10.16.3); ORCA estimates use the model again (orca-step 2026.10.16, .1) and record their source (2026.10.16.2). mbe’s corrections to section 2; science’s stages (section 3.1-3.4, 3.7); mlff’s stages (3.1, 3.4-3.6) and the priorities (section 4); mlff’s corrections.