Getting Started#

seamm_exec runs flowcharts (run_flowchart, run_from_jobserver) and runs the external codes that steps need. A step gets the executor from its flowchart and asks it to run a program; the executor handles the program’s configuration from <root>/<program>.ini (a conda environment, environment modules, or a plain executable), where the program runs, and which files come back.

Running a code: executor.run()#

The original interface, used by most steps:

result = self.flowchart.executor.run(
    config,                      # the program's section of <root>/<program>.ini
    cmd=["{code}", "input.dat", ">", "output.txt"],
    directory=self.directory,
    files={"input.dat": text},   # written before the run
    return_files=["output.txt", "*.log"],
    shell=True,
)
output = result["output.txt"]["path"].read_text()

The returned files are left in the step’s directory ("@subdir+name" in subdir/name) and are not read: result[name] gives their "path" (and an "exception" if one could not be put in place), so a step reads only what it needs – an MD run’s trajectories can be gigabytes. result["stdout"] and result["stderr"] are text.

in_situ chooses where the code runs: None (the default) runs in a scratch directory under a scheduler such as SLURM ($TMPDIR, so node-local storage) and in place otherwise; True always runs in place, so the output can be watched as it is written; False always uses scratch. Only the return_files come back to the step directory. While a code runs in scratch, its return_files (each up to 50 MB) are copied to the step directory every 10 minutes as <name>.partial, and once more if the evaluator is stopped (SIGTERM, e.g. at a job’s time limit), so a calculation stopped part way still leaves its output. A finished one removes the copies. The environment variable SEAMM_PARTIAL_INTERVAL sets the seconds between the copies; 0 leaves only the copy when stopped.

Running codes as tasks#

Since 2026.10.2 a step can describe each calculation as a Task and run any number of them through a TaskSet:

from seamm_exec import Resources, Task, TaskSet

tasks = TaskSet(self, archive=True)
for key, text in inputs.items():
    tasks.add(
        Task(
            key=key,                       # stable across reruns, e.g. a fragment key
            program="orca",
            cmd=["{code}", "orca.inp", ">", "orca.out"],
            files={"orca.inp": text},
            return_files=["orca.out", "orca.engrad"],
            resources=Resources(ntasks=4),   # mem_per_cpu in bytes if needed
            success_text={"orca.out": "ORCA TERMINATED NORMALLY"},
            estimated_seconds=30,
            shell=True,
        )
    )
for result in tasks.run():
    if result.ok:
        parse(result.read("orca.out"))    # read now, from disk or the archive
    else:
        print(f"{result.key} failed: {result.reason}")

What this gives a step:

  • Concurrency. The tasks run through a pool sized to the machine or to the SLURM allocation, each in its own process group; a task’s Resources say how many cores and how much memory it needs, and ntasks=None means all of it. tasks.capacity() reports the pool’s size, for a step that sizes its input (ORCA’s %pal) before writing it.

  • Restart. The set keeps tasks/manifest.json in the step directory and a DONE marker in each task’s directory, tasks/<key>/. Rerunning the job in the same directory restores finished tasks from their markers and never recomputes them; a task whose inputs changed is recomputed. Each DONE records what ran the program – the modules loaded, the conda environment, or the command (TaskResult.code) – but a change of code does not cause a recomputation: for a mature code a new version rarely changes the results. When the results of a set come from more than one code, e.g. some reused from a run with VASP/6.6.0 and the rest run with 6.6.1, a warning says so, with the counts (TaskSet.codes). A task that failed is not retried within a run, but is tried again on a rerun, up to three attempts in all (the count resets when the inputs change). A task that stopped only because the evaluator stopped (killed, out of walltime), or because the step cancelled it, does not count as an attempt. Nor does a task running out of its own time, the first four times (max_timeouts): its estimate was too low, and each retry gets twice the time. Later timeouts count, so a task that cannot finish in the longest time the queue allows still stops. A process that a crashed run left behind is killed on the rerun, after checking it is the same process.

  • Success beyond the exit code. success_text names text that must appear in a result file; ORCA exits 0 on an error termination, for example.

  • Archiving. With archive=True finished task directories are packed into one tar per bundle, so a step with thousands of tasks leaves a handful of files.

A step with a single calculation uses the same machinery through run_task(), which also keeps the output in the step directory, as executor.run() did:

result = run_task(task, self, directory=self.directory)

A step that handles <program>.ini itself passes the program’s config (and env) on the task, and the pool uses them as they are. A task without config is configured where it runs, from that machine’s <root>/<program>.ini and the program’s resolver (an entry point in org.molssi.seamm.exec.resolvers; ORCA’s adds its full path and the OpenMPI paths). In the command, {code} is the program and {code_dir} its directory, when it has one.

Many structures, one model chemistry#

A step that needs the energy (and gradients, and stress) of many structures with the flowchart’s model chemistry uses an Evaluator rather than tasks or MDI directly:

from seamm_exec import Evaluator

with Evaluator(self, properties=("energy", "gradients")) as evaluator:
    for configuration in configurations:
        evaluator.submit(configuration, key=f"c{configuration.id}")
    for result in evaluator.results():
        if result.ok:
            store(result.key, result.energy, result.gradients)  # kJ/mol, kJ/mol/Å
        else:
            print(f"{result.key} failed: {result.reason}")

The evaluator chooses the path, not the step or the user. On a queue target a program that can make tasks runs as tasks there; otherwise a warm MDI engine evaluates the structures one after another, unless the program prefers tasks (ORCA, whose engine starts ORCA for each structure anyway). Both paths give the same numbers. A structure the program cannot run as a task (a periodic system for ORCA or MOPAC) goes to its MDI engine; if there is none here it fails alone, with the reason. The batch path keeps the task layer’s restart, so a rerun reuses finished structures. options on submit carries what a fragment needs: atom_indices, ghost_atoms, charge and multiplicity.

resources on the Evaluator (a Resources: ranks, memory per rank) sets the size of each calculation on the batch path; the provider’s get_task receives it. Without it the provider chooses.

A program offers the batch path through three classmethods beside get_model_chemistry_options: get_task, analyze_task and, optionally, can_run_task; see seamm_exec.evaluator. An analyze_task that takes task= is given the task that produced the result, so that it can append the run’s timing record (see below); one that does not is called without it.

The sign of the stress. The stress comes back as the program gives it, in GPa, and programs differ in its sign. A provider that returns a stress must therefore declare its convention in its get_model_chemistry_options entry: options["stress_convention"] is "pressure" (positive when the system pushes outward: VASP’s in kB line, MDI’s <STRESS) or "stress" (sigma = -P, as ASE and xnn use). The Evaluator passes the stress through unchanged; a consumer converts it with that declaration and refuses a level that lacks it. The declaration covers both paths, batch and MDI, so a provider whose two paths would give opposite signs must make them agree. Pinning one convention for every provider later would be a documented change of this contract.

Where things run#

Tasks go to the job’s target, a section of the JobServer’s <root>/<jobserver-name>.ini (see seamm_scheduler.config). The JobServer writes the job’s section into the job directory as target.json; for a run by hand, SEAMM_TARGET=<section> (with SEAMM_TARGETS=<ini file> if it is not <root>/<hostname>.ini) does the same. Without a target, or with tasks = pool, tasks run on this machine or inside the current allocation, as before.

With tasks = queue the TaskSet submits them, in bundles, as batch jobs:

[arc]
type = local
tasks = queue
scheduler = slurm
transport = ssh
host = tinkercliffs
remote_root = /projects/seamm/psaxe/tasks       ; task directories, staged
remote_python = /projects/seamm/SEAMM/venv/bin/python
account = seamm
partition = normal_q
qos = tc_normal_short
export = NONE
bundle_tasks = 8                 ; or bundle_walltime = 04:00:00
max_queued_tasks = 800

  • Each bundle is one job that runs python -m seamm_exec.task_worker bundle.json on the cluster, which runs the bundle’s tasks through a pool sized to the allocation. The program is configured there, from that machine’s <root>/<program>.ini and the program’s resolver (an entry point in org.molssi.seamm.exec.resolvers), so the code need not be installed where the evaluator runs.

  • Without a shared filesystem the task directories are copied to remote_root with rsync before the job and back after it; with shared_filesystem = yes (or the local transport) nothing is copied.

  • A rerun of the step polls bundles still in the queue rather than submitting them again, and a bundle that ran out of time leaves its finished tasks done.

  • A task whose bundle the queue stopped for running out of time is submitted again with twice the estimated time, and twice that after a second timeout, within bundle_walltime. A walltime the task gives itself is used as given the first time, and doubled in the same way after running out of time. The first four such timeouts do not use up any of the task’s attempts.

  • A bundle asks for twice its tasks’ estimated time plus ten minutes. Where the estimates are too low on a machine, the target section can guard against them: estimate_scale multiplies each estimate, and min_task_walltime (seconds, or a SLURM time such as 30 minutes) gives each task at least that much. A bundle is full when the time it would ask for – twice the estimates and ten minutes, or at least the tasks’ least walltimes – would be more than bundle_walltime, so even a full bundle keeps that margin. With bundle_walltime = 08:00:00, for example, tasks estimated at 1.5 h go two to a bundle, which asks for 6 h 10 min; three would need 9 h 10 min. A single task too long for the limit with its margin still runs, with a warning.

  • Tasks estimated to take less than inline_below seconds (default 60) run on the evaluator’s machine when their program is installed there.

  • A task that carries its own config was configured for the evaluator’s machine, so on an ssh target it runs there instead, with a warning. Steps that still configure their program themselves do this; ORCA (from orca_step 2026.10.3.1) and the Evaluator’s tasks name only their program.

Bundle files are in <step>/tasks/_bundles/<bundle>.<n>/: bundle.json, run.sh and the scheduler’s log.

Resuming a job#

Since 2026.10.4.1 a flowchart run in its own directory keeps a checkpoint, and a job that stopped part way – killed, out of walltime on a cluster, the machine rebooted – can continue from where it got to instead of starting again:

run_flowchart flowchart.flow --resume      # in the job's directory

--resume continues at the first step that had not finished, or in a Loop at the iteration (and step) it had reached. The JobServer asks for it itself when it resubmits a job a cluster lost (it sets SEAMM_RESUME=1 in the batch script).

  • What is kept. Each step’s database writes (structures, properties, tables) are one transaction, committed together with the checkpoint when the step finishes. A step that was stopped part way has none of its writes kept, and runs again; its finished calculations are reused from its task manifests. The checkpoint also holds the flowchart’s variables, the current system and each Loop’s position. It is stored in the job database, seamm.db; checkpoint.json beside it is a copy for people to read.

  • When it resumes. Only with --resume (or SEAMM_RESUME), and only if the flowchart and its command line are the same as before. A rerun without it moves the previous run’s seamm.db to previous/<date and time>/ and starts from the top, saying so in job.out; if that run could have been resumed it says that too. (A changed input file is not detected, which is why resuming is never automatic.) Changed package versions are noted, not refused.

  • Variables. Numbers, strings, lists, dictionaries, numpy arrays, quantities, paths, tables and the like are saved; a step that makes other objects saves them itself (the Forcefield step does). Anything else, such as a function a Custom step defined, is not available after a resume, and a step that uses it stops with a message saying so.

  • Not resumable: a run with --read-only, :memory:, or a --database outside the job directory.

  • Visibility. While a step runs, other programs reading the job’s database (the Dashboard, the web UI) see it as of the last finished step.

Stopping a task set#

TaskSet.cancel() – or Evaluator.cancel() for a step that works through an evaluator – may be called from another thread while the set is running: the tasks in flight are cancelled on their back end (scancel of their bundle, or the local process killed), those still held for room are dropped, all are marked cancelled in the manifest so that a later run of the step submits them afresh rather than adopting a killed job, and run() yields a cancelled result for each before returning. A cancelled task gets its attempt back. Results already produced stand, and so does a task that had already finished but was not yet collected – in a bundle still running, or finishing while being cancelled: it is collected with its result, not marked cancelled. A step that runs several task sets at once can therefore stop the others when one fails.

Several task sets submitting at once to the same cluster share one count of the user’s queued jobs, so the queue’s max_queued_tasks holds for them together rather than for each.

Parallel loops#

Since 2026.10.5 a Loop can run its iterations in parallel (loop_step’s “Run iterations in parallel”). Each iteration is a task of the program seamm: its resolver (seamm_exec.iteration) runs run_flowchart on the machine that runs it, with the iteration’s share of cores and memory in SEAMM_CE. The iteration’s evaluator resumes the job’s flowchart into just that iteration, from a checkpoint the Loop writes into a snapshot of the job’s database in the iteration’s _evaluator directory; its step directories are where a serial loop puts them. The tasks go where any tasks go – the evaluator’s own pool, or bundled into queue jobs when the job’s target sends tasks to a queue – and the Loop merges each iteration back in order when it is done. With “inline” placement an iteration’s own calculations run in its share (its SEAMM_TARGET is empty); with “separate tasks” (experimental) they go to the job’s target. The Loop’s user guide describes the options and the contract.

Timing records and the cost model#

Every code step appends one row per run to ~/.seamm.d/timing/<program>.csv through record_task_timing() (a run made through the task layer) or record_timing() (one made with executor.run): the machine class (cluster, partition and CPU model, not the hostname), the cores, the wall time and outcome, and the descriptors of the calculation – the kind of task, the method’s class, basis functions, electrons, atoms, how crowded the structure is (neighbours, the mean number of atoms within 8 Å), the iterations taken, the code’s own time. A task run in a queue bundle gets its wall time from the worker on the node that ran it. A file is set aside when it grows past 50 MB or when the columns change, and many runs may append at once. A run never waits long for another to finish writing: if the file stays locked for a second, its row goes to a file of its own in ~/.seamm.d/timing/<program>.pending/, which it moves into the shared file the next time it writes one. Reading the records includes these rows. The design is in docs/developer_guide/campaigns/2026-10-05.

The fit is a power law in the size variables a step declares, with an offset per method class, task and machine, and a parallel exponent that grows with size. Each run counts in proportion to its time, so the fit is best where the time is. Every run has a start-up time, measured on a machine’s smallest runs as the gap between the wall time and the code’s own time; a parallel run has a second one, fitted per machine – launching the MPI processes, for ORCA about 11 s on Owl and 1-3 s on TinkerCliffs whatever the number of processes, so that a tiny molecule runs slower on more cores. No size exponent may be negative: when related variables (basis functions and electrons) would split one effect that way, the fit leaves one out and its report says so; records that vary the basis at a fixed number of electrons, as the seed benchmark’s basis-set ladder does, let it keep both. A step whose code has regimes that scale differently names the column that tells them apart (slope_by, MOPAC’s regime) and each gets its own exponents; one whose first step is much dearer than the rest names a column (setup_by) whose groups each get a fixed cost per run, in iterations, found by the fit.

A model only predicts what its records cover: a calculation whose size lies outside the fitted range (by more than a factor of 1.5), whose method class or task the records lack, or whose descriptors lack a size variable gets no prediction, and the step’s own estimate stands. A step asks for its estimate with seamm_exec.timing_model.estimate(), giving its own formula as the fallback: it returns the seconds, where they came from (model, formula or none), which the step puts in the task’s estimate_source and the timing record keeps, and the reason the model was not used, which it logs once per process – as a warning when the step left out a size variable the model was fitted with, since then every estimate falls back to the formula. A step’s tests check that its estimate has every size variable its records have with seamm_exec.timing_model.missing_sizes(). The parallel exponent comes from paired runs when there are at least four: the same calculation (machine, method class, task, sizes and options) on several core counts, as the seed benchmark makes, with the start-up times removed. Production cannot show it, since its core counts follow its sizes (small runs on 4 cores, large on 16). Paired runs count from every record, including older ones that lack a size column added since (and so are left out of the rest of the fit). Without paired runs a refit keeps the exponent the previous fit measured; failing that the exponent is fitted with the rest, or assumed when the core counts follow the size (or are all the same), and the fit’s summary says which.

A step may also name the column holding a run’s options (flags in its spec: ORCA’s ‘!’ line, less the task, basis sets and method), and each option gets a factor of its own, shrunk toward none. An option gets a factor only where the records contrast it – at least three calculations run both with and without it on one machine. An option that always comes with its own calculations (one campaign’s settings, a molecule’s elements) cannot be told from them: fitted anyway, the factors took up differences between machines and campaigns instead. The seed benchmarks make the contrasts with toggle ladders, a few runs repeated with one option changed (ORCA: NoCOSX, VERYTIGHTSCF). An option the records have never seen widens a prediction’s upper quantiles. The fit’s report lists the learned factors and possible cost drivers: recorded values and options without a factor whose runs are still systematically off, the candidates for a new size variable, class or toggle ladder.

A cost model is fitted to those rows with seamm-timing (also python -m seamm_exec.timing_model):

seamm-timing fit            # every program with records
seamm-timing fit orca vasp

which writes ~/.seamm.d/timing/models/<program>.v<N>.json (one file per version of the model) and prints a report: the rows used, R^2 in log space, the fraction of runs predicted within 1.3x and 2x, the fitted power law, and a factor per method class, task and machine class. The model is a product of separable factors – start-up constants, a unit cost as a power law in the program’s size variables with an intercept per method class, an offset per kind of task, a factor per contrasted option, a parallel exponent, and an offset per machine class – so rows from every machine improve it and a new machine needs only a few runs to be placed.

A prediction is a quantile of the fitted spread, never a mean:

seamm-timing predict orca task=energy \
    method_class="global hybrid" nbf=240 --ntasks 8 -q 0.95

or, from a step, seamm_exec.timing_model.predict(), which returns the seconds at the quantile, the median, the iteration count it assumed, and whether this machine class was in the fit (an unseen class gets no offset and a wider spread). The 95th percentile is the right choice for a queue walltime, the median for packing tasks into a bundle.

The models keep themselves current: a prediction refits the model when the records have grown by a fifth since the fit, or when it is a week old and the records have changed, or when the step’s spec or the fitting method has changed since, under a lock so that concurrent runs do not all refit, and only replaces the old model when the new one is fitted to at least as many rows and predicts at least as well. The fit command remains for a report, or to refit on demand.

Whether the estimates the tasks were given match their times is checked from the records themselves, which carry both (estimated, before a target’s estimate_scale, and wall):

seamm-timing check                  # every program, last day
seamm-timing check orca --since 2h
seamm-timing check orca --since 2026-10-16T10:30

It prints, for each machine and calculation (method class, task, method, basis and atom count, those the records have; --by names others), the runs, their mean time and estimate, and the geometric mean of estimate / time with its 10th and 90th percentiles. It flags a group whose estimates are further than a factor of two from the times (--limit), and a calculation estimated the same on machines whose times differ, the sign that the estimates came from a step’s fallback formula, which knows nothing of the machine, and not from the model, and a group with estimates recorded as from the formula. --since takes a span (90m, 2h, 1d, the default) or a local date and time (the records’ dates are in UTC), or all; runs that timed out are left out. The exit status is 1 when a group is flagged. Run it after every update of an installation that runs tasks, on the rows written since: a wrong estimate shows in the first finished runs, long before bundles run out of time or are packed too thinly.

A machine the model has not seen is placed by the seed benchmark, a few minutes of standard runs whose rows the fit can tell apart (benchmark=<set>):

python -m seamm_exec.timing_benchmark --codes orca,mopac --cores 1,4,8 --fit

Each code step declares what to run (TIMING_BENCHMARK, beside its TIMING_SPEC): the systems that drive its cost, as the flowchart steps that build them, and its model chemistries, tasks and variants with a size limit per tier – ORCA a few molecules from water to a 300-atom alkane at two or three method classes, including the MLFF labelling level (REVDSD-PBEP86-D4_2021/def2-TZVPPD) with every energy also as a gradient, a basis-set ladder, and toggle ladders (a chemistry may carry its own variants); MOPAC the same molecules to 3000 atoms in both its regimes. seamm_exec finds the declarations through the step entry points, assembles one flowchart per run and sweeps the core counts where a code is parallel; it knows nothing of any code. --bin names the installation whose run_flowchart to use when the benchmark is driven from another environment; --build-only writes the flowchart to run by hand or through a JobServer.