Command-line reference

The executable is scaffoldscope. Every command accepts -h or --help. The same interface is available as python -m scaffoldscope.

usage: scaffoldscope [-h] [--version]
                     {init,validate,plan,run,report,status,trials,replay,budget,check,demo,clean,bundle,verify-bundle,plugins,schema,import-swebench,export-swebench,export-swebench-matrix,ingest-swebench,doctor} ...

Hold the model still. Measure the scaffold.

positional arguments:
  {init,validate,plan,run,report,status,trials,replay,budget,check,demo,clean,bundle,verify-bundle,plugins,schema,import-swebench,export-swebench,export-swebench-matrix,ingest-swebench,doctor}
    init                create a safe, runnable starter experiment
    validate            validate an experiment config
    plan                expand and persist a trial matrix without running it
    run                 run or resume an experiment
    report              regenerate Markdown, HTML, JSON, and CSV reports
    status              show durable progress without rewriting experiment
                        files
    trials              list every planned trial and its durable outcome state
    replay              inspect one trace strictly offline without running
                        tools or models
    budget              estimate grid size, power, and configured spending
                        bounds
    check               verify result-bundle completeness and identity hashes
    demo                run the bundled offline engine demonstration
    clean               remove generated workspaces but retain evidence
    bundle              create a deterministic, workspace-free evidence
                        archive
    verify-bundle       verify an evidence archive without extracting it
    plugins             list extension points or import-check installed
                        plugins
    schema              print or export the packaged experiment JSON Schema
    import-swebench     convert downloaded SWE-bench rows to a local task
                        manifest
    export-swebench     export one paired cell for the official evaluator
    export-swebench-matrix
                        export every treatment cell with unique official-
                        evaluator run IDs
    ingest-swebench     attach an immutable official-evaluator overlay to one
                        cell
    doctor              check local or experiment-specific prerequisites

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

Path conventions

Commands accept three distinct path types:

Path

Source of truth

Commands

Configuration file

JSON you edit

validate, doctor --config, budget, plan, run

Experiment directory

Generated evidence directory

status, trials, replay, report, check, bundle, clean, SWE-bench exports and ingestion

Bundle archive

Deterministic ZIP

verify-bundle

plan and run print the resolved experiment directory. Use that path. Its suffix is derived from the full configuration identity.

Exit codes

Code

Meaning

0

Command completed. Episode-level non-solves can still be valid results.

2

Configuration, integrity, operating-system, or infrastructure failure. run also uses this when it records infrastructure-invalid trials.

130

Interrupted by the operator.

Use status --json, trials --jsonl, replay --json, and budget --json for automation. Human-readable output is not a stable machine protocol.

init

Create a safe, runnable starter project.

usage: scaffoldscope init [-h] [--name NAME] [directory]

positional arguments:
  directory    starter destination (default: ./scaffoldscope-study)

options:
  -h, --help   show this help message and exit
  --name NAME  experiment identifier written to the starter config

scaffoldscope init my-study --name my-study

directory defaults to ./scaffoldscope-study. --name sets experiment.name; it does not merely label the folder. The initializer refuses nonempty unowned directories, conflicts, and symlink targets. Repeating the same initialization can recover missing managed files while preserving operator edits.

validate

Load the source config, task manifest, task sources, prompt, and selected plugins. Validate strict types and fields, then compute the full experiment identity.

usage: scaffoldscope validate [-h] config

positional arguments:
  config

options:
  -h, --help  show this help message and exit

scaffoldscope validate experiment.json

This command does not call a model, execute tests, or inspect Docker images.

doctor

Check package or experiment-specific local prerequisites without running a trial.

usage: scaffoldscope doctor [-h] [--config CONFIG]

options:
  -h, --help       show this help message and exit
  --config CONFIG  preflight one experiment without running it

scaffoldscope doctor
scaffoldscope doctor --config experiment.json

With --config, the command resolves the provider, reports credential presence without exposing the secret or variable name, validates selected plugins, and preflights the Docker image when selected. It does not contact model endpoints; the JSON output reports provider connectivity as not-checked.

schema

Print or export the packaged Draft 2020-12 experiment schema.

usage: scaffoldscope schema [-h] [--out OUT]

options:
  -h, --help  show this help message and exit
  --out OUT

scaffoldscope schema --out scaffoldscope-experiment.schema.json

Without --out, the schema is printed to standard output. Export refuses to replace a different existing file.

budget

Show the planned matrix size and declared resource ceilings before a run.

usage: scaffoldscope budget [-h] [--json] config

positional arguments:
  config

options:
  -h, --help  show this help message and exit
  --json

scaffoldscope budget experiment.json
scaffoldscope budget experiment.json --json

The estimate is based on configured limits. It is not a provider invoice and does not predict early termination.

plan

Freeze the resolved configuration and full task, replicate, and treatment matrix without making provider calls.

usage: scaffoldscope plan [-h] config

positional arguments:
  config

options:
  -h, --help  show this help message and exit

scaffoldscope plan experiment.json

plan writes the experiment identity files and plan.jsonl. Host runtime and observed Docker identity remain unpinned until the first real run so a reviewed plan can move to its intended worker.

run

Execute or safely resume the complete experiment matrix.

usage: scaffoldscope run [-h] config

positional arguments:
  config

options:
  -h, --help  show this help message and exit

scaffoldscope run experiment.json

The command performs provider, plugin, sandbox, runtime, and evidence preflight before reusing or starting trials. It pins runtime identity on first execution, creates isolated per-trial workspaces, writes atomic results, rebuilds the aggregate in plan order, and regenerates reports. Integrity-invalid partial attempts are archived before re-execution.

There is no flag to run only selected treatments or tasks. Filtering a paired matrix at execution time would alter the declared experiment.

status

Summarize progress without changing experiment evidence.

usage: scaffoldscope status [-h] [--json] experiment_dir

positional arguments:
  experiment_dir

options:
  -h, --help      show this help message and exit
  --json

scaffoldscope status runs/my-study-1a2b3c4d
scaffoldscope status runs/my-study-1a2b3c4d --json

The output distinguishes planned, completed, pending, failed, interrupted, and external-evaluation states and reports recorded usage-provenance gaps.

trials

List the stable per-trial inventory.

usage: scaffoldscope trials [-h] [--status FILTER_STATUS] [--variant VARIANT]
                            [--task TASK] [--jsonl]
                            experiment_dir

positional arguments:
  experiment_dir

options:
  -h, --help            show this help message and exit
  --status FILTER_STATUS
  --variant VARIANT
  --task TASK
  --jsonl

scaffoldscope trials runs/my-study-1a2b3c4d
scaffoldscope trials runs/my-study-1a2b3c4d --variant selective --status resolved
scaffoldscope trials runs/my-study-1a2b3c4d --task parser-001 --jsonl

Filters affect display only. --jsonl emits one JSON object per selected trial.

replay

Verify and display one persisted event timeline without invoking a provider, model, tool, evaluator, or mutable workspace.

usage: scaffoldscope replay [-h] [--json] experiment_dir trial_id

positional arguments:
  experiment_dir
  trial_id

options:
  -h, --help      show this help message and exit
  --json

scaffoldscope replay runs/my-study-1a2b3c4d <trial-id>
scaffoldscope replay runs/my-study-1a2b3c4d <trial-id> --json

JSON replay can contain prompts, source excerpts, model content, tool observations, evaluator output, and error details. Treat it as sensitive evidence.

report

Rebuild derived analysis files from frozen raw evidence.

usage: scaffoldscope report [-h] [--bootstrap-samples BOOTSTRAP_SAMPLES]
                            [--analysis-seed ANALYSIS_SEED] [--sesoi SESOI]
                            [--open]
                            experiment_dir

positional arguments:
  experiment_dir

options:
  -h, --help            show this help message and exit
  --bootstrap-samples BOOTSTRAP_SAMPLES
  --analysis-seed ANALYSIS_SEED
  --sesoi SESOI
  --open

scaffoldscope report runs/my-study-1a2b3c4d
scaffoldscope report runs/my-study-1a2b3c4d --open

--bootstrap-samples, --analysis-seed, and --sesoi perform a post-run sensitivity analysis and overwrite the reports in that directory. They do not change raw trials. Evidence bundling always regenerates the canonical report from the frozen configuration, so keep sensitivity outputs separately.

check

Validate the experiment’s cross-file evidence contract.

usage: scaffoldscope check [-h] [--strict] experiment_dir

positional arguments:
  experiment_dir

options:
  -h, --help      show this help message and exit
  --strict        also fail on report warnings

scaffoldscope check runs/my-study-1a2b3c4d
scaffoldscope check runs/my-study-1a2b3c4d --strict

Normal mode verifies required artifacts, identities, exact plan coverage and ordering, task and treatment provenance, pricing, aggregate equality, trace lifecycle and context evidence, patch hashes, evaluator overlays, and canonical reports. Strict mode also fails on analysis warnings. Tiny scripted demos are expected to fail strict publication checks.

bundle

Create a deterministic, workspace-free evidence archive.

usage: scaffoldscope bundle [-h] --out OUT experiment_dir

positional arguments:
  experiment_dir

options:
  -h, --help      show this help message and exit
  --out OUT

scaffoldscope bundle runs/my-study-1a2b3c4d --out my-study-evidence.zip

The output must be outside the experiment directory and must not already exist. The command validates raw evidence, regenerates canonical reports, and hashes every included file. It never includes mutable generated workspaces.

verify-bundle

Verify a ScaffoldScope evidence ZIP without extracting it into the caller’s workspace.

usage: scaffoldscope verify-bundle [-h] archive

positional arguments:
  archive

options:
  -h, --help  show this help message and exit

scaffoldscope verify-bundle my-study-evidence.zip

Verification checks safe canonical archive paths, allowed files, declared sizes and hashes, outer and inner experiment identity, experiment semantics, and canonical report regeneration for the current integrity profile.

clean

Remove generated trial workspaces after evidence has been verified.

usage: scaffoldscope clean [-h] --workspaces experiment_dir

positional arguments:
  experiment_dir

options:
  -h, --help      show this help message and exit
  --workspaces    confirm removal of generated trial workspaces

scaffoldscope clean runs/my-study-1a2b3c4d --workspaces

The confirmation flag is mandatory. The command retains configs, plans, results, traces, patches, reports, overlays, and archived failed-attempt evidence. It rejects symlinked or escaping workspace paths.

plugins

Discover installed extension metadata or explicitly import and validate extensions.

usage: scaffoldscope plugins [-h] [--check] [--json]

options:
  -h, --help  show this help message and exit
  --check     explicitly import and validate every discovered third-party
              plugin
  --json

scaffoldscope plugins
scaffoldscope plugins --check
scaffoldscope plugins --json

Discovery reads distribution metadata without importing plugin modules. --check authorizes imports, validates registration and compatibility, and computes implementation provenance for every discovered plugin.

demo

Copy and run the larger bundled offline engine demonstration.

usage: scaffoldscope demo [-h] [--directory DIRECTORY] [--open]

options:
  -h, --help            show this help message and exit
  --directory DIRECTORY
                        copy the demo project here (default: ./scaffoldscope-
                        demo)
  --open

scaffoldscope demo
scaffoldscope demo --directory ./engine-demo --open

The demo is deterministic and costs nothing. It exercises four context policies across three fixtures. It is a system demonstration, not a benchmark.

import-swebench

Convert SWE-bench dataset JSON or JSONL rows into a ScaffoldScope task manifest using trusted local repository clones.

usage: scaffoldscope import-swebench [-h] --repo-cache REPO_CACHE --out OUT
                                     source

positional arguments:
  source

options:
  -h, --help            show this help message and exit
  --repo-cache REPO_CACHE
  --out OUT

scaffoldscope import-swebench swe-bench-lite.json \
  --repo-cache /bench/repos \
  --out tasks/swe-bench-lite.jsonl

The imported tasks intentionally have no local evaluator command. Official SWE-bench grading is attached later through immutable external-evaluation overlays.

export-swebench

Export one treatment and replicate cell as official prediction JSONL.

usage: scaffoldscope export-swebench [-h] --strategy STRATEGY
                                     --replicate REPLICATE --out OUT
                                     experiment_dir

positional arguments:
  experiment_dir

options:
  -h, --help            show this help message and exit
  --strategy STRATEGY
  --replicate REPLICATE
  --out OUT

scaffoldscope export-swebench runs/lite-1a2b3c4d \
  --strategy selective \
  --replicate 1729 \
  --out predictions.jsonl

The output contains instance_id, model_name_or_path, and model_patch.

export-swebench-matrix

Export every treatment and replicate cell, unique evaluator run IDs, checksums, matrix identity, and a generated runbook. The operator must separately pin the evaluator installation and images used to execute it.

usage: scaffoldscope export-swebench-matrix [-h] --out-dir OUT_DIR
                                            [--dataset-name DATASET_NAME]
                                            [--split SPLIT]
                                            experiment_dir

positional arguments:
  experiment_dir

options:
  -h, --help            show this help message and exit
  --out-dir OUT_DIR
  --dataset-name DATASET_NAME
  --split SPLIT

scaffoldscope export-swebench-matrix runs/lite-1a2b3c4d \
  --out-dir evaluator-matrix \
  --dataset-name SWE-bench/SWE-bench_Lite \
  --split test

The destination must be absent or empty and outside the experiment directory.

ingest-swebench

Attach official evaluator results as an immutable overlay for one treatment and replicate cell.

usage: scaffoldscope ingest-swebench [-h] --strategy STRATEGY
                                     --replicate REPLICATE
                                     --evaluator-version EVALUATOR_VERSION
                                     --evaluator-run-id EVALUATOR_RUN_ID
                                     --image-set-digest IMAGE_SET_DIGEST
                                     experiment_dir results

positional arguments:
  experiment_dir
  results

options:
  -h, --help            show this help message and exit
  --strategy STRATEGY
  --replicate REPLICATE
  --evaluator-version EVALUATOR_VERSION
  --evaluator-run-id EVALUATOR_RUN_ID
  --image-set-digest IMAGE_SET_DIGEST

scaffoldscope ingest-swebench runs/lite-1a2b3c4d official-results.json \
  --strategy selective \
  --replicate 1729 \
  --evaluator-version <commit> \
  --evaluator-run-id <unique-run-id> \
  --image-set-digest <image-manifest-digest>

Overlays are keyed to the frozen experiment cell and never rewrite generation results. Re-ingestion must be byte-identical. Missing or incomplete evaluator outcomes remain explicit instead of becoming silent non-solves.