Skip to content

Validate a solver change

Workflow

Goal
a verdict on whether a Newton or Steel change (a version bump, a solver patch, a changed default) breaks or slows any example.
Use when
before updating the pinned Newton revision, before merging a solver change, or when asked which presets look wrong across the collection.
Done when
both revisions were rendered and checked on the same examples, every grade change is explained, and speed changes on the flagged scenes are measured.

Any coding agent, or a person, can follow this workflow from agents/workflows/validate-a-solver-change/README.md. Tools that discover skill folders also find it assteel-validate-a-solver-change.

Inputs. A base and a candidate revision (two checkouts of this repository, or one checkout with two Newton pins); GPUs. Run each revision from its own frozen checkout; never edit a tree a job imports from.

Steps

  1. In the base checkout: render-and-grade for every example, then visual-review. Keep the run dir.
  2. In the candidate checkout: render-and-grade with collect.py --previous BASE_RUN_DIR, then visual-review. results.json lists regressed, improved, new and removed scenes.
  3. check-example --all --dry-run on the candidate.
  4. For each regressed scene → debug-instability on the candidate, with the base render as the reference. For each scene a new setting could now speed up → tune-a-scene (later, as its own change).
  5. benchmark-throughput on a few representative scenes and every flagged one, base against candidate, same GPU, same world counts.

Decision

  • Any scene going from ok to unstable or failed without an explained, accepted cause blocks the change.
  • A slowdown beyond noise on representative scenes needs a reason in the change’s description.

Deliverables

Both galleries, the grade changes with a one-line cause each, the check results, the throughput table, and the verdict.

Steps it uses

  • Render and grade renders examples with their Steel presets on one or more GPUs, records stability metrics, and grades each scene ok, suspect, unstable or failed.
  • Visual review looks at frames across each episode and records a grade and a note per scene the metrics may have misjudged.
  • Check an example runs an example's own test headless with a preset and records the result in presets.toml.
  • Benchmark throughput measures simulation and application throughput and memory for two configurations (two backends, or two settings) under one timing contract, and tunes Steel in a fixed order.

Source: agents/workflows/validate-a-solver-change/README.md, the copy agents read.