Validate a solver change
Workflow
- Goal
- a verdict on whether a Newton or Steel change (a version bump, a solver patch, a changed default) breaks or slows any example.
- Use when
- before updating the pinned Newton revision, before merging a solver change, or when asked which presets look wrong across the collection.
- Done when
- both revisions were rendered and checked on the same examples, every grade change is explained, and speed changes on the flagged scenes are measured.
Any coding agent, or a person, can follow this workflow from agents/workflows/validate-a-solver-change/README.md. Tools that discover skill folders also find it assteel-validate-a-solver-change.
Inputs. A base and a candidate revision (two checkouts of this repository, or one checkout with two Newton pins); GPUs. Run each revision from its own frozen checkout; never edit a tree a job imports from.
Steps
- In the base checkout: render-and-grade for every example, then visual-review. Keep the run dir.
- In the candidate checkout: render-and-grade with
collect.py --previous BASE_RUN_DIR, then visual-review.results.jsonlists regressed, improved, new and removed scenes. - check-example
--all --dry-runon the candidate. - For each regressed scene → debug-instability on the candidate, with the base render as the reference. For each scene a new setting could now speed up → tune-a-scene (later, as its own change).
- benchmark-throughput on a few representative scenes and every flagged one, base against candidate, same GPU, same world counts.
Decision
- Any scene going from ok to unstable or failed without an explained, accepted cause blocks the change.
- A slowdown beyond noise on representative scenes needs a reason in the change’s description.
Deliverables
Both galleries, the grade changes with a one-line cause each, the check results, the throughput table, and the verdict.
Steps it uses
- Render and grade renders examples with their Steel presets on one or more GPUs, records stability metrics, and grades each scene ok, suspect, unstable or failed.
- Visual review looks at frames across each episode and records a grade and a note per scene the metrics may have misjudged.
- Check an example runs an example's own test headless with a preset and records the result in
presets.toml. - Benchmark throughput measures simulation and application throughput and memory for two configurations (two backends, or two settings) under one timing contract, and tunes Steel in a fixed order.
Source: agents/workflows/validate-a-solver-change/README.md, the copy agents read.
Steel docs and tutorials are licensed under Apache-2.0. Third-party notices. "Skild", "Skild AI" and the Skild logo are trademarks of Skild AI and are not licensed under Apache-2.0.