Run-to-run determinism

51 · Solver and runtime

Forty objects poured into a bin twice from identical inputs. Run A is solid and run B is a wireframe in the same place; each body of A turns green as soon as its pose differs from B, and the readout shows the largest difference. By default rows are reserved with atomics, so the Gauss-Seidel order can change between runs and the chaotic pile amplifies the difference. With deterministic=RUN_TO_RUN and a deterministic collision pipeline the recorded body poses of the two runs are bitwise identical.

drag to orbit · scroll to zoom

0.00 s

Cost

Time per step, CUDA graph replay, median of 7 interleaved rounds on one NVIDIA RTX PRO 6000 Blackwell Server Edition; ratios are against the first row. Solver allocation is the GPU memory the solver allocates at construction, before warm-up and graph capture. Scene: the 8-box stack in each world, 8 steps of 1/240 s per frame.

Option	1 world	64 worlds	1024 worlds	solver allocation, 1024 worlds	result in this tutorial
default	1.00 ms	1.21 ms	1.64 ms	215 MB	pours end 20 cm apart
RUN_TO_RUN	1.06 ms 1.06×	1.26 ms 1.04×	1.70 ms 1.04×	219 MB	bitwise identical

What to look at

Default: the first bodies turn green at about 0.3 s and the runs end about 20 cm apart.
RUN_TO_RUN: every body stays neutral; the readout shows 0, bitwise identical, for the whole run.

Key parameter

deterministic

wp.DeterministicMode.RUN_TO_RUN, together with CollisionPipeline(model, deterministic=True).

Try changing

Make only the solver deterministic.
Time both modes on a large scene.

Concept

Run-to-run determinism

What it is

Rows are reserved with atomics by default, so their order, and with it the Gauss-Seidel result, can differ between runs. RUN_TO_RUN reserves rows in a fixed order; with a deterministic collision pipeline the same inputs give bitwise-identical results on the same GPU and build.

In this scene

Two default pours end 20 cm apart; RUN_TO_RUN gives byte-identical recorded poses.

Considerations

Cost: About 3.5% per step (1.64 → 1.70 ms in 1024 worlds) and about 5 MB more solver memory for 1024 worlds.
Accuracy: Neither order is more accurate; only reproducibility changes.
Interactions: body_parent_f sums with atomics and stays outside the guarantee.
Use it when: Debugging, regression tests and reproducible data.

The script

cd tutorials && python 51_determinism.py prints each variant's final metric; --record out.json also writes the replay. Every tutorial imports replay.py from the same folder.

51_determinism.py
replay.py

# SPDX-FileCopyrightText: Copyright (c) 2026 Skild AI and the Steel contributors
# SPDX-License-Identifier: Apache-2.0

"""Run-to-run determinism: the same pile poured twice, with and without deterministic row order."""

import newton
import numpy as np
import warp as wp
from newton.solvers import SolverSteel

from replay import Recorder

DT = 1.0 / 240.0
SUBSTEPS = 8


def build():
    builder = newton.ModelBuilder()
    cfg = newton.ModelBuilder.ShapeConfig(mu=0.5)
    builder.add_ground_plane(cfg=cfg)
    for s in (-1, 1):
        builder.add_shape_box(
            -1,
            xform=wp.transform((0.22 * s, 0.0, 0.1), wp.quat_identity()),
            hx=0.01,
            hy=0.22,
            hz=0.1,
        )
        builder.add_shape_box(
            -1,
            xform=wp.transform((0.0, 0.22 * s, 0.1), wp.quat_identity()),
            hx=0.22,
            hy=0.01,
            hz=0.1,
        )
    rng = np.random.default_rng(3)
    for i in range(40):
        axis = rng.normal(size=3)
        q = wp.quat_from_axis_angle(wp.normalize(wp.vec3(*axis)), float(rng.uniform(0.0, np.pi)))
        body = builder.add_body(
            xform=wp.transform(
                (rng.uniform(-0.12, 0.12), rng.uniform(-0.12, 0.12), 0.15 + 0.06 * i), q
            )
        )
        if i % 2:
            builder.add_shape_box(body, hx=0.03, hy=0.025, hz=0.02, cfg=cfg)
        else:
            builder.add_shape_sphere(body, radius=0.025, cfg=cfg)
    return builder.finalize()


def simulate(rec, deterministic, slot):
    """Pour twice and record both runs, the second as a translucent ghost over the first."""
    name = "deterministic=RUN_TO_RUN" if deterministic else "default"
    runs = []
    for k in range(2):
        model = build()
        mode = (
            wp.DeterministicMode.RUN_TO_RUN
            if deterministic
            else wp.DeterministicMode.NOT_GUARANTEED
        )
        # a deep pile needs more free-body rows than the default 512
        solver = SolverSteel(model, deterministic=mode, free_body_constraint_max=4096)
        # the contact order must be fixed too
        pipeline = newton.CollisionPipeline(model, deterministic=deterministic)
        contacts = pipeline.contacts()
        state_0, state_1, control = model.state(), model.state(), model.control()
        run = rec.variant(f"{name}, run {k + 1}", model, ghost=k == 1, slot=slot)
        for _ in range(rec.frames(4.0)):
            run.frame(state_0)
            for _ in range(SUBSTEPS):
                pipeline.collide(state_0, contacts)
                solver.step(state_0, state_1, control, contacts, DT)
                state_0, state_1 = state_1, state_0
        run.check(solver)
        runs.append(run)
    # metric: the largest position difference between the two runs at each frame
    a, b = (np.stack(r.poses)[:, :, :3] for r in runs)
    divergence = 1000.0 * np.linalg.norm(a - b, axis=2).max(axis=1)
    for r in runs:
        r.metrics = [{"divergence_mm": round(float(d), 4)} for d in divergence]


if __name__ == "__main__":
    rec = Recorder()
    simulate(rec, False, 0)
    simulate(rec, True, 1)
    rec.save()

# SPDX-FileCopyrightText: Copyright (c) 2026 Skild AI and the Steel contributors
# SPDX-License-Identifier: Apache-2.0

"""Record Steel tutorial runs as replay JSON for the tutorials page, or print a summary.

Every tutorial imports this module. ``python NN_name.py`` runs the variants and prints
the final metric of each; ``--record out.json`` also writes body poses, shapes and metric
series that the page replays in 3D.
"""

import argparse
import base64
import json
import os
import subprocess
import sys
import time

import newton
import numpy as np
import warp as wp

from source_hash import source_sha256

_RENDERED_TYPES = {
    newton.GeoType.PLANE,
    newton.GeoType.SPHERE,
    newton.GeoType.CAPSULE,
    newton.GeoType.ELLIPSOID,
    newton.GeoType.CYLINDER,
    newton.GeoType.BOX,
    newton.GeoType.MESH,
    newton.GeoType.CONE,
    newton.GeoType.CONVEX_MESH,
}


class Recorder:
    """Collect the variants of one tutorial; ``fps`` is the replay frame rate."""

    def __init__(self, fps: int = 30):
        parser = argparse.ArgumentParser()
        parser.add_argument(
            "--record", metavar="OUT.json", help="write a replay file for the tutorials page"
        )
        parser.add_argument("--device", default=None, help="Warp device, for example cuda:0")
        self.args = parser.parse_args()
        if self.args.device:
            wp.set_device(self.args.device)
        self.fps = fps
        self.script = os.path.abspath(sys.argv[0])
        self.variants: list[Variant] = []

    def frames(self, seconds: float) -> int:
        """Number of replay frames in ``seconds``, including the first."""
        return int(round(seconds * self.fps)) + 1

    def variant(
        self,
        label: str,
        model: newton.Model,
        *,
        ghost: bool = False,
        slot: int | None = None,
        scene: str | None = None,
    ) -> "Variant":
        """Start one run; ``ghost`` overlays run ``slot``, ``scene`` groups runs for charts."""
        variant = Variant(label, model, self.fps, ghost, slot, scene)
        self.variants.append(variant)
        return variant

    def save(self) -> None:
        """Print a summary and, with ``--record``, write the replay file."""
        for variant in self.variants:
            last = variant.metrics[-1] if variant.metrics else {}
            flag = "  (rows dropped)" if variant.overflow else ""
            print(f"{variant.label:<48} {json.dumps(last)}  {variant.wall:.2f} s{flag}")
        if not self.args.record:
            return
        source_sha = source_sha256(self.script)
        doc = {
            "script": os.path.basename(self.script),
            "script_sha256": source_sha,
            "newton_commit": _newton_commit(),
            "newton_version": newton.__version__,
            "warp_version": wp.config.version,
            "gpu": wp.get_device().name,
            "fps": self.fps,
            "variants": [v.to_json() for v in self.variants],
        }
        with open(self.args.record, "w") as f:
            json.dump(doc, f, separators=(",", ":"))
        print(f"wrote {self.args.record}")


class Variant:
    """One recorded run: a pose per frame for every body, plus named metric values."""

    def __init__(self, label, model, fps, ghost, slot, scene):
        self.label, self.model, self.fps, self.ghost, self.slot = label, model, fps, ghost, slot
        self.scene = scene
        self.poses: list[np.ndarray] = []
        self.metrics: list[dict] = []
        self.awake: list[list[int]] = []
        self.particles: list[np.ndarray] = []
        self.overflow = False
        self.start = time.perf_counter()
        self.wall = 0.0

    def frame(self, state: newton.State, awake=None, particles=None, **metrics) -> None:
        """Record ``state.body_q``, optional awake flags, particle positions and metric values."""
        self.poses.append(state.body_q.numpy().astype(np.float32).copy())
        if particles is not None:
            self.particles.append(particles.numpy().astype(np.float32).copy())
        self.metrics.append({k: _round(v) for k, v in metrics.items()})
        if awake is not None:
            self.awake.append([int(x) for x in awake.numpy()])
        self.wall = time.perf_counter() - self.start

    def check(self, solver) -> None:
        """Note whether the solver dropped rows; call once after the run."""
        try:
            solver.check_constraint_capacity()
        except RuntimeError:
            self.overflow = True

    def to_json(self) -> dict:
        poses = (
            np.stack(self.poses)
            if self.poses
            else np.zeros((0, self.model.body_count, 7), np.float32)
        )
        return {
            "label": self.label,
            "fps": self.fps,
            "ghost": self.ghost,
            "slot": self.slot,
            "scene": self.scene,
            "bodies": int(self.model.body_count),
            "frames": int(poses.shape[0]),
            "poses": base64.b64encode(poses.astype(np.float32).tobytes()).decode(),
            "shapes": _shape_table(self.model),
            "metric": self.metrics,
            "awake": self.awake or None,
            "particles": self._particles_json(),
            "wall_s": round(self.wall, 3),
            "overflow": self.overflow,
        }

    def _particles_json(self):
        if not self.particles:
            return None
        frames = np.stack(self.particles)
        tris = self.model.tri_indices
        radius = self.model.particle_radius.numpy()
        return {
            "count": int(frames.shape[1]),
            "frames": base64.b64encode(frames.astype(np.float32).tobytes()).decode(),
            "radius": round(float(radius.mean()), 5) if radius.size else 0.0,
            "tris": [int(v) for v in tris.numpy().reshape(-1)]
            if tris is not None and tris.shape[0]
            else None,
        }


def _shape_table(model):
    types = model.shape_type.numpy()
    scales = model.shape_scale.numpy()
    xforms = model.shape_transform.numpy()
    bodies = model.shape_body.numpy()
    flags = model.shape_flags.numpy()
    shapes = []
    for i, t in enumerate(types):
        if int(t) not in _RENDERED_TYPES or not flags[i] & int(newton.ShapeFlags.VISIBLE):
            continue
        shape = {
            "b": int(bodies[i]),
            "t": int(t),
            "s": [round(float(v), 5) for v in scales[i]],
            "x": [round(float(v), 6) for v in xforms[i]],
        }
        if int(t) in (newton.GeoType.MESH, newton.GeoType.CONVEX_MESH):
            mesh = model.shape_source[i]
            shape["v"] = [round(float(v), 5) for v in np.asarray(mesh.vertices).reshape(-1)]
            shape["i"] = [int(v) for v in np.asarray(mesh.indices).reshape(-1)]
        shapes.append(shape)
    return shapes


def _round(value):
    value = float(value)
    return round(value, 5) if np.isfinite(value) else None


def _newton_commit():
    if os.environ.get("NEWTON_COMMIT"):
        return os.environ["NEWTON_COMMIT"]
    try:
        root = os.path.dirname(os.path.dirname(newton.__file__))
        return subprocess.check_output(["git", "-C", root, "rev-parse", "HEAD"], text=True).strip()
    except (OSError, subprocess.CalledProcessError):
        return None

Recorded with SolverSteel on Newton 1.9.0.dev0, Warp 1.19.0, NVIDIA RTX PRO 6000 Blackwell Server Edition. Replays read the state back every frame, so they are not a speed benchmark. Script · page text

Steel docs and tutorials are licensed under Apache-2.0. Third-party notices. "Skild", "Skild AI" and the Skild logo are trademarks of Skild AI and are not licensed under Apache-2.0.