Testing Blender Scenes Built by LLM Agents

Why Agents Need This

A person modelling in Blender checks their work by looking at it from every angle, constantly. An agent driving bpy gets none of that. I saw four failure patterns come up again and again in the project log:

• It can't see. The agent knows what its script asked for. It doesn't know what the evaluated scene looks like after modifiers, constraints, drivers and parenting have all had their say.
• It believes its own description. "The rifle sits in the right hand" is a sentence in the build log. Whether the rifle is actually in the hand on frame 260 is a separate question, and nobody has asked it yet.
• It fixes the measurement. Pushed to make a check go green, an agent will loosen a tolerance, add an exception or switch to a test that happens to pass. Then it reports the defect as fixed.
• It loses track of which file is current. Fifty-five slices produce a lot of .blend files, spec versions and backups. A check that passed on the wrong one proves nothing.

The obvious fix for the first one is to render the scene and hand the image to a model. That helps less than you'd hope. A vision model gives you decent context: what's in the shot, whether it reads as a hangar, whether the mech looks heavy. It's bad at the specifics, like a part sitting 2 cm out of line or a finger pushed through a grip. Those are exactly the defects that pile up in an agent-built scene.

Every family below catches one of those failure patterns by measuring the built scene with a script instead of trusting a picture or a description of it.

Ground Rules for Every Check

These held across every family. Most were learned the hard way.

1. Measure the evaluated result. Read the depsgraph-evaluated meshes and the posed bone matrices in world space. Don't read the parameters the build script passed in. blackbetty's engagement check won't pass anything on a declaration alone. If the spec says two parts have 3 cm of running clearance, the check measures it on the built mesh.
2. Keep checkers read-only and separate from builders. A verifier never edits product code, and the agent that built something never certifies it. For the claims that mattered most, the project also kept a second implementation of the check that shared no code with the first.
3. Prove each checker fails. This is mutation testing, borrowed straight from software. Write a script that plants known defects in a copy of the scene and confirm the checker goes red on every one. blackbetty's mech_selftest.py flips a face, inverts a cube, empties a material slot, unparents a part and parents another to the wrong bone. rig_selftest.py adds a 3 cm foot slide, a knee flip and a detached toe. A checker you've never seen fail is untested.
4. Return numbers. JSON with PASS, FAIL, WARN or SKIP, the measured value, and the worst offender with its frame, plus a nonzero exit code on any FAIL. "Looks fine" isn't a result.
5. Pin what you checked. Record the .blend path and hash, the spec version and the frame range in every result, and re-run the gates on the exact file you ship.
6. Save a picture. Render a still with the offending parts colour-coded and a human-sized figure for scale. A person reviewing it understands a failure in seconds, where the JSON takes a page. The picture is there to explain the number, and the number still decides the verdict.

Families 1–4: Geometry and Structure

I'd run these after every model build. They're cheap and they catch the most common mistakes. Nothing downstream is worth checking until they pass.

1. Scene integrity and hygiene

Broken data the agent never looked at.

• Mesh: edges that aren't closed or shared properly, inconsistent face winding, inside-out shells (negative signed volume), zero-area faces, loose vertices.
• Materials: missing materials, image nodes pointing at files that don't exist, node trees with no output. blackbetty calls this the "pink" check, after Blender's missing-texture colour.
• Data: orphan datablocks, unapplied scale, objects outside their expected collection, names that break the convention.
• Budgets: triangle count per object and per scene, after modifiers.

2. Spec conformance

The model drifts from the design while the agent believes it matches.

• Part dimensions, bounding boxes and joint pivot positions against a spec file, within a tolerance.
• Shape against a reference surface for parts with a defined form. blackbetty's shape gate builds the expected crown curvature, wall draft and thickness of each cast armour part from the spec and scores the built mesh against it.
• Inventory: everything in the spec exists, and nothing exists that isn't in it.
• A spec lint that runs without Blender. If the spec contradicts itself (a piston whose rest length is outside its own stroke, a joint pivot that isn't on its bone), there's no point building from it.

3. Symmetry and naming

Left/right mistakes are among the most common things an LLM gets wrong in 3D. Usually the agent mirrors something with the wrong sign.

For every .L/.R pair, compare dimensions, vertex counts and mirrored world position. Then check that the side in an object's name matches the side it's actually on, and the side of the bone it's parented to. blackbetty has a separate scan just for actuators that compares the side in each bone name, object name and actual position.

4. Hierarchy and rig structure

Parts follow the wrong thing, or come loose once they move.

• Every part is parented to the bone it's meant to follow.
• Each part's transform stays constant relative to its bone across frames: no drifting, no single-frame pops.
• Joint limits, axes and signs agree with the rig contract and the spec.
• Drivers and constraints get dumped and compared against an expected list.
• Socket switches are clean: exactly one active socket at full influence, with no half-blended frames where a weapon floats between two hands.

Families 5–6: Collision and Contact

5. Clash and interpenetration

This was the biggest family in blackbetty by a long way. Geometry passing through other geometry is the defect an agent is most likely to "fix" without fixing.

• At rest: every pair of parts that can move relative to each other.
• Across joint ranges: sweep each degree of freedom, and pairs of them, and flag clashes that weren't there at rest.
• Across the animation: every frame, or a dense sample with refinement around the worst frames.
• Swept volumes: moving parts against static set geometry. blackbetty checked the hangar's beams against the space the mech's parts swept through over the whole film.
• An explicit list of allowed overlaps. Some overlaps are on purpose, like a piston rod running inside its barrel. Each one goes into a versioned file with a reason and a tolerance. Nothing gets allowed silently.

"At rest" sounds like the easy case. It's where blackbetty's largest known issue came from. The joint sweeps only flagged clashes that showed up during motion. Parts that already touched at rest got treated as mounting points. That hid 245 pairs of parts, on bones that move relative to each other, sitting 3 cm or more inside one another before anything moved. The worst was a shoulder housing 0.34 m into the upper-arm girder. A dedicated rest-pose scan found them late in the project. They shipped as a known issue, since most were buried inside armour shells.

How you test "is this point inside that mesh" matters more than any other detail in this family. I'll come back to it in the section on when the checks were wrong.

6. Contact and grounding

Floating, sinking and sliding.

• The lowest point of each foot, prop or resting object against the ground.
• Planted feet drift less than a tolerance. Take the planted windows from the timeline, or detect them automatically as frames where the foot control isn't moving.
• Hands on held objects: how deep fingers sink into the object, and how big the gap at each fingertip is.
• Objects resting on surfaces: the gap, and whether the centre of mass sits over the support area.

Families 7–8: Physics and Motion

7. Mechanism and physical plausibility

Motion or design that's geometrically valid and physically absurd.

• Actuator and linkage lengths stay inside their stroke, and pistons stay engaged in their barrels at full extension.
• Balance: centre of mass and zero-moment point inside the support polygon, with a margin.
• Joint torque against capacity, from an inverse-dynamics replay. blackbetty exported every frame's bone matrices and replayed them through a simplified MuJoCo model of the mech.
• Acceleration limits. A 550 t machine doesn't jolt its pelvis at 5 g.

This one is easy to build and hard to satisfy. The MuJoCo checker found the walk peaking at 2.45 g, the sword tip at 100 g during the slash, and the balance point leaving the feet on every rifle kick. Fixing that properly meant re-choreographing the whole film, so the failures were logged and accepted as a decision. I think accepting a failure is fine, as long as it's written down and nobody quietly retunes the checker until it passes.

8. Motion quality

Some problems only show up in playback.

• Joint angular velocity and acceleration per frame. Use the shortest-arc angle between orientations. If you don't, a quaternion sign flip reads as a joint spinning at 10,800°/s. blackbetty's first pacing tool did exactly that, and v2 exists to fix it.
• Knees and elbows never flip and never lock straight.
• Pose continuity: no bone or prop moves further than a threshold in one frame, except at intended cuts.
• Pacing by beat against the intended rhythm, and keyframe counts and checksums diffed between versions so you know what a change actually touched.

Families 9–10: Camera and Pixels

9. Camera and composition

Correct geometry can still look wrong on screen.

• Project the evaluated vertices through the shot camera to get the subject's size on screen, clipping at the frame edge, and whether key objects are in frame.
• Visibility: is a defect actually on screen, inside the frame and not hidden behind something? If nobody can see it, you can usually waive it. blackbetty ray-tested every clash deeper than 1 cm from the shot camera. That turned "a thumb up to 25 cm inside the rifle" into "right-hand clipping visible on 14 of 29 frames," which told the animators exactly where to work.
• Depth order and scale cues, like crew figures standing at the right distance from the mech's feet.

10. Render output

Some problems only exist in pixels.

• Frame count and continuity, black or NaN frames, flicker (brightness jumps outside of cuts), fireflies, and temporal noise.
• Exposure: crushed and clipped percentages, contrast between the subject and the background, colour cast that stays consistent across shots.
• Before/after pairs with one layer toggled, to isolate exactly what that layer changed.
• Delivery checks on the encode: codec, pixel format, frame rate, resolution and frame count, read back with ffprobe.

Families 11–12: Pipeline and Review

11. Pipeline determinism and isolation

Nondeterministic builds, and agents stepping on each other's work. That second one gets common fast once several agents edit one scene.

• Build twice and compare hashes. blackbetty kept two hash tools on purpose: the pipeline's own, and a stricter one the QA lead wrote without looking at it, covering every datablock, per-vertex digests, node trees and every keyframe handle. A mutation script then nudged one vertex, swapped a material and changed a light colour to prove the hash noticed.
• Pin the environment: Python hash seed, colour-management config, Blender version, add-ons, --factory-startup. One probe existed only to confirm that hash randomisation was really off inside the Blender process the build launched.
• Ownership: each module owns one collection, and a module that writes outside it fails the build. A probe breaks this rule on purpose to confirm the build catches it.
• Resource locks, so checks don't fight renders for the GPU.

12. Visual review

Some things no metric expresses. Does it read as heavy? Does the hand look like it's holding the grip?

• Fixed-view evidence stills (front, side and top orthographic, plus a 3/4 view) with offending parts colour-coded and a human-sized figure for scale.
• Contact sheets and playblasts.
• A separate reviewer ranks the result against a reference and the brief. The reviewer recommends, and someone else decides.
• This is the one family where a vision model pulls its weight, because the questions are about the overall read of a shot. Alignment and clipping stay with families 2 through 6.

When the Checks Were Wrong

Three times in blackbetty, a check gave a confident answer that turned out to be wrong.

The trigger finger that passed at 0.80 cm

A verifier passed the right index finger on the rifle trigger with 0.80 cm of penetration, inside the 1 cm limit. A later builder working on the trigger pull measured the same finger 3.05 cm into the trigger at rest.

The trigger paddle was one merged mesh: a plate, a grip rib standing 2.5 cm proud, and a pin. The original check found the nearest face to each finger point and used that face's normal to decide inside or outside. On a merged mesh, the nearest face to a point buried in the rib is often a face of the rib itself, pointing the wrong way. So the measured depth could never go past about 1.25 cm, no matter how deep the finger went. The PASS was an artefact of the method.

Fixing the model was easy. Flattening the rib dropped worst-case penetration to 0.96 cm. Fixing the process took more work. Every PASS in the project that had relied on a nearest-face measure was re-audited. Five held, the trigger failed, and two more were flagged for fingers sitting 8–21 cm inside the rifle in older animation files.

The parity test that cried wolf

The usual alternative is ray parity. You cast a ray from the point, and an odd number of surface crossings means you're inside. On meshes built from several overlapping closed shells, like the sword's frame, parity reports points as inside when they aren't. One check reported a finger 0.23 m inside the sword. A slower test that cast 96 rays in different directions showed it was outside.

So one method under-reports and the other over-reports. The shared clash library the project ended up with only calls a point inside a part when both agree, per closed shell: the nearest-face normal points away from the point, and at least two of three skewed rays have odd parity. Before anything was allowed to depend on it, a verifier proved it caught three known disasters (a gun 0.54 m into the chest, fingers 0.25 m into the rifle, a sword 0.31 m through a cradle beam) and still passed the fixes that had already landed.

The rifle that was right at rest and 57 cm off in the animation

The trigger work was verified on the rig's rest placement of the rifle. In the animated build, the index finger sat about 57 cm from the trigger. The root timeline had a socket-flip flag set to true on top of a socket that was already flipped, so the rifle turned around twice. Every check on the rest pose was correct, and none of them mattered. It's ground rule 1 again: measure the built, evaluated scene over the frames that matter.

What to Run When

I'd get the spec-only checks running as early as you can. blackbetty had a small engine that posed the spec's own boxes and cylinders, ran clash and balance checks on them, and solved a whole self-righting sequence from lying face down to standing, all before any real geometry existed. A problem caught there costs a JSON edit instead of a rebuild.

What to Take Away

• Measure the evaluated scene in world space, over the real frames. The spec, the rest pose and the build script's intentions are only stand-ins.
• Mutation-test every checker before trusting it. The worst PASS in blackbetty came from a check that couldn't measure deep enough to see the defect.
• Use two inside tests and require both to agree. Nearest-face normals under-report on merged meshes, and ray parity over-reports on overlapping shells.
• Keep builders and verifiers apart. The agent that made the change doesn't get to certify it.
• Use vision models for the overall read of a shot. Leave alignment and clipping to scripts.
• Ask whether a defect is visible before spending a day on it. Projecting through the shot camera turns a vague worry into a list of frames.
• Write down what you accept. A logged waiver is fine. A tolerance that got quietly loosened hides the problem.

I've packaged most of this into a Claude Code skill, scripting-blender-scenes. It tells an agent which checks a job needs based on its size, and how to prove a check works before relying on it. If you're having agents build anything in Blender, I'd start there. Then add whichever families above your scene actually needs.

All of this came out of a brute-force attempt at learning: run the agents, watch what broke, write a check for it, and go again. There's a lot of room left to improve the process itself, and some of that could make whole families of tests unnecessary. blackbetty planned to wire the clash library straight into the posing solvers, so a hand couldn't be posed through a grip in the first place, but that got parked when time ran out. If you have thoughts on any of this, find me on LinkedIn or email me.