Test the Design, Not Just the Code

A Kepler mission is a handful of planets, a starting point and a target. Nothing in the code says whether it can be won, whether winning takes a perfect run, or whether any run at all gets through. Those are properties of the design, and a unit test does not reach them. The code can be correct and the mission can still be impossible, or trivial, or no fun.

So every mission goes through a harness before it ships. The harness flies it in simulation thousands of times and keeps the trajectories that work, and what it finds is what we redesign the mission around.

Sample the design to see how forgiving it is

The first pass is random. The harness flies the mission over and over with varied burn timings and records how each attempt ends.

On S08 Close Pass that run took 10,600 attempts. 660 finished. 4,970 crashed into a planet, 4,771 drifted off the edge of play, and 199 ran out of time. The finish rate, 6.2%, is the width of the corridor: how much imprecision the mission absorbs before it punishes you. Nobody had to play it to find that out.

Search the design to find the best it can do

The second pass is directed. A beam search looks for the least fuel that completes the mission, and on Close Pass it converged on 0.200 units over 12.5 seconds. That is the floor, the best a perfect player could ever do.

The floor is not just a test result. It is what the Daily score is measured against, so a player’s score is how close they came to the best run the harness could find.

Sampling and search answer the same question from two sides. One says how hard the mission is to finish, the other says how well it can be finished, and together they say whether it deserves to ship.

Read the result, change the mission, run it again

The harness does not decide what a good mission is. We do. A mission early in the archive is teaching, so its corridor should be wide enough to absorb wrong inputs. A mission late in it is asking for mastery, so its corridor can be narrow. Run the harness, read the result against what the mission is for, move a planet or the target, and run it again.

The same search works at the scale of the whole game. Kepler’s missions come out of a design tool that generates candidate missions, solves them and keeps the ones worth playing: one batch alone generated, solved and traced 10,000 candidates. 121 missions ship.

Search is where an agent earns its keep

The most visible use of AI tooling is generation: code that writes itself, copy, pictures. The part that changes what a small team can do is search. A studio with twenty playtesters can put people through every mission for a week. We cannot, and with a harness we do not need to. We specify the boundaries and the outcome, and the search finds what works. It is the same pattern as making every expected value checkable by code instead of trusting a suite that tests itself, applied to a design instead of a calculation.

It transfers well past games. Any design with an outcome you can measure can be tested this way: a pricing rule against historical orders, a schedule against a month of real shifts, a layout against the paths people take through it. When “does this design work” is the question, build the harness before the design gets polished, because it is what tells you which designs are worth polishing. It is the same judgment behind where a spec earns its cost: the harness belongs where a wrong answer is expensive and a right one can be measured.

A player runs the same loop

A kid flying Close Pass does not know they are running a harness in their head. They know I went too close that time and let me try farther out. A few dozen attempts, a sense of where the corridor is, and eventually a clean run near the floor. Sample, notice the pattern in the failures, search toward what works. The harness exists for engineering reasons, and the game ends up teaching the method as well as the physics.

If you have a design whose outcome you can measure and nobody can test by hand, tell us what you’re working on.