The tests I actually run

Every time I fix something in the bot the same question comes up: how do I know today’s fix did not break what worked yesterday? With a normal program the answer is tests. With a bot that talks to customers the answer is the same, but a test has to deal with a model that never answers the same way twice.

The short answer is that there are four layers of tests, and only two of them block a change. The other two watch and report.

Layer 1: the verdict, with no model

In the post about guards I said that what the customer receives is decided by a pure function: it takes the draft, the tool outputs, and the history, and returns reply, handoff, or fallback, with the reason. Pure means it calls nothing outside itself. No model, no database, no network.

That makes it trivial to test. I build a run by hand, with a made-up draft and made-up tool outputs, and check what it decides. There are about fifteen hundred cases like that, and they run in seconds on every commit. It is the layer that gives the most confidence for the least money, because it covers almost every hard rule and costs nothing.

Layer 2: the contract with the store’s system

The bot reads a copy of the store’s catalog. That copy is built from the system where the store loads its prices and stock, and that system is not mine. One Tuesday in July they renamed the price fields without notice, the sync kept running as if nothing had happened, and for twelve hours the bot worked with a catalog with no prices in it.

Out of that came a contract test: it hits the real system and checks that the fields the bot expects are still there, in the shape it expects. It does not run on its own, because it depends on someone else’s service and their hours. I run it by hand when something smells off, and it is the first thing I look at when the bot says “out of stock” about something that is in stock.

Layer 3: scripted conversations

This is where bot-specific testing starts. A scenario is a multi-turn conversation with the customer’s script fixed: “hi, do you have the 15 Pro?”, then “256”, then “and in installments?”. It runs against the real bot, with the real model, but on a frozen database: a snapshot of the catalog that never changes, so the scenario always asks for the same thing and always expects the same thing.

Each scenario asserts two things. What the bot did: whether it replied, whether it handed off and for what reason. And what the reply contains or does not contain: it names the model, it does not promise shipping, it does not invent a warranty. There are ninety-six today, and they are the gate for merging: if one of the gating ones goes red, the change does not go in.

Two things I learned writing them:

• A text assertion can go empty without you noticing. One scenario required the reply to contain “17 pro”. The bot answered with the 17 Pro Max, which also contains “17 pro”, and the scenario stayed green for weeks. “This model and not the other one” is written with a stricter rule, not with a contains.
• The bot never answers the same way twice, and some scenarios flap. The same correct scenario passed one run and failed the next. Two measures: lower the model’s temperature (the knob that controls how much it varies), and mark the scenarios that keep flapping as “observation”. Those run and get reported, but do not block the change. A red in an observation scenario is a signal to look, not a regression.

Layer 4: a simulated customer

Some conversations cannot be captured by a fixed script without lying. If I improve how the bot quotes a trade-in, what the bot says changes, and the customer’s next scripted message stops making sense. For those back-and-forth chats I use a replay: one model plays the customer, with a persona and a goal taken from a real chat, and reacts to what the bot actually answers. Another model reads the full transcript and scores it against a rubric in plain language.

The replay never blocks a change. It is three models in a chain (the bot, the simulated customer, and the judge), and with three models the result varies too much to give a merge a yes or no. I read it as a trend, and I run it when a change touches its area.

The most expensive lesson in this layer was the judge. At first a single judge read the transcript and judged everything, including whether the bot was telling the truth about stock. The problem is that the judge cannot see the catalog: it judges against what it remembers of the world, which was cut off at some date. Once it graded as correct a reply where the bot said a model had not been released yet, when it was in the catalog with a price. The judge agreed with the bot, and both were wrong.

Out of that came two judges. The conversation judge only reads the transcript and judges what can be judged by reading: tone, whether it asked before quoting, whether it closed well. The catalog judge looks at the frozen database and checks, turn by turn, whether the bot claimed something the data contradicts. Mixing the two criteria in one judge produced false reds and false greens, and neither was of any use.

What it costs

All of this calls the real model, and the model charges per call. A full run of the scenarios costs about three and a half dollars. It sounds cheap until you look at how many times it runs.

In the week of August 31 I ran it, between me and the agents that maintain the bot, thirty-two times. A hundred dollars in tests to protect real traffic that cost four dollars that week. The rule we had was “one full run per fix”, and we followed it. The spend was not carelessness: it was the rule.

What changed:

• The runner refuses to run the full suite without a filter. You have to ask for it on purpose, and before running it prints what it will cost. By default you run the flow the change touched, or a small set of twenty cross-cutting scenarios that costs half a dollar.
• A red gets measured, not re-run. Before, every red triggered three attempts, and three attempts of a full suite are three suites. While I am waiting for something to go red, a single attempt.
• The corpus has a budget. A test pins how many turns the suite may cost, and raising it means touching the cap in the same PR. Before adding a scenario, I first try to strengthen, absorb, or replace one in the same flow.
• One scenario per mechanism. In one pass I went from a hundred and fifty-nine scenarios to ninety-two. Gone are the ones that repeated a mechanism with a different product, the ones that only asserted what another scenario asserted better, and the ones that flapped without gating anything. What is knowingly lost is finding out about a regression in a variant that is no longer there. If real feedback lands there, it gets absorbed into the mechanism’s scenario.

A test nobody runs because it is expensive protects exactly as much as one that does not exist.