The maintenance loop is a set of skills

After “how does it work”, the next question is who looks after it. The bot handles around six thousand conversations a month, and at that volume there is always something that went wrong yesterday that nobody saw. The short answer is that it is not a person reading chats but a loop of five written procedures, run by Claude Code against the repo and against production.

Each procedure is a skill, and there are five because a bot does not break in just one way. A fact that came from no tool, a rule the prompt stopped holding, a fix that fixed something else: none of that sets off an alarm. It only shows when someone reads the chat and compares it with what happened inside, and that is the work I did not want to do by hand.

The loop, end to end

Five skills, each with a defined input and output. Two find problems, one sorts them, one fixes and one watches. Every fix ends in a note the store reads. The rest of the post walks those five stages in that order: find, sort, fix, watch, tell.

Find: the audit and what the store reports

The first entry is proactive. The audit skill takes a time window, lists the conversations where the bot wrote, and reviews every turn against the production traces. It checks three things per turn: that every concrete fact in the reply came out of a tool in the same turn, that the sequence of tools makes sense for what the customer asked, and that the bot followed the rules currently in force, which the skill reads from the code on every run instead of remembering them.

The rule that governs the audit is that every finding carries evidence from the traces, never a loose hypothesis. If a turn looks wrong and the traces do not explain it, the issue says exactly that: “I could not determine why”. A finding a customer would notice, or one that breaks an explicit rule, gets in. A wording preference does not.

The second entry is the store. It has its own page for reporting: it uploads the WhatsApp screenshot, writes what looked wrong, and that is already an issue in its tracker. Another skill takes that screenshot, locates the conversation in Chatwoot by the words in the message, and goes to the traces of the flagged turn. What it finds goes in as a comment on the issue the store opened, without touching what the store wrote, because the tracker shows it back to them.

Both paths end in the same format: the conversation, what the bot did inside with concrete evidence, and a likely cause pointing at a file. That way the skill that fixes cannot tell where it came from.

Sort: triage before fixing

Over time the backlog gets messy. The same behavior shows up twice, once reported by the store and once found by the audit, or something that had already been closed comes back. The triage skill reads both trackers together and groups by mechanism, not by symptom: two issues belong together if they would touch the same place in the code.

Its principle is that no action is taken without evidence. An issue is not closed because it “looks resolved”, and it is not marked as a duplicate because two “sound alike”. It is closed if the current code handles the case and a test that passes today covers it. If the code covers it but there is no test, the issue stays alive, and the pending work is the test.

Fix: the test comes first

The skill that fixes takes one issue and gets it to production in the same session. Before proposing anything, it checks that the finding is still alive, because an issue ages between being filed and the session. Sometimes the code already covers it and only the test is missing. Sometimes the file it pointed at no longer exists, and it has to be diagnosed again against today’s code instead of implementing the old hypothesis.

Then comes the ladder I described in the post about hallucinations: data, tool, guard, and the prompt last. With one extra alarm: if this is the second time the same rule is written into the prompt, it is forbidden, because that rule has already shown the prompt cannot hold it.

The step that is never skipped is the test that fails before the fix and passes after. And the cheapest test first: a unit test costs nothing, and a scenario against the model costs money on every run, as I measured in the post about tests. So the order is fixed, from the unit test up to a brand new scenario, and if the fix lands on the last rung it is said out loud, not added quietly.

Before touching code, the plan goes through an interrogation: root cause, scope, where the test goes, and what could regress. It is the skill’s only brake. With that closed, the fix goes to the main branch and is deployed. The deliverable is fixes, not new issues.

Watch for fifteen minutes

A deploy without a watch is incomplete. The post-deploy skill looks at three things for about fifteen minutes: that the release is alive and responding, that there are no new errors from that release, and that the symptom just fixed does not reappear in the flow that was touched. If there was no traffic in that flow, fifteen minutes of silence also close the watch, and it says so. If the symptom or an error from that release shows up, it is reverted and the cause is fixed right there.

That third condition is not decoration, and I learned it the hard way.

What went wrong: the fix that did not fix

On August 18 I did a large refactor of the function that decides what the customer receives. What I did not see is that one part of the retry had been dead for a while, and the refactor left it exposed: when the bot rejected its own draft and tried again, the second attempt failed in front of the customer. That same day I saw the failures and shipped a fix. It touched the retry and did not find the real gap. The fix that did find it landed on the 25th.

Between August 1 and 17, about 1.2% of the bot’s turns ended in a visible failure. Between August 18 and 24, about 2.1%: ninety-seven failures in seven days, counted by crossing the failure messages in Chatwoot with the turns recorded in the traces. For a week I was convinced I had fixed something that was still broken.

What that week left behind is spread across the loop. The skill that fixes demands a test that fails first, because the August 18 fix went out without one that reproduced the failure, and nothing warned me the cause was something else. The skills that open issues demand traces, because a failure shows up in Chatwoot but the cause only shows up in the trace. And the watch skill asks for one turn in the touched flow without the symptom, because a deploy nobody watches is a deploy nobody knows worked.

Tell: the note the store reads

Every fix adds one more file to a release-notes folder. One line, in the store’s language: a bold title and a detail. There are platform notes, which every client sees, and client notes, which only that client sees. Today there are seventy-eight.

The titles sound like this: “It no longer asks again for the storage you already gave”, “It survives a CRM outage without losing the reply”. They do not say guard or retry. They say what changed for the person answering the chats. A format test rejects the entry if it does not have that shape, and a repo hook rejects the commit if it brings no note and does not declare that none is needed.

The store does not read them in the repo. Production exposes them as a tool on the MCP, the server the store queries from its own Claude. They ask “what changed in the bot this week?” and get those lines back. It is the same channel they use to query their conversations, and it is the next one in this series.

A bot is not maintained by watching it: it is maintained by a procedure that demands evidence to open, a test to close, and one sentence in the store’s language to tell it.