A small judge before the commit
At work we put a System One model to answer one question per file before every agent commit. The hard part was writing the question well.
Can you have a model check an agent’s work before every commit, without it taking as long as a review or handing back three paragraphs of opinion? The short answer is yes, if the model answers a single question with a number and a script writes the repair message.
We ran that experiment at work recently, on a large Rails application. We used a model from the System One class, the one I described in A model that does not write: it generates no text, it takes a statement and returns the probability that it is true. We tried it in two places. One was the agent’s commit, which is the part I like most. The other was estimating how risky a PR is before merging it.
Two kinds of check
The first step was separating the checks that have a known answer from the ones that need judgment.
The first kind was already solved. RuboCop, the Ruby linter, says whether the code follows the style rules. Packwerk says whether a module uses something it should not. A path rule says whether a change touches only tests. None of them needs a model, and putting one on top means paying to be right less often.
The second kind is what used to land in a person’s review. Does this test assert what the user sees on screen? Does this comment repeat what the code already says? Is this PR safe to merge without anyone looking at it?
That is where the model comes in, with one constraint: it answers a narrow question and it does not write a review.
Where the first check came from
We did not make it up. We went looking for what agents actually get wrong.
We went through old implementation sessions. Out of 110 candidate chats, 24 were finished implementations, and 22 of those 24 had needed a manual correction. It is the same symptom I described in Self-improving loops, now with a count.
The judgment problem that came up most was thin UI tests. The agent would write a spec that visited a page, clicked, and checked that the URL was the right one. The test passed and said nothing about what the person saw on the screen.
That became the first check.
The commit that bounces
The mechanism is a pre-commit hook. When the agent tries to commit, a command runs and does three things: it runs RuboCop on the Ruby files in the commit, it asks the model the questions that apply, and it exits with an error if any of them fails. Git rejects the commit.
The design decision that paid off most was that split. The model makes the judgment, and our script supplies the repair instruction. When a check fails, the agent does not get an opinion composed on the spot. It gets a fixed message, written by us once, that says what is missing and what to do.
That keeps the output short and actionable. An agent that reads three paragraphs of review argues with them or fixes half, and one that reads an instruction carries it out and tries again.
This is what it looks like from the agent’s side. The test it writes passes: it opens the page and checks that an element with a certain CSS class exists. That class is an implementation detail. The page can be empty, or show an error inside that element, and the test stays green.
1. The agent writes the test
2. It tries to commit and the commit bounces
This spec does not assert anything the user sees. Add at least one expectation on visible content: a heading, a table row, a message. A path or a CSS selector does not count. Fix the spec and commit again.
3. It fixes the test and tries again
The second step has two blocks, and they have different owners. The number and the FAIL come from the model. The paragraph below them is ours: it is written by hand once per check, and the script prints it unchanged every time that check fails.
That paragraph is what steers the agent. It does not argue the particular case or explain why it scored 0.027. It says what counts as a fix and what does not, which is exactly what the agent would have tried next. The commit goes in on the second attempt and nobody had to write a review comment.
Later we pulled the command out to the console, as bin/pre-commit-checks. With no arguments it checks the files in the commit, which is what the hook uses. With files as arguments it reads them from disk. That last part looked like a detail and turned out to be what let us try a new wording of the question against loose files, without building a commit.
What went wrong: a good test, rejected
The first version of the UI test question seemed to work. We tried three cases we wrote ourselves:
- Visit, click and check the URL: 0.004. Fails, as it should.
- Check a table and the visible copy: 0.964. Passes.
- Check only a heading: 0.982. Passes.
Then we ran it against a real test from the project. It used have_section "Employees", a helper of our own that checks the page has a section with that title. It scored 0.415 and failed.
That rejection was wrong. The test followed the project’s convention of asserting text the user sees. The model did not know the helper, and our question did not tell it that this counted.
We changed the question, not the threshold. With the new wording:
have_section "Employees": 0.998. Passes.- Table and visible copy: 0.995. Passes.
- Check a test ID and a CSS class: 0.027. Fails.
- Visit and check the URL: 0.007. Fails.
For me this was the most useful part of the experiment. The rubric changed because of a false rejection on real code, and the cases we wrote ourselves would never have shown it.
We also simplified when it runs. The first version required the commit to include a UI template as well as its test. But the template was only acting as a switch: the model never read it. We dropped it. The question now runs for full-page tests and component tests, and every other test only goes through RuboCop.
The second check: comments that repeat the code
The second question was whether a comment merely repeats what the code already says.
A method with the comment # Increments the visit count by one scored 0.999 and failed. The same method without the comment scored 0.256 and passed. Language directives and comments that explain a constraint you cannot see in the code were allowed by the rubric.
It is a small check, and it is exactly the kind of correction a person makes over and over and never gets around to writing down as a rule.
The risk of a PR
The original idea for the experiment was a different one: a score in continuous integration that would mark a PR as mergeable on its own or as needing human review. There were several candidate signals, among them whether the change sits behind a feature flag, whether it touches shared contracts, how much coverage it lacks and how reversible it is.
The first thing we saw is that a single weighted score can hide a dangerous signal. Several calm signals can cover up a serious one. The safer model was hard gates plus softer judgments.
We tried the soft version anyway. We built bin/pr-risk, which takes a PR and returns a number. The first version answered low, medium or high risk. We replaced that with a single figure, safe 0.527, where higher means safer to merge without review. With the number pointing that way, any later gate reads without translation.
And here came the second stumble. A PR that changed when a core business rule fires, with 36 files and more than 3,000 added lines, scored 0.730. A change like that does not get merged without someone reading it.
We tested three possible causes:
- The model: we repeated the same request three times and all three returned 0.730. It was not noise.
- Diff truncation: we removed almost all of the diff and it returned 0.713. It was not missing context.
- The question: this was it.
The rubric said that what was unsafe was “a behavior change with no spec”. That PR had specs, so to the model it could be safe. We took out three words and the criterion became “a behavior change”. The result dropped to 0.016, and after more tuning to 0.003.
The model was following the question to the letter, and the question was wrong.
After that we tuned the rubric against the latest 100 merged PRs, with examples from the project for each side. On the safe side went changes that only touch tests, fixes for flaky tests, dependency bumps, continuous integration changes and internal tooling. On the unsafe side, new features, production behavior changes, payments, authentication and migrations.
Some results after tuning:
- Flaky test fix: 0.969.
- Dependency bump: 0.990.
- Permission change with an effect in production: 0.005.
- Change to a core business rule: 0.003.
This is still an experiment. Nobody merges anything on that number alone, and the conclusion about hard gates still stands.
The repository’s history is the training material
Looking back, all three times a check got better it was because we put it up against something that already existed in the repository.
- The agents’ old sessions told us what to ask. Without that count we would have written whichever check came to mind, not the one that was needed.
- The tests that were already written exposed the false rejection.
have_sectionis a convention of ours, and it only shows up if you run the question against the project’s code. - The latest 100 merged PRs defined what is safe and what is not for this team. Each one already had a verdict: someone had reviewed it, or had let it through.
I put “training” in quotes because the model does not change. What gets adjusted is the question, and the history serves as a set of examples with the answer already filled in. A repository with years of commits, reviews and tests comes with the most expensive part of any classifier already done, which is getting real cases labeled by people who know the project.
It also explains why these checks do not copy over from one team to another. A generic rubric about UI tests would have rejected our best tests. The one we ended up with only works for this repository, and that is exactly why it works.
What I take from it
The interesting part of the experiment was not putting a model in pre-commit but turning corrections a person kept repeating into small, executable judgments. The recipe we ended up with:
- One narrow question. One statement per check, with enumerated criteria for true and for false.
- Only when it applies. The file type decides whether the question runs, and code settles that.
- A probability back. The threshold is ours and it can move.
- The repair message, fixed. The model judges and the script instructs.
- The commit bounces. The agent corrects itself, before a person sees it.
- The rubric is tuned against the repository’s history. Cases you write yourself make it look better than it is.
- A bad result is first a suspicion about the question. Both times the number surprised us, the fault was in our wording.
A probabilistic check asks for the same precision as any other code. A fast model does not fix an ambiguous rule: it answers it fast.