An agent's PR said one line. It changed five files.

17 August 2026 · a real incident from reviewing agent-authored pull requests

A coding agent opened a pull request titled "harden: redact git stderr before surfacing it". I read it. One file, one line — route an error message through a redaction helper before showing it to the user. Good change, obviously safe, merged it.

The squash commit looked like this:

.github/workflows/ci.yml    |  24 +-----
scripts/verify_merge.py     | 114 +++-------------
src/ai_jury/patches.py      |   3 +-
tests/test_verify_merge.py  | 170 +++---------------------
uv.lock                     |   2 +-

The one line I reviewed is in there. So are 300 deleted lines I never saw — a CI guard, its tests, and the job that ran it, all removed under a commit message about redacting an error string.

How it happened

Nothing malicious, and nothing exotic. The order of events:

08:18 — I read the PR. Its file list was one file. I left a review comment asking for a retitle.
11:01 — the agent pushed a new commit responding to that comment. Its working tree was stale — it predated a guard that had merged in between — so the push carried that guard's removal along with the retitle.
later — I merged, on the strength of the review I had done three hours earlier.

I approved one change and merged a different one.

Why nothing objected

Three things could have caught it, and each failed for its own reason:

The file list I had read was three hours old. GitHub does not re-notify you that the diff you reviewed has been replaced. The PR page shows the current state; my memory showed the old one.
CI was green. Deleting a guard and its tests together leaves a self-consistent repository. There was nothing left to fail.
The guard that would have flagged it was the thing being deleted. It could not report its own removal.

This is the shape worth remembering: agent PRs are not static. A human contributor who pushes after review usually says so. An agent responding to a comment pushes silently, and whatever is in its working tree comes with it.

What would have caught it

Ordered by how cheap they are.

1. Re-read the diff at merge time, not at review time

Free, and it is the one that would have worked. If any commit landed after your approval, the review is stale — treat it as unreviewed. GitHub can enforce this for you: branch protection has "Dismiss stale pull request approvals when new commits are pushed". For agent-authored PRs it should be on.

2. Compare the declared change to the actual change

The PR body said one file. The squash touched five. That mismatch is mechanically checkable before merging:

gh pr view "$PR" --json files --jq '.files[].path'

Run it at merge and compare it to what the description claims. A PR whose stated scope and real scope disagree is worth a second look regardless of who wrote it.

3. Have something read the diff that was not the thing that wrote it

The deeper problem is that a single reader — human or model — is checking work against the same assumptions that produced it. I had already decided this PR was a one-line hardening change; every later glance confirmed the belief instead of testing it.

A reviewer with no stake in that belief, looking at the diff cold, would have asked why a stderr-redaction change deletes a CI job. That is not a clever question. It just needs someone who was not there for the first conversation.

The honest limit

None of this is specific to AI agents. A human can force-push after approval too, and this failure has existed for as long as code review has. What changed is the rate: agents open more PRs, push more often in response to comments, and their working trees go stale faster than a person's would.

So the old advice — re-read before you merge — stops being good hygiene and starts being load-bearing.

What we changed

Restored the guard, and added assertions that fail when it is missing rather than when it is wrong — so deleting it now takes a diff that visibly removes assertions.
Kept the one-line redaction change. It was a good change; it just was not the whole change.

ai-jury exists for the third point above: it runs several coding agents over the same diff independently — Claude Code, Codex, Gemini, Cursor and others — and reconciles what they find into one verdict. Not because models are better readers than people, but because they have not already decided what the change is.

pipx install ai-jury
jury --pr 555

It is open source, and it would not have caught this one on its own either — the diff was correct in isolation. What catches this class is re-reading at merge time. The tool helps with the reading; the discipline is the part that matters.

← back to ai-jury