Testing AI-Generated Code: Where the Bugs Hide

Agent-written code fails differently than a colleague's does, and the difference is not that it is worse. It is usually syntactically clean, it usually passes the tests sitting next to it, and it usually looks done. The failure that survives review is the case nobody named, sitting in a diff that reads as complete because nothing in it looks unfinished.

The direct answer: testing AI-generated code means checking whether the tests would fail if the fix were reverted, not just whether the suite currently passes. An agent that writes both the fix and the test proving it can produce a test that passes against almost any version of the code, including a broken one, and a green checkmark on that pairing tells you nothing except that the two pieces agree with each other.

TL;DR

A passing test suite proves the code satisfies the tests, not that the tests check the right thing, and agent-written code makes that gap worse because the same agent often writes both halves. The fastest independent check is the revert trick: undo the fix and confirm the test actually fails. If it still passes, the test proves nothing and the review has to look at the code directly. Review order matters more than review effort: check presence and the revert result before judging style or coverage percentage, because those two catch the failures a skim misses and polish does not.

Why Agent-Written Code Fails Differently

A person under deadline pressure who skips a case usually leaves a trace: a TODO comment, a Slack message, a PR description that says "didn't get to the retry path." An agent asked to fix a bug and write a test for it does not reliably leave that trace. It returns a diff and a passing test run, and the missing case is simply absent, inside a change that reads as finished because the part that exists is well formatted and green. AI agents in software development covers why this shape of work suits an agent in the first place: a checkable target rewards fast, mechanical output, and testing AI-generated code has to confirm the target being checked is the right one.

The Revert Trick: Does the Test Actually Prove Anything?

Undo the fix, keep the test, and run it. A test worth trusting fails the moment the bug it was written against comes back; a test that still passes with the bug reinstated was never checking for that bug, it was checking that some code ran without throwing, which almost any change satisfies. This is the practical, single-change version of mutation testing: instead of auditing a whole suite against generated mutants, you audit the one test that matters most, the one an agent just wrote to prove its own fix, against the one mutation that matters most, reverting the fix itself. It takes under a minute and it is the cheapest check in this process for what it rules out.

Signal	What it tells you	What it misses
Suite is green	Every existing assertion passed against the new code	Whether any assertion actually targets the bug that was fixed
New test passes	The agent's test agrees with the agent's fix	Whether the test would fail without the fix
Revert, test fails	The test genuinely detects the reverted bug	Whether the fix is complete, only that this one case is covered
Revert, test still passes	The test proves nothing about this bug	Nothing new; this is the result that should stop the merge

The diagram below shows the revert check as a decision tree: one branch confirms the test is trustworthy, the other stops the merge before review time is spent on anything else.

Testing AI-Generated Code: The Review Order That Catches What a Passing Suite Misses

Read the original report before the diff. Write down the specific behavior that was supposed to change; a test can only be judged against what it was meant to prove.
Run the revert check. Undo the fix, run the new test, confirm it fails. If it passes, treat the test as decorative until it is rewritten.
Check the test's assertion, not just its existence. A test that asserts a response is not null is weaker than one that asserts the specific value the bug report described; read what is actually being checked.
Only then review style, coverage percentage, and formatting. These are the least informative signals in the whole review and the ones most likely to make a weak test look finished.

How an AI agent fixes a bug end to end covers the four mechanical steps, reproduce, investigate, draft, test, that an agent can own unwatched. The revert check sits at the boundary between that loop and the review gate: cheap enough to run on every agent-opened pull request, and the one step that tells a reviewer whether the tests in front of them are evidence or theater.

Where Agent-Written Tests Lie to You

The specific pattern worth watching for is a test that exercises the changed code path without asserting the behavior the bug report complained about: a null check added for a crash, tested by confirming the function no longer throws, without ever asserting it returns the right value once it stops throwing. The suite goes green, the crash is gone, and the original wrong output ships untouched because nothing checked for it. Reviewing AI-generated work covers the same asymmetry at the review-process level: check presence against the original scope before judging quality, because a polished, passing, incomplete change is the failure mode that survives a skim.

Making the Revert Check Routine

Treat it as required on any pull request where the same agent wrote the fix and the test, the same way a linter is required rather than optional. It does not replace human review. It replaces the two minutes a reviewer would otherwise spend trusting a green checkmark that never earned the trust.

The rest of the Onplana blog's AI and Innovation coverage goes further into the review-gate and permission design a team needs before agent-opened pull requests reach production code at volume.

Frequently asked questions

How do you test AI-generated code differently than code a person wrote?

Check whether the tests would fail if the fix were reverted, not just whether they currently pass. An agent that writes both the fix and the test can produce a test that passes against almost any version of the code, including a broken one, which is a failure a person reviewing only the diff will not see.

Can a test an agent wrote pass without actually proving the fix works?

Yes, and it is the single most common failure mode. A test that asserts something trivially true, or exercises a code path without checking the specific behavior the bug was about, goes green whether the fix is correct or absent entirely, which is why a passing suite is evidence, not a verdict.

Can an agent mark a bug fixed when the underlying issue is still there?

Yes, if nothing independently checks the fix against the original failure. An agent reports what it believes is true based on the tests it ran, and if those tests do not actually exercise the reported bug, a 'fixed' status can ship with the defect still live in a code path the suite never touched.

Who is accountable if AI-generated code passes review and breaks something later?

The person who approved the merge, exactly as with a colleague's pull request. Agent involvement does not create a new category of accountability; it raises the number of pull requests arriving at review speed, which is why the revert check has to be fast enough to run on all of them, not just the ones that look risky.

Can a misleading commit message or code comment make a reviewer trust a weak test?

Yes. A comment that says 'covers the edge case' next to a test that does not is a claim the agent believed or generated to look complete, not a verified fact, and a reviewer who trusts the comment instead of reading what the assertion actually checks inherits whatever gap the comment papered over.

What happens to CI cost and review load once agents can open pull requests faster than people can merge them?

The mechanical steps, writing code and running the existing suite, scale with however many agents run in parallel, but the revert check and the human review gate do not get faster because more pull requests arrived. Staff and CI budget for the volume an agent can generate, not the volume a team produces today, or the queue backs up at the one gate meant to hold.

Where does testing fall short as a way to trust AI-generated code?

It only proves the code does what a test asserts, not that the test asserts the right thing, and it says nothing about code with no test at all. A green suite is strong evidence for the cases someone thought to check and no evidence whatsoever for the ones nobody named, which is exactly the gap a revert check narrows but cannot close by itself.