With AI coding assistants now part of daily development for most of the team, we wanted to test the other side of the workflow: using an AI tool as a first-pass reviewer before a pull request reaches a human, to see whether it changed what humans actually needed to catch, and whether it was worth the added step in the review pipeline at all.
Setting up the experiment
We ran the tool as an automated check on every pull request across two client projects for the full quarter, posting its findings as a comment before a human reviewer was assigned. Reviewers were asked to note, informally, whether the AI's comments matched issues they would have caught anyway, missed something they considered important, or flagged something that wasn't actually a problem. That last category, false positives, turned out to matter more to the team's actual trust in the tool than raw catch rate did.
What it reliably caught
The tool reliably caught mechanical issues, missing null checks, inconsistent naming conventions relative to the rest of the file, obvious edge cases like an off-by-one in a loop boundary or an unhandled empty-array case, freeing human reviewers to focus more of their attention on architecture and intent rather than the kind of issue that's tedious to spot but mechanical to verify once flagged.
- Roughly a third of the AI's comments across the quarter were things a human reviewer confirmed they would have caught anyway, which isn't wasted effort, catching an issue faster and earlier in the review cycle still has value, but it tempered any expectation that the tool was finding things humans systematically miss.
- A smaller but meaningful share, closer to one in eight comments, flagged something a first human reviewer had genuinely missed on an initial pass, mostly small consistency issues rather than serious bugs, but the kind of thing that accumulates into real technical debt if nobody catches it across dozens of pull requests.
What it missed entirely
It missed context-dependent problems entirely, decisions that were technically correct but wrong for this specific codebase's conventions, an API response shape that worked fine in isolation but didn't match the pattern the rest of the service used, a caching strategy that was reasonable in general but wrong for this particular endpoint's actual traffic pattern. Human reviewers, particularly ones with real history on the specific codebase, still catch every one of these, because the judgment depends on context the tool simply doesn't have access to, prior incidents, team conventions that were never written down, the specific reasons a similar pattern was rejected six months ago.
- Security-relevant logic, authentication checks, permission boundaries, anything touching how user input reaches a database query, is a category we deliberately kept under full human review regardless of what the automated pass reported, since a false sense of coverage there is a genuinely dangerous failure mode, not a minor inefficiency.
- The tool occasionally flagged stylistic preferences as though they were correctness issues, confidently phrased comments about code that was working as intended, which took some early calibration from the team to learn to discount without dismissing everything else it said.
The false positive problem, and how we handled it
False positives were the biggest threat to the tool's usefulness, not because they cost much time individually, but because a reviewer who's been burned by a couple of confidently wrong comments starts skimming past all of them, including the correct ones. We addressed this by having the tool's comments appear as suggestions rather than blocking checks, and by periodically reviewing a sample of dismissed comments as a team to recalibrate configuration when the false positive rate crept up on a particular category of check.
How review conversations changed
An AI first pass is a useful filter, not a replacement for human review, and the clearest change wasn't in what got caught but in what human reviewers spent their attention on. It changed the kind of feedback our reviewers give, shifting it from nitpicks about naming and null checks, largely absorbed by the automated pass, toward the judgment calls that actually need a person: is this the right approach, does this match how the rest of the system works, will this hold up under a load pattern the author might not have considered. Review comment volume didn't drop much overall, but the composition shifted meaningfully toward the kind of feedback that's actually hard to give well, which is the outcome we'd call a genuine win rather than just a redistribution of the same work.
Effects on review turnaround time
We tracked time from pull request opened to first human comment across both projects for the quarter. It dropped modestly, not because the AI pass replaced the human reviewer's job, but because authors increasingly fixed the mechanical issues the automated pass flagged before a human ever looked at the code, meaning the first human comment was more often already at the substantive-feedback stage rather than starting with a round of "please add a null check here." That shift in where the human review conversation starts was a bigger practical win for the team's actual velocity than the raw comment-count numbers alone would suggest.
Developer reactions, which weren't uniformly positive
Not every developer welcomed the automated first pass equally. A couple of more senior developers, whose pull requests already had a low rate of the mechanical issues the tool catches, felt it added noise without much corresponding benefit for their specific work, and their skepticism was fair given their actual track record. Less experienced developers found it more consistently useful, both for the issues it caught and, less expectedly, as a kind of ambient teaching tool, seeing the same category of comment repeatedly across several pull requests seemed to help a few newer developers internalize conventions faster than waiting for a human reviewer to make the same point each time individually.
Where we're taking this next
We're planning to extend the automated first pass to check for a couple of project-specific conventions beyond the tool's general defaults, custom lint-style rules around how errors are surfaced to users in this specific codebase, for instance, since the generic version of the tool has no way to know about conventions unique to us. That customization work is still in progress, and we'd treat the current setup as a solid first version rather than the final shape of how AI-assisted review fits into our process long-term.
Cost and licensing of running the tool
Running an AI review tool as a standard part of every pull request has a real, ongoing cost, both the direct subscription or API cost and the CI time added to run it on every push, and we tracked both explicitly against the time savings it produced. At our current pull request volume, the tool paid for itself comfortably in reviewer time saved on the mechanical issues it caught, though we'd recheck that math before recommending it unconditionally to a much smaller team with a lower volume of pull requests to amortize the fixed cost across.
Extending the experiment to a third project
Given the results across the first two projects, we've added the automated first pass to a third, larger client codebase with a more established, more senior-heavy review culture, specifically to test whether the earlier finding, that more experienced developers see less value from it, holds at a different team's scale, or whether it was specific to the two teams in the original experiment.
Team-wide rollout considerations
Before extending this beyond the original pilot teams, we spent time writing a short internal guide covering how to interpret the tool's comments, why a dismissed suggestion isn't necessarily a wasted comment, and how to recalibrate configuration when a specific check's false-positive rate creeps up, since without that context a new team adopting the tool cold would likely repeat the same early trust-calibration process we went through ourselves rather than benefiting from what we'd already learned.
What we'd measure differently next time
If we ran this experiment again from scratch, we'd track reviewer time spent per pull request more rigorously from day one rather than relying partly on informal reviewer impressions, since a cleaner before-and-after time measurement would have made the cost-benefit case, and the team-size threshold at which it clearly pays for itself, considerably easier to state with confidence.