Last year's AI-assisted first-pass review was advisory only, a human could ignore it entirely, and mostly did when the suggestions felt like noise. This year we promoted a narrower version of it to an actual required status check in our PR pipeline, empowered to block a merge on a specific, well-defined class of issues. The distance between "advisory" and "required" turned out to be much larger than the distance between "no AI review" and "advisory AI review," and getting it right meant relearning some lessons we thought we'd already absorbed about automated gates in general.
Why advisory review wasn't working
The problem with an advisory check that a human can freely ignore is that it only adds value when the human reading it happens to agree it's worth acting on, and over time, engineers develop a sense for which categories of advisory feedback are usually worth their attention and which usually aren't. Our advisory review had drifted into the second category for a meaningful fraction of the team by the end of last year: useful often enough to keep around, but not trusted enough to change behavior reliably. A check that's silently ignored isn't really providing the safety net it's nominally there for, it's providing the appearance of one.
Making a check required, rather than advisory, forces a different kind of discipline on whoever builds it, because now every false positive has a real cost: an engineer blocked from merging a legitimate change because the check got something wrong. That cost concentrates attention on getting the check's scope right in a way that advisory feedback never demanded, since nobody was ever forced to reckon with an advisory suggestion that missed the mark.
Scoping the check narrowly on purpose
The check is deliberately narrow: missing test coverage on new code paths and a short list of known anti-patterns, not general code quality, which keeps false positives low enough that the team trusts it. We resisted real internal pressure to expand its scope earlier in the year, several engineers reasonably pointed out that the underlying model was clearly capable of flagging a much broader range of issues, and it was. Capability wasn't the limiting factor. Trust was, and trust doesn't scale the same way capability does.
Expanding its scope beyond that narrow list, in early testing, produced enough false positives that engineers started reflexively overriding it, which defeats the purpose of a required check entirely. We ran a two-week pilot with a broader scope, including general code style and a wider set of pattern suggestions, on a subset of repositories before deciding against it more broadly. The override rate during that pilot climbed high enough, engineers hitting "override and merge anyway" on a majority of flagged pull requests by the second week, that we could see the same trust erosion that had killed the advisory version starting to happen again, just one layer further down the trust hierarchy.
- The check flags only missing test coverage on new code paths and a short, explicitly maintained list of known anti-patterns.
- A two-week pilot with broader scope produced an override rate high enough to signal the same trust erosion that undermined the advisory version.
- Every override is logged and reviewed weekly, both to catch genuine false positives and to catch engineers overriding legitimate flags out of habit.
- Expanding the check's scope now requires evidence from the override log, not just confidence that the underlying model has improved.
The override log as a feedback mechanism
One thing we added that we hadn't originally planned: every override gets logged and reviewed weekly by whoever owns the check's configuration. That review serves two purposes that pull in opposite directions and need to be held separately. First, it catches genuine false positives, cases where the check was simply wrong, which get fed back into refining its logic. Second, it catches cases where an engineer overrode a legitimate flag out of habit or impatience rather than because the check was actually mistaken, which is a signal about trust and process rather than about the check's accuracy, and needs a different kind of response, usually a conversation rather than a configuration change.
Separating those two categories in the weekly review has been more valuable than we expected going in. Early on we were conflating them, treating every override as evidence the check needed adjusting, which would have pushed us toward loosening the check's standards in response to engineers simply not wanting to fix legitimate gaps in test coverage. Distinguishing "the check was wrong" from "the check was right and got overridden anyway" turned out to require actually reading the diffs behind each override, not just counting them.
What we're carrying forward
A narrow, well-scoped automated gate that the team trusts is worth far more than a broad one people learn to ignore. We're expanding its scope slowly and only where we've built real confidence in its accuracy, using the override log as the primary evidence for whether a given expansion is ready, rather than our own intuition about what the model is theoretically capable of flagging. That's a slower path than the check's underlying capabilities would technically allow, and we're deliberately fine with that. The whole value of a required check comes from the team not needing to think about whether to trust it, and that trust, once eroded, has proven much harder to rebuild than it was to establish carefully the first time.
Deciding exactly which anti-patterns belonged on the check's short, explicit list took more deliberation than building the check's underlying detection logic. We started from a list our senior engineers had informally flagged in code review over the previous year, things like catching an exception broadly and swallowing it silently, or introducing a new database query inside a loop where a single batched query would do, patterns specific enough to describe unambiguously and common enough that catching them automatically actually saved real review time. We deliberately left off anything that required subjective judgment about naming or structure, since disagreement about those categories is exactly what erodes trust in an automated gate the fastest.
Each item on the list went through a trial period as an advisory-only flag before being promoted to a blocking condition, giving us a chance to see its real-world false positive rate on live pull requests before it had the power to actually stop someone from merging. A couple of candidate anti-patterns never made it past that trial period because their false positive rate on our actual codebases turned out to be higher than expected, usually because a pattern that looks like an anti-pattern in isolation was, in a specific recurring context in our code, actually the correct and intentional choice.
We treat the check's configuration, its list of anti-patterns and its test coverage thresholds, as a piece of code with its own owner and its own review process, rather than a black box someone configured once and forgot about. Changes to what the check flags go through the same pull request review as any other change to shared infrastructure, and we've required that any proposed addition to the anti-pattern list come with examples of real code the new rule would have flagged, pulled from our own history, so the team can evaluate the false-positive risk concretely rather than in the abstract. That discipline has kept the check's scope from creeping upward through a series of individually reasonable-sounding additions that would have, in aggregate, reproduced the exact trust problem the broader pilot ran into.
We built an explicit, low-friction path for an engineer to flag that the check got something wrong, distinct from simply overriding it and moving on, specifically because we wanted a signal that was easier to act on than parsing intent out of override logs alone. That path routes directly to whoever owns the check's configuration that week, and a genuine false positive reported this way gets addressed, either by refining the detection logic or by adding an explicit exception for the pattern in question, within a few days rather than sitting in a backlog indefinitely.
Making that feedback loop fast turned out to matter more for trust than we expected going in. An engineer who reports a false positive and sees it acknowledged and fixed within days develops real confidence that the check is being actively maintained rather than left to quietly degrade, and that confidence generalizes to how they treat the check's other flags going forward, even ones unrelated to the specific issue they reported. A required check that visibly improves in response to feedback earns a kind of trust that a static, unresponsive check never does, regardless of how accurate it is on average.