All posts
Independent AI reviewers converge on a pull-request diff that is protected by a glowing verification shield.
Generation and review should be independent systems — especially when the change controls billing.

Last week we rebuilt QualityMax’s billing system: an atomic credit wallet, Stripe subscription lifecycles, sandbox quotas, trial overlays. Sixty pull requests in about three weeks, almost all of them written by coding agents — vibe coded, if you like. Every single one went through the same AI PR reviewer we sell.

The most dangerous PR of the batch — the one that decides who gets to spend whose money — got blocked four times in twenty minutes. Not four copies of the same complaint. Four layers of the same bug, each one hiding under the fix for the previous one.

TL;DR

Across our last 60 merged PRs, the QualityMax reviewer scanned every one, issued 25 BLOCK verdicts, and found authorization bugs in our own billing code that survived the generating agent, its tests, and its self-review. Eleven of twelve findings were fixed in the same PR, usually within minutes. This post is the receipts.

Twenty minutes on PR #1384

PR #1384 enforces hosted-AI credit scopes — the code that meters what an AI workflow is allowed to spend. Deep in it was a repository method that released a “claim” on a security finding. The generating agent was confident. The tests passed. Then the reviewer started pulling the thread:

That escalation is the part that matters. Any linter can flag a missing check once. Reviewing the fix and finding the ownership gap, then the race, then the privilege hole — that’s what a senior security reviewer does, and it’s exactly the work nobody wants to do at 8pm.

What else it caught in those 60 PRs

The billing epic gave the reviewer plenty to do. A sample of what it blocked, quoted from the actual review comments:

“RLS policies for the sast_finding_fixes table are set to ‘USING (true)’ and ‘WITH CHECK (true)’, which effectively disables row-level security and allows any authenticated user to read, insert, or modify any record.”

PR #1339 · critical · fixed in the same PR — and the fix itself was re-reviewed, which caught a UUID/text cast that could have broken the policy

“‘ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT EXECUTE ON FUNCTIONS TO anon, authenticated, service_role’ grants execution rights on all future functions in the public schema to unprivileged roles.”

PR #1381 · critical · fixed in 3 minutes

“The functions reserve_sandbox_quota, finalize_sandbox_quota, and release_sandbox_quota are marked SECURITY DEFINER but do not verify that the p_user_id matches the authenticated user context, leading to potential cross-user quota manipulation.”

PR #1386 · critical · fixed the same morning, re-review confirmed

Beyond the database: a command-injection path in a gcloud compute ssh maintenance script, an IDOR on a job-status endpoint, a spoofable user_id flowing into a Celery task, a deprecated GitHub auth header — and, my favorite, a CI change that quietly deleted the migration-integrity check. The reviewer noticed the safety net was gone before anyone missed it.

Roughly 95% of the findings were authorization and injection issues — the class of bug that coding agents produce most reliably, because the happy path works and the tests the same agent wrote all pass.

Mature means boring in all the right places

Catching bugs is table stakes. What makes a reviewer usable at 60-PRs-a-month pace is everything around the catch:

  • One canonical review card per PR, updated in place on every push — no comment spam. Each finding gets a stable marker, so a re-review updates it instead of re-posting it.
  • Receipts on everything. Every card footer records the analyzed commit, the model, the re-review number, and a proof legend: VERIFIED (independently judged patch), REPRODUCED, GROUNDED (deterministic evidence), or MODEL-ONLY. You always know how much to trust a claim.
  • Stale findings retire themselves. When your push fixes a finding, the card says so — the reviewer doesn’t re-nag about code that no longer exists.
  • False positives die before you see them. A Finding Verifier re-judges every finding; on one recent PR it auto-dismissed an XSS flag with written reasoning (“the variables injected are numeric or controlled by internal state, not user-supplied input”) — and told us dismissals are advisory and reversible.
  • It learns from you. A 👎 on any finding feeds the reviewer’s memory for that repo — we wrote about that loop in Teaching the Reviewer.
  • Verified fixes, never auto-applied. For confirmed critical findings it drafts up to three patches, grounds them in your PR head, has an independent judge score them — and then leaves the merge button alone.

Honesty requires one caveat: the reviewer is occasionally over-eager. On the PR that added our concurrency test for credit allocation, it warned about a race the test existed to disprove. We’ll take that trade — on money paths we want a reviewer biased toward paranoia, and a 👎 teaches it the difference.

The reviewer reviews the reviewer

Here’s the recursive part: many of those 60 PRs were upgrades to the reviewer itself — canonical cards, idempotent submissions, PR labels, verified fixes. The reviewer blocked several of its own upgrade PRs, including the one teaching it to label reviewed PRs (it caught a deprecated auth header in that very diff). Since you can’t review your own work, we’re glad the previous version was there to check the next one.

This is what we mean by dogfooding: not a demo repo, but our production billing system, gated by the product, with public receipts.

Put a reviewer on your vibe-coded PRs

The reviewer in this post ships with the Vibe plan at €19.99/month — you generate with Claude Code, Codex, or your own keys; QualityMax independently reviews every PR. Next in this series: what Studio brings.

Start free →