I had been tossing around the idea for a while about having AI do all code review and so I just asked the team: what if we just...didn't review the PRs?

We decided to do a couple things instead:

- Switch to a risk-based system
- Improve our guardrails (unit and e2e testing, post-deploy o11y, stricter linting and type checking, etc)

With a risk-based system, we agreed that if your change touched the public API/MCP, auth, design system, non-additive database schema changes, or agent skills, it needed a human review.

We then enforced that with a shell script to add a github label.

Improving guardrails was pretty easy, just expensive in tokens and attention.

We enabled nearly every rule in ruff/prettier/eslint/ty and we improved our unit test coverage to a floor of 85%.

We took a pretty high-level approach to o11y, preferring to instrument the customer-facing signals that indicate a bad time is about to happen (eg, ingestion, data processing, response times, auth). There's a few areas we went deeper on as needed, of course.

We also spent a bunch of time rewriting our agent skills to ensure we were giving our agents better instructions. We had a lot of cruft from 2025-era AI.

We wrote evals for our skills then tested them to see which had been consumed by modern LLM knowledge. We ultimately deleted a lot and then improved what remained.

While we were there, we found a lot of markdown docs had been accumulating from doc-happy agents and leading to context poisoning

We're now centralizing our docs into a single docs folder and requiring those be written by humans. Location gets enforced by another shell script.

The shell scripts is actually a fun bit: why use an AI for something that can be deterministic? We wrote a bunch of scripts that CI runs to enforce various things like the aforementioned docs.

We also force any changes to agent skills / agents.md go into their own PR.

Final results, before vs after:

PRs merged: 353 → 684 (80/wk → 154/wk, +94%)
Merged within 1h: 28% → 45%; within 24h: 76% → 80%

Human-reviewed PRs median merge time: 26h
No human-review median merge time: 1h