Quality gates for AI-written code: what I check before anything merges

AI-written code on the platform I'm building goes through the same deterministic checks as any other code before it merges: lint, typecheck and the full test suite, run against a real Postgres database and including tests that enforce architecture boundaries. AI reviewers comment on pull requests, but they are advisory. The deterministic checks decide, and merging is a human decision. A few checks are advisory by design, and two of those still fail if they measured nothing at all.

I'm Alex Vakhitov, an AI and software architect in London and the founder of Comonad, where I bring AI into each stage of software delivery. The examples here come from a multi-tenant AI knowledge platform I architect and build. About nine in ten of my own pull requests on it carry a coding-agent footer or co-author trailer.

What do I write before an agent writes code?

Decision records and plans, not tickets. On this platform each significant decision gets a short architecture decision record (ADR) saying what was decided, why, and what was ruled out. There are now more than a hundred of them. Larger programmes of work get a plan split into numbered buckets. Before code starts on a risky call, it's checked against the existing ADRs.

The decision records also take precedence over the written operating manual. If the two drift apart, the manual is the one that gets corrected.

What constraints does the agent work under?

The agent runs lint, typecheck and tests locally before a pull request opens. On larger changes, a hardening loop with several AI reviewers runs first, and the agent works under a few fixed rules.

The deterministic checks are the authority on pass or fail, and the AI reviewers' findings are input to them, not a verdict. The agent can't claim a fix without showing the command it ran, the raw output and the exit code. It can't get to green by weakening config, deleting or narrowing tests, or adding suppressions, and the number of unsafe casts and suppression comments can't go up. Each finding gets its own commit, touching only the files the finding names. If a correct fix would need changes outside that scope, if the same finding keeps coming back, or after a fixed number of rounds, the loop stops and reports why.

Which automated checks run on a pull request, and in what order?

The main CI job runs in this order:

1. Lint and package-graph checks.
2. A check that the native binaries used in document processing are pinned and carry build provenance.
3. Typecheck.
4. Build a real Postgres test database from the actual migrations.
5. The full test suite, including the architecture-boundary tests.
6. A dry-run build of the background-worker bundle.

Alongside it run a browser layout smoke test at phone and tablet widths, a visual-regression check on every UI component, and an advisory calibration run for the LLM judges. AI review bots comment too.

Outside pull requests, each model in the registry is checked against the live AI gateway every night or on demand, and each live external data source is checked against its contract. On merge, the same test gate runs again before database migrations and deploys. Nothing ships unless the test job passes.

Which checks must be green, and which only warn?

Must be green: lint, typecheck, tests (including the architecture and real-Postgres tests), the worker build, the layout smoke test, and the visual text and pixel checks. A red one means the change waits until it's fixed. The model conformance check isn't part of the pull request run, but by convention a change to a model's registry entry isn't merged until that check has passed.

Advisory: the LLM-judge calibration, the paid eval runs and the live data-source contract probes.

The paid eval and the live contract probes each have one blocking step: they fail if they measured nothing at all. An advisory job that measured nothing shouldn't look green. The pattern is simple:

- name: Run the eval
  continue-on-error: true           # advisory: a regression doesn't block
  run: run-eval --out reports/ci
- name: Fail if the eval wrote no report
  if: always()
  run: ls reports/ci/*.json >/dev/null 2>&1 || { echo "::error::No report produced"; exit 1; }

I promote checks from advisory to blocking against a stated exit condition. The layout smoke test became blocking once it met its condition of consecutive green runs.

Do I gate on complexity or file size?

No. I've written about measuring cyclomatic complexity in Python and about common software anti-patterns, but on this platform neither is a merge gate. Lint runs the recommended rule set with no complexity threshold.

What I use instead are ratchets: counts that may only go down.

• Unsafe type casts and suppression comments may not increase on any pull request.
• Queries whose failure looks the same as an empty result have a ceiling per UI view. When you fix one, you lower the ceiling in the same commit, so it can't creep back.

The one size limit I changed went the other way. A test file over a megabyte was being silently skipped by the linter while lint still passed, so I raised the limit.

Who writes the tests, and how do I know they can fail?

The coding agent writes most tests alongside the code. Separate reviewer agents, one of them focused on test coverage, then flag gaps, and each gap becomes its own test commit.

Who writes a test matters less than whether it can fail. Fixes are mutation-checked: revert the change, watch the test go red, then restore it. A test that cannot fail is not evidence.

The same goes for the test environment. Tests that only record SQL can't see a query Postgres would reject, so the tests now run against a real database built from the real migrations, and on CI a missing test database fails the run instead of being skipped.

What does the AI reviewer do, and who has the final word?

I use several AI reviewers: a code-review bot, a second vendor's reviewer, and multi-agent review loops. Written review instructions are kept in the repository.

Their findings are input. The deterministic checks decide pass or fail, and merging is a human decision.

AI-written code is labelled. Most pull request descriptions end with a footer naming the coding agent, and about three-quarters of the commits on the main branch carry the agent as co-author.

Which checks have I stopped running?

A few, and I kept a written record of why each one went.

• An export-time check that had never caught a real problem, and whose design meant it could refuse a correct export. I removed it rather than patch it.
• An eval criterion that checked whether numbers were supported. It gave false positives for strong and weak models alike, and the same configuration could pass or fail on a re-run.
• A migration-parity check, once the migration was finished.

The evals themselves have their own gates and their own rollback story, which I've covered in The evals I actually run on LLM features, and how I roll them back.