The Evals I Run on LLM Features, and How I Roll Them Back

For the LLM features on the platform I'm building, I run a small, fixed eval set through the real product code whenever a model or the harness around it changes. I score it with deterministic checks first and a panel of model judges second, and I only promote a change that beats the current setup by more than a noise margin, with no fabricated citations. Rolling back is mostly a settings change rather than a deploy: new AI judges start in shadow mode, new automated-approval flags default to off, and the model itself is a setting that points at a row in a registry.

I'm Alex Vakhitov, an AI and software architect in London. The platform is a multi-tenant AI knowledge platform I architect and build: it answers questions from the documents people give it and generates longer documents from them. This post covers the engineering mechanics. The governance side sits elsewhere, and I link to it near the end.

What does a regression look like for an LLM feature?

Worse answers are the obvious case, but not the only one. A regression can also be the wrong format, a missing section, higher cost for the same result, or a response that never arrives.

Two of the regressions my evals caught didn't look like worse answers at all. A model upgrade produced no text on some tasks, because the new model reasoned by default and used its whole output budget before writing anything. Another eval run showed a request for a short briefing coming back in the shape of a completely different document, because a router had fallen through to the wrong internal skill.

Noise is the other problem. Some probes swung by more than half the scale across identical runs. A single green run proves little, so any rule for promoting a change has to allow for that.

Where do the eval cases come from?

For question answering and document generation there is a fixed development set of about a dozen tasks, split between questions and full documents. They run against a synthetic knowledge base with no client data in it, through the real product code rather than a test double.

Document ingestion has its own corpus, organised by failure class. Faithfulness checking has a golden set of synthetic records. The human-labelled core that should sit alongside it isn't built yet.

When do evals run?

On every pull request: deterministic faithfulness checks, which must be green, and a calibration run for the LLM judges, which is advisory.
On model or harness changes: the paid end-to-end evals are dispatched by hand, and the convention is that any pull request changing a model or the harness attaches the eval's promote or hold verdict.
Nightly or on demand: every model in the registry is checked against the live AI gateway. By convention, a change to a model's entry goes in only after that check has passed.

Every paid end-to-end eval run prices itself before the first paid call, and refuses to start if the estimate is above a spend ceiling.

I deliberately didn't schedule a nightly ingestion eval before the judge was stable. It would only have produced a red history that people learn to ignore, and that's worse than having no history.

How do I score, and how do I check the judges?

Deterministic checks come first:

Citations must resolve. This is a hard veto: a fabricated citation fails the whole run.
Output format, references and required sections are checked by rule.
Must-contain and must-not-contain checks catch specific content.

Rubric items are then scored by a panel of two model judges from two different vendors, and their verdicts are averaged. One of the two comes from the same vendor as the model being tested, so the panel isn't fully independent of it.

Checking the judges takes its own work:

I ran a deliberately weak model through the eval to prove the judges separate capability rather than passing everything.
Every report records which judges scored it, and the compare tool refuses to compare runs scored by different judges.
A missing verdict is never scored as a pass. It's excluded and flagged, and a run with a missing judge can't become a baseline.

I also removed one criterion. It checked whether numbers in an answer were supported, and it gave false positives for strong and weak models alike, while the same configuration could pass or fail on a re-run. The part I haven't finished is calibrating the judges against a human-labelled set.

What is the pass bar for release?

The rule is written down:

score   = criteria pass rate + weighted all-pass rate − small token-cost penalty
promote = score beats the incumbent by more than the noise margin
          AND no fabricated citation in any trial

The score blends how many criteria pass, how many tasks pass every criterion, and a small penalty for token cost.

It has said no twice in ways that mattered. First, a premium model costing about twice as much scored exactly the same as the incumbent, so I held.

Second, an upgrade to a new model generation scored lower, 1.35 against a bar of 1.50, and the gate refused it. The only failures were the document generations with the smallest output budget. The new model reasons by default, and the reasoning used up the whole budget before any text came out. I fixed the budget handling, lowered the default reasoning effort and fixed the routing bugs that the transcript exposed. Before re-baselining, I defined "done" as a clean run under the current judges.

For features that approve things automatically, the bar is set in numbers before anyone's opinion counts: a precision bar in the high 90s and recall around 90% on the eval corpus, consecutive clean runs, a clean shadow period in production, and sign-off.

How do I switch an LLM feature off or roll it back?

Mostly with settings rather than deploys. These are per-organisation or per-workspace settings, and they take effect without a release:

Shadow first. New AI judges run in shadow mode: they record verdicts but decide nothing. Turning one on is a flag that defaults to off, and so is automated corroboration. Switching back to shadow is always allowed, with no evidence needed.
A kill switch for the nightly automation.
Cross-provider fallback can be turned off for deployments that must stay with one provider.
A model pin per workspace, so pinning back to a model is a change of setting.

A code revert goes through the normal pull request checks and release pipeline, and a release takes around 10 minutes. I haven't had to revert an LLM feature in production yet. For the one large change to how the platform reaches the models, the rollback plan was a revert plus restoring the old credentials, which I kept valid for a week.

This is the principle I described in Why I named my company Comonad: know the boundary, prove the new path, keep a way back. Shadow mode is how a new path gets proved.

Do I pin model versions?

Yes. Each model is a registry row with its exact ID and host. The AI SDK packages are pinned to exact versions, and the lockfile pins the rest. Eval baselines record the judge models too.

When I moved to a new model generation, I removed the old ID from every path that could select it, including eval configurations. I kept its pricing row and label, so historical cost records still add up. A nightly catalogue check fails if the gateway stops listing any model I rely on.

One gap is written down rather than hidden: a vendor's list of allowed fallback targets isn't checked automatically yet.

What do I watch in production that evals don't catch?

Cost per call as billed by the gateway, and how many calls nothing priced. The per-step ledger behind this is part of my agent harness.
Fallback rate, with rate-limit failovers tagged separately from provider failures.
Timeouts. The retry absorbs them, so they never show up as failures. I log each one as its own warning.
Gateway credit, checked hourly, with an alert to a person below a floor.
Error rates.
Drift in the AI judges' verdicts from one night to the next.
A human-agreement rate for the automated fact-checker. It still runs in shadow, and its agreement with human audits is published over a rolling window.
Contract checks on every live external data source, nightly, because an upstream API changing underneath you is something no eval of your own code will see.

Evals and flags are the engineering half. Ownership, approvals and audit trails are the other half, and I've covered them in Comonad's guide to governing AI agents in production.