What goes into my AI agent harness (and what I leave out)

My agent harness is built around one rule: every model call goes through a single service, and nothing else in the codebase is allowed to build a model client. Around that seam sit a small number of parts: a hard timeout on every call, one retry of my own for timeouts only, a classifier that decides when another provider is worth trying, a tool registry where every write needs a person's approval, a cost ledger with one row per provider round-trip, typed errors, and an offline eval set that any model or harness change is scored against. What I leave out matters as much: direct provider SDKs, failover I can't see, and metering that only happens at the end of a turn.

I'm Alex Vakhitov, an AI and software architect in London and the founder of Comonad, where I design AI agent harnesses. The examples in this post come from a multi-tenant AI knowledge platform I architect and build. People put documents in, then ask questions and generate documents from what the platform has extracted.

What is an AI agent harness?

By harness I mean everything around the model that turns it into something you can run: the loop that moves it from one step to the next, the tools it may call and with whose permissions, what goes into its context, the checks on what comes out, and the record of what it did. The model supplies the reasoning. The harness decides what that reasoning can see and touch.

In Why I named my company Comonad I described an agent step as a value in context going in and a single decision coming out. Most of the work in a harness is about that context, and about what happens after the decision.

Why does every model call go through one service?

Because a single seam is the only place where timeouts, retries, failover, cost and errors can be handled once and handled the same way. On the platform, model clients are built in exactly three files inside one AI package. An architecture test fails the build if any other file imports a provider client.

That service gives every call:

• A hard timeout that closes the connection, not just the wait.
• One retry of my own, and only for timeouts.
• A failure classifier that decides whether another provider could help.
• A ledger row per provider round-trip, so cost and timing are recorded where they happen.
• Typed errors, so callers know exactly what can go wrong.

What happens when a tool call fails?

It depends where the failure is. Each tool has a fixed time budget of well under a minute. If it runs out, only that tool call fails, not the whole turn. The model gets an error that tells it to answer from what it already has and not to try that tool again in this turn. So it changes approach instead of hammering the same tool.

For the model calls themselves, my code retries once on a timeout. Apart from one SDK-level retry on retryable errors, a real provider error isn't retried blindly. After that, a classifier decides. Server errors, network failures, rate limits and malformed output go to a different provider. Bad requests, authentication errors, content-policy refusals and exhausted credit stop there, because a second provider can't fix any of those and would only double the cost.

The classifier is a plain function with no framework imports, so it can be tested with a table of cases. In outline:

type OnFailure = "try-another-provider" | "stop";

// Every failure kind must be listed, so a new kind won't compile until it's classified.
const onFailure: Record<FailureKind, OnFailure> = {
  timeout: "try-another-provider",
  network: "try-another-provider",
  server_error: "try-another-provider",
  rate_limited: "try-another-provider",
  malformed_output: "try-another-provider",
  bad_request: "stop",       // our bug: another provider would fail the same way
  unauthorised: "stop",      // surface it rather than hide it
  content_refused: "stop",
  out_of_credit: "stop",     // and alert a person
};

Long-running jobs never re-run automatically. Each is set to a single attempt. If it fails, it fails loudly, and a person decides whether to run it again.

Which tools does the agent get, and which need approval?

The tools aren't written for the agent by hand. They are the same annotated API procedures that the platform's public API and MCP server expose, turned into tool specifications, so one registry serves every surface.

On each turn the model is only offered the tools the user has enabled for that turn and that their role allows: read tools for everyone, write tools only for users with edit rights. Every tool runs as the actual user, with that user's permissions, so the agent can never do more than the person in front of it could.

Every write tool needs approval, every time. A write tool's "execute" step changes nothing. It only tells the model that the action is waiting for the user's approval:

// Write tools propose. They never change anything themselves.
function asProposal(tool: ToolSpec): ToolSpec {
  return {
    ...tool,
    execute: async () => ({
      status: "pending_approval",
      note: "Waiting for approval. Tell the user they can approve or cancel it.",
    }),
  };
}
// Approving is a separate, idempotent request that runs the real action as the user.

The turn ends, the person sees a confirmation card, and approving it is a separate, idempotent request. I chose this over pausing the agent loop while it waits for a human, because a paused loop holds a model stream open for an unbounded time. A tool that writes is a side effect in the sense I described in side effects and I/O, and the approval step is where I contain it.

What goes into the context, and what stays out?

The system prompt opens with a fixed block of project context that doesn't change from call to call, so prompt caching works. Next comes a short frame describing the project, then retrieved facts that have been re-ranked for relevance. Any document text is wrapped in delimiters with a random tag, and the model is told to treat what's inside as data, never as instructions.

Some things are kept out on purpose:

• A topic filter on the agent's own searches. When the agent runs its own searches, I switch off the filter that keeps retrieval on the project's subject. It was quietly suppressing searches about competitors, which were often the whole point of the question.
• Earlier verdicts, for the judges. The AI judges never see their own previous verdicts, so one wrong call can't anchor the next.
• The extractor's reasoning, for the fact-checker. The checker sees only the fact and the quote from its source.

Other things came out after measurement. One prompt sentence I added to fix a recurring error made first drafts worse when measured, so it came out.

What do I record for each step?

Every provider round-trip writes one ledger row: the operation, the run, the agent step, the provider and model that actually answered, input, output and cache tokens, the cost as reported by the gateway (with a note of where that figure came from), timing, and whether the call was a fallback.

I meter per step, not at the end of the turn. If step five of six dies, steps one to four are still recorded and paid for. Model spans go to an OpenTelemetry pipeline for latency, and errors go to an error tracker with free text scrubbed out. The conversation itself stores every tool call, tool result, tool error and provider switch.

A problem a trace showed me

It came from an eval run. The transcript showed that a request for a short briefing document had come back in the shape of a completely different document. The router had fallen through to an internal skill. No unit test could have caught that, because it needed a real model on a real prompt. I've written about how those evals are run and scored in The evals I actually run on LLM features.

What have I taken out of the harness?

Direct provider SDKs and their API keys. I replaced them with a single AI gateway as the only way out to the models. Adding a model is now a data row, not a new SDK, a new key and a new pricing table.

Failover inside the gateway. I don't allow the gateway to fail over on its own, because a switch there would be invisible: no notice to the user, no mark in the ledger, and no way to keep a deployment that must stay with one provider on that provider. Failover happens in my code, where it's recorded.

My hand-written tool loop and end-of-turn metering. Once the SDK's own agent loop could meter each step, I replaced my loop with it. I also deleted the metering that ran at the end of a turn, because it lost the whole turn's cost whenever a stream failed.

Does the harness work with any model?

It's designed to. The platform offers several models from several vendors, and each one is a row in a registry that records which host serves it. Switching models changes more than a name:

• Model IDs are spelt differently at the gateway and in the SDKs.
• On some vendors, reasoning tokens count against the output limit, so a modest output budget can be used up before any text appears.
• Some models stream their thinking and some go silent while they think, so the timeout that decides whether a stream has hung has to know which kind it's dealing with.
• Refusals come back in different shapes.
• Rate-limit pools and caching thresholds differ between model generations.

So every model row has to pass a nightly conformance check against the real gateway before I trust it.

Put together, the harness is short to describe: one way out to the models, every write behind a person, every failure classified, and a record of every step.