The Model Is Rarely the Bottleneck: Anatomy of an Agent Harness

A lot of the debate about AI agents seems to center on the model: which one is smartest, which benchmark moved, which lab is ahead this quarter. That matters, but in my experience, once you ship an agent to production the lesson lands fast: the model is just one input into a running system. Everything else is the harness, the system prompt, the tools, the context policies, the hooks, the sandboxes, the subagents, the feedback and recovery loops. I’ve come to think of that as the quiet category error, and the harness is where, more often than not, agents live or die.

It’s a view I’ve slowly come around to:

A decent model with a great harness tends to beat a great model with a bad harness.

The bottleneck isn’t the model

Here’s a figure that reframed things for me: by widely cited industry estimates, the great majority of AI agent projects (on the order of 80 to 90%) never reach production. And anecdotally that gap hasn’t obviously closed as the models got better. We went through several generational leaps in raw capability and the production rate still looks stubbornly low. If the model were the only bottleneck, you’d expect better models to have moved that number more than they seem to have.

To me that suggests the bottleneck often sits somewhere else: in the runtime orchestration layer wrapping the reasoning loop. That layer is the harness, and harness engineering is the discipline that seems to be emerging in 2026 to take it seriously.

The reframe that helped me: try not to treat the LLM as the agent. The LLM is a stateless function that maps (prompt) -> (text). The agent is the loop, the state, the tools, and the policies that decide what goes into that function and what happens to what comes out. That’s all harness.

flowchart TB
  subgraph HARNESS[The Harness, Runtime Orchestration]
      SP[System Prompt
identity + policies]
      TOOLS[Tool Registry
name + schema + description]
      CTX[Context Policy
what goes in, what gets pruned]
      SUB[Subagents
delegated sub-tasks]
      HOOKS[Hooks + Sandboxes
guardrails + recovery]
  end
  SP --> LOOP
  TOOLS --> LOOP
  CTX --> LOOP
  SUB --> LOOP
  HOOKS --> LOOP
  LOOP[Reasoning Loop] --> MODEL[(LLM
one input among many)]
  MODEL --> LOOP

The model sits inside the harness, not above it. It’s a component you can swap. The harness is the part I’ve found you spend most of your engineering effort on.

The core loop: Thought-Action-Observation

Strip away the vocabulary and most agents I’ve worked on run the same loop. Call it TAO, call it ReAct, it tends to be the same cycle:

flowchart LR
  START[Goal] --> ASM[Assemble Prompt
system + history + context]
  ASM --> CALL[Call LLM]
  CALL --> PARSE[Parse Output
thought + tool calls]
  PARSE --> DONE{Done?}
  DONE -->|Yes| RESULT[Final Answer]
  DONE -->|No| EXEC[Execute Tool Calls]
  EXEC --> OBS[Observe Results]
  OBS --> ASM

1. Assemble the prompt: system instructions, conversation history, retrieved context, tool definitions.
2. Call the LLM.
3. Parse the output into a thought and zero or more tool calls.
4. Execute the tool calls against real systems.
5. Feed the observations back into the next prompt.
6. Repeat until the model decides the goal is met.

Simple to draw, hard to get right. Every iteration the harness makes decisions: what to include, what to drop, which tools to expose, what to do when a tool errors. Those decisions compound. A loop that runs 20 turns has made hundreds of small harness choices before it produces an answer.

The four pillars

When I design a harness, I tend to think about four things. In my experience, get these right and a lot of the rest follows.

System prompt. The agent’s constitution: identity, operating principles, output contracts, escalation rules. This is where you encode the policies that don’t belong in code because the model needs to reason about them.

Tools. The agent’s hands. What it can actually do in the world.

Context. What the model sees on each turn. This is the pillar I’ve found hardest, and the one that has quietly broken the most agents I’ve built.

Subagents. Delegation. When a task has a self-contained sub-goal, you spin up a fresh agent with its own clean context and a narrow toolset, let it finish, and fold the result back. It’s how you keep the main loop’s context from drowning.

Hooks, sandboxes, and recovery paths matter too, but I tend to treat them as the guardrail layer wrapping these four, not a fifth pillar. Get the four right and they’re what you’re guarding.

Tool design is prompt design

Here’s something that’s easy to miss: every tool’s name, description, and JSON schema gets stamped into the prompt on every single request. Tools aren’t free function calls sitting off to the side; they’re tokens the model reads every turn, and they shape how it reasons.

This is why, in my experience, ten focused tools beat fifty overlapping ones. With ten clean tools, the model can hold the entire menu in its head and tends to pick correctly. With fifty, you start to see tool confusion: search_documents versus find_files versus query_index, and the model coin-flips. The menu is part of the prompt, so a bloated menu is a bloated prompt.

I’ve found it helps to treat tool design as prompt engineering. Name tools the way you’d name them for a new engineer. Write descriptions that say when not to use them. Make the schemas tight so invalid calls are hard to represent.

The two failure modes that actually kill agents

After a fair number of production incidents, the failure modes I keep running into sort themselves into a rough ranking.

Context rot is the one I see most. As the loop runs, context accumulates: old observations, stale tool outputs, dead ends the agent explored and abandoned, half-relevant retrieved chunks. None of it gets cleaned up by default, and it all tends to degrade the next generation. The model starts attending to noise. Output quality drops turn over turn, even though nothing “errored.”

The mental model that has helped me most: treat context rot as a sensor problem. Your agent perceives the world through its context window. If the window fills with stale and noisy signal, the agent is effectively perceiving a degraded version of reality. You wouldn’t run a robot off a fogged-up camera; I’d avoid running an agent off a fogged-up context.

State loss is the second one I hit often. The agent forgets what it already did, redoes work, contradicts an earlier decision, or loses the plot on a long task. The context window is finite and the loop is long, so eventually something important scrolls out of view.

Both failure modes are quiet, nothing errors, so I’ve found it helps to name them precisely:

Mitigations: progressive disclosure and externalized memory

Two techniques have done most of the heavy lifting for me against both failure modes.

Progressive disclosure. Don’t expose everything up front. For each sub-task, surface the minimum set of tools and telemetry the agent needs right now. Drilling into logs? Reveal the log tools then, not before. This keeps the per-turn prompt lean and directly fights context rot: less surface area, less noise, fewer ways to get confused.

Externalize memory to the filesystem. Instead of trying to cram all state into the context window, push it out to standardized artifacts the agent reads and writes: an AGENTS.md describing the environment and conventions, a todo.md tracking the plan and what’s done. The context window becomes working memory, scoped to the current turn; the filesystem becomes long-term memory, durable across the whole run. The agent re-reads its todo.md instead of trying to remember a 40-step plan from 30 turns ago. That’s the state-loss fix.

flowchart TB
  LOOP[Reasoning Loop] -->|reads minimal slice| CTX[Context Window
working memory]
  CTX -->|prune stale signal| CTX
  LOOP -->|write plan + progress| FS[(Filesystem
AGENTS.md / todo.md)]
  FS -->|re-read on demand| LOOP
  LOOP -->|progressive disclosure| TOOLS[Per-task Tool Subset]

The MCP angle

One reason harness engineering seems to be crystallizing as a discipline now is the Model Context Protocol (MCP). MCP standardizes how agents talk to tools; it draws a clean line between tool implementation and agent logic. Your agent doesn’t need to know how a tool is built; it speaks a common protocol and the tool answers. That separation is what lets you treat the tool registry as a first-class, swappable layer of the harness instead of bespoke glue welded into the loop.

Where this connects to my own work

I wrote earlier about building Ordinance, my RAG system for Hong Kong building regulations. That post was about retrieval: hybrid search, RRF fusion, citation verification. But stepping back, the questions that turned out to matter most for me were harness questions: which retrieval tools to expose per query type, how to keep multi-hop regulatory reasoning from rotting its own context, when to spin up a subagent to chase a cross-reference, how to externalize the audit trail as durable state. The retrieval was, for me, the more tractable half. The harness was the next half, which is exactly what I built next, written up in From RAG to Agent.

That’s the through-line I keep coming back to. The model keeps getting better and that’s great, but in my experience it was rarely the binding constraint. More often the constraint is everything around it, and that everything has a name now. Others may well see this differently, but I’ve found it pays to build the harness like it’s the product, because to your users, it largely is.