One Retrieval Wasn't Enough: Rebuilding Ordinance as an Agent

One retrieval is a bet you place before the model has thought about anything. On a simple lookup that bet usually pays off; on a real regulatory question, “is this fire door still approved, and what does the clause it points to actually say?”, I found it tended to lose quietly.

A few months ago I wrote about building Ordinance, a RAG system that answers questions about Hong Kong building regulations across 224 government PDFs and 5,700+ chunks. Hybrid search with pgvector and PostgreSQL FTS, RRF fusion, Cohere reranking, GPT-4o generation, citation verification, faithfulness scoring. For what it is, a one-shot question-answering machine, it served me well.

For my use case, though, one-shot turned out to be a ceiling. After watching real queries hit that ceiling, I rebuilt the core as an agent. This is the story of what broke for me, the reframe that helped, and the honest tradeoffs. It is one path through a specific problem, not a recipe that will fit every project.

Where static RAG breaks

The naive RAG loop is: embed the query, retrieve once, generate, done. That single retrieval decision is made before the model has thought about anything. In my experience with regulatory questions, three failure modes kept showing up:

• It can’t decide when to go live. Most answers are in the indexed PDFs. But “is this fire door model still on the approved list?” needs fresh data from data.gov.hk. The static pipeline fired that live lookup in parallel on every query, wasteful when freshness doesn’t matter, with no way to decide for itself when it actually does.
• It can’t cross-reference. A clause that says “subject to the requirements in Part IV” requires a second retrieval that depends on the result of the first. One-shot retrieval has no second hop.
• It can’t hold project context. “Does my 12-storey residential building meet the means-of-escape rules?” should remember the building type across follow-ups. Stuffing every prior turn back into the prompt is how you rot the context window.

For me these weren’t retrieval-quality problems. They read more like control-flow problems. The system needs to decide what to do next, and static RAG, at least as I had built it, has no place to put a decision.

These aren’t retrieval-quality problems. They’re control-flow problems.

The split is easiest to see side by side:

The reframe: the model is one input

Here’s the thesis I came around to, offered as a working belief rather than a law: a decent model with a great harness can beat a great model with a bad harness. (I unpack that idea on its own terms in Anatomy of an Agent Harness.) The way I think about it now, the model is a single component, a function from context to next action. A lot of what makes the system useful lives around it: the tools you expose, the policy for what goes into context, the memory, the recovery paths when a tool fails, the subagents you fan work out to.

For a while I had been spending most of my effort on the model call. For this problem, the leverage turned out to be in the harness.

The agent loop

The harness is built on a Thought-Action-Observation cycle, the ReAct pattern. Instead of one retrieval baked in upfront, the model reasons about what it needs, picks a tool, observes the result, and decides whether it’s done or needs another step.

flowchart TB
  Q[User Query] --> CTX[Assemble Context
system + tools + memory
+ scratchpad]
  CTX --> LLM[Call LLM
Thought + Action]
  LLM --> PARSE{Parse Action}
  PARSE -->|tool call| EXEC[Execute Tool]
  PARSE -->|final answer| VERIFY[Citation Verify
+ Faithfulness Score]
  EXEC --> OBS[Observation]
  OBS --> APPEND[Append to Scratchpad
update todo list]
  APPEND --> BUDGET{Step Budget
remaining?}
  BUDGET -->|yes| CTX
  BUDGET -->|no| VERIFY
  VERIFY --> RES[Response with
citations + scores]

The model emits a thought and an action. The harness parses the action, runs the tool, captures the observation, and loops. Two guardrails earned their place for me: a step budget so a confused agent can’t loop forever, and the existing verification layer from the RAG post (citation checking and faithfulness scoring) now running as the exit gate of the loop rather than as a fixed final stage. The agent can take a different path to the answer, but it can’t skip the part that makes the answer trustworthy.

The tools

The agent’s capability is its toolset, and the temptation is to expose everything. I tried to resist it. In my experience, a handful of focused tools tends to work better than a sprawl of overlapping ones. Overlapping tools force the model to disambiguate between near-identical options, and that’s where I saw it make bad calls. Ordinance’s agent gets four:

• retrieve: the existing hybrid search pipeline (vector + FTS, RRF, Cohere rerank), now callable on demand with a query the model composes, possibly multiple times.
• gov_lookup: live queries against data.gov.hk datasets for fire door ratings, glazing specs, and MiC approvals. The agent decides when freshness matters.
• resolve_reference: takes a cross-reference like “Part IV, Clause 4.3.2” extracted from a retrieved chunk and fetches that exact clause. This is the second hop static RAG couldn’t make.
• verify_citation: the same citation-matching logic from the RAG pipeline, now exposed as a tool the agent can call mid-flight, so the model can self-correct before it commits to a phantom citation.

Each tool has a tight schema (validated with Zod, like the rest of the stack) and a one-line description of when to use it, not just what it does. For me, the “when” was what the model actually needed; your mileage may vary depending on how distinct your tools already are.

Context engineering: fighting context rot

The failure mode that bit me hardest in an agent loop was context rot. Every step appends an observation, and retrieved regulatory chunks are long. Left unchecked, by step five the window filled up with stale tool output and the model lost the plot.

Three tactics helped me keep the working context lean:

Progressive disclosure. The model doesn’t get all 224 documents’ metadata upfront. It gets a compact index and pulls detail only when a step requires it. Tool results are summarized back into the scratchpad rather than pasted in full, so the agent keeps a pointer to the chunk, not the whole chunk.

Externalized memory. Instead of relying on the context window to remember the plan, the agent maintains a todo/scratchpad artifact outside the prompt. Each step it reads the current todo state, acts, and writes back. The working context carries the current objective and recent observations; everything else lives in the artifact.

Project context as durable memory. The building type, storey count, and use class from a session get pinned to a small structured memory object, not replayed as raw chat history. Follow-up questions read from it.

flowchart LR
  subgraph WC[Lean Working Context]
      SYS[System + Tool Specs]
      TODO[Current Objective]
      REC[Recent Observations]
  end
  subgraph EXT[Externalized State]
      SCRATCH[(Scratchpad /
Todo Artifact)]
      MEM[(Project Memory
building type, use class)]
      CHUNKS[(Chunk Pointers
not full text)]
  end
  SCRATCH -->|read current step| TODO
  TODO -->|write progress| SCRATCH
  MEM -->|pin context| WC
  CHUNKS -->|fetch on demand| REC

Subagents for parallel cross-document work

Some questions span documents that are independent of each other. “Compare the means-of-escape requirements across residential, commercial, and industrial occupancies” is really three lookups that don’t depend on one another. Running them sequentially in one context window is slow and pollutes the window with three documents’ worth of detail.

So the lead agent spawns subagents, one per occupancy, each with its own clean context and the retrieve and resolve_reference tools. Each returns a compact, cited summary. The lead agent synthesizes. The parent’s context never sees the raw chunks, only the distilled findings, which in my testing kept the synthesis step sharper. This added real complexity, so I would only reach for it once a problem clearly fans out the way this one does.

Honest tradeoffs

This is not free, and an agent is usually the wrong call. I want to be honest that for most of my own queries, going agentic would have been overkill.

Latency. The one-shot RAG path returned in roughly eight seconds (and ~15ms cached). An agentic query that takes three or four tool-calling steps is meaningfully slower: multiple model round-trips plus tool execution. For a simple factual lookup, that’s pure overhead. These figures are illustrative of the shape, not a benchmark; the point is the loop multiplies the cost of a single pass.

Cost. Every loop step is a model call, and reasoning tokens add up. Subagents multiply this further. The agent path can cost several times a single RAG query.

When an agent is overkill. If the answer is in one document and needs one retrieval, which was most of my queries, plain RAG is faster, cheaper, and just as correct. So Ordinance routes: a lightweight classifier decides whether a query is single-hop (straight to the RAG pipeline) or multi-hop, freshness-sensitive, or context-dependent (into the agent loop). The agent is the exception path, not the default, and I would gently push back on anyone reaching for an agent before they have hit a wall with something simpler.

That routing is the part I would carry to the next project. Going agentic didn’t mean throwing away the RAG pipeline. It meant wrapping it in a harness that knows when not to use the fancy machinery. The model got no smarter. The system around it did.

The live system is at ordinance.maniksoin.com.