Multi-Agent Systems: The Patterns That Actually Ship

Every agent demo looks impressive. Very few survive contact with production. After shipping multi-agent systems for clients at XAI.ma and building the multi-agent paper studio behind Waraqa, I have a pretty firm view of which patterns are real and which are theater. This is the field guide I wish someone had handed me.

The honest starting point: most work is a single agent

Before you architect a swarm, check whether the problem is actually a single well-prompted agent with good tool use. A huge fraction of "multi-agent" systems are one agent doing several steps — and forcing a second agent in adds latency, cost, and nondeterminism for no benefit. The rule of thumb: add an agent when a different role, knowledge, or permission boundary genuinely exists, not when the marketing would look better.

The patterns that ship

Orchestrator–workers

One orchestrator decomposes the task, dispatches to specialized workers, and assembles the result. This is the workhorse — it maps onto most real business processes, it is easy to reason about, and it degrades gracefully.

class Orchestrator:
    def run(self, task):
        plan = self.planner.plan(task)           # break task into subtasks
        results = [
            self.pool.submit(w, s)               # dispatch to specialist workers
            for s, w in plan.specialists
        ]
        return self.assembler.join(plan, results)

The orchestrator–workers pattern ships because the orchestration logic is deterministic. The LLM decides what to do; a fixed state machine decides who does it and in what order. When you keep the flow deterministic and let models fill the leaves, debugging becomes tractable.

Supervisor

A supervisor agent watches workers, decides when to escalate, and can hand off between specialists. This pattern earns its complexity when you have genuinely open-ended work — a support triage flow, for example, where the first agent must decide whether the problem is billing, technical, or product. The supervisor is the risk layer: it needs the strongest evaluation, because a wrong escalation is a failed interaction.

Pipeline

The pipeline (agent one → agent two → agent three) is the seductive one and the one to be careful with. It's fast and legible, but errors compound: a mistake early in the chain is amplified downstream, and the agents cannot recover or revisit. Pipelines ship fine when each stage has a validation gate — check, and only then pass forward. Without the gates, you've built a compounding-error machine.

Mesh: the pattern for demos

Full two-way meshes — every agent talks to every agent — are where demos live and products go to die. The combinatorial explosion of interactions makes them unobservable and untestable. I have never needed a true mesh in production. If your design requires one, you are probably missing an orchestrator.

Tools, not vibes

Agents are only as useful as the tools they hold. The tool layer is where the real engineering budget belongs:

• Every tool has a schema and a contract. The agent calls a function; the function returns structured data. No free text back-channels.
• Tools enforce constraints. A search tool returns results with sources; a write tool validates before committing. The tool is the enforcement point for safety, not the prompt.
• Failures are first-class. Tools return errors as data, and agents are trained (and tested) on what to do when a tool fails.

Tool-calling is also the part that benefits from standardization. MCP (Model Context Protocol) gives you a uniform way to expose tools and context to different agents and models — the protocol layer that turns a bespoke integration into a repeatable platform.

Observability is not optional

The moment you have two LLM calls collaborating, you have a distributed system — and distributed systems without tracing are untraceable. Every agent run in production gets a trace: the plan, each tool call with inputs and outputs, token usage, latency per hop, and the final assembly. When a customer says "it gave a wrong answer," the trace is the first thing you open.

Logging every step is also the cheapest evaluation harness you'll ever build. Traces from production become the test set for the next version — and they catch regressions no synthetic benchmark will.

The evaluation habit

Evaluate the system, not the agents. A single agent can be excellent while the system fails on routing, tool contracts, or assembly. I evaluate on end-to-end outcomes with a labeled golden set, and I hold the pipeline's output format constant so regressions are visible. The discipline of keeping the output contract stable is what lets you iterate on internals without breaking consumers.

Cost and latency discipline

Agents multiply cost and latency the way they multiply complexity, and the compounding is easy to miss. A single request that fans out into three agents and fifteen tool calls can consume an order of magnitude more tokens than a monolithic answer — and take five seconds instead of one. Production agent systems need an explicit budget the way any other distributed system does.

Three practices that keep the bill honest:

• Route before you call. A cheap classifier that decides "this needs the swarm or a single agent" saves the swarm for the work that actually needs it. Most requests should not spawn a swarm.
• Cap the fan-out. Set a hard limit on parallel tool calls and retries, and make the orchestration degrade: if the plan needs twenty steps, run the first five and surface partial work rather than burning the budget silently.
• Trace the spend. Token usage and latency per hop are first-class metrics in every production trace. If a trace can't tell you which hop cost $1, the system is ungovernable.

The uncomfortable truth: the highest-ROI change to most agent systems is fewer agents. Removing a hop cuts cost, latency, and failure surface at the same time — the rare optimization that wins on every axis.

When not to build agents

If the task is deterministic, script it. If the task is a single well-scoped Q&A, a RAG pipeline will beat an agent on latency, cost, and predictability. Agents earn their complexity on tasks that are open-ended and multi-step — research, triage, document production — where a fixed pipeline would need a different branch for every real-world case.

That is the honest takeaway: agents are a tool for managing irreducible complexity, not a way to make simple things fashionable. Build the orchestrator, gate the pipeline, trace everything, and evaluate end-to-end. If you're doing the same, let's compare notes.