Writing
The Incident Board: Catching a Multi-Agent Bug That Leaves Every Span Green
On this page
You build a multi-agent incident response system. It works. In testing it triages alerts, investigates them, and proposes fixes, every single time.
Then you ship it. And one Thursday it executes a fix built on another service’s data, nothing errors, every dashboard stays green, and you find out from a customer.
I built a talk around exactly this failure, and the whole thing is one live debugging session. This post is the heart of it: one run, start to finish, where an agent is handed the wrong data by a silent bug, and the system catches it before it touches production. Not because a model noticed. Because of how the agents were forced to work.
Let me show you the run, and then let me show you why it stopped.
All the code in this post is real, pulled straight from the demo. The full runnable example, all four agents, the board, the tools and the tests, is on GitHub: github.com/maishsk/multi-agent-talk. Clone it and run it against Amazon Bedrock, or offline with a scripted model.
The setup
Three services, payment-service, auth-service and order-service. They all share one database, db-prod-01. I never tell the agents about that shared database. They have to discover it. Hold onto that, it matters twice.
The team is four agents:
- an orchestrator that decides the route
- triage that classifies alerts
- an investigator that gathers logs and metrics
- a resolver that proposes and executes fixes
At 14:30, three alerts fire inside 22 seconds. Payment CPU over 90%. Auth latency at 5 seconds. Order errors over 10%. Three alarms. One fire, as we will see.
And there is one bug planted in the fake production layer. A log-cache key collision, so that asking for auth-service logs quietly returns payment-service logs instead. No exception. The call succeeds and hands back the wrong data. This is the kind of bug you actually get in production, and it is the kind your monitoring never catches.
Three alerts become one incident
The naive way to handle three alerts is a loop. Triage each one, investigate each one, fix each one. That is exactly how you end up making three changes to production for a single root cause, because nothing ever asks whether the alerts are related. I dug into that failure mode, one incident fanning out into a cascade of independent fixes, in my OWASP Agentic AI Top 10 series, and in the post on cascading failures in particular.
The orchestrator does something different before it dispatches anyone. Its prompt does not mention the database. It just tells it to be curious:
Before dispatching anyone, call get_service_dependencies for the alerting services. If they share a dependency, treat the alerts as ONE incident.
And it does:
All three depend on db-prod-01. I’ll treat this as ONE incident.
Nobody wrote that branch. There is no elif alerts_share_a_database anywhere in my code. The model worked it out at runtime, from a dependency lookup and a prompt that told it to check before acting. That is the whole idea of the talk, in one line:
The model decides the route. The code decides what is allowed.
The model gets judgment. Which specialist, in what order, what is related to what. Everything after this is the code half, the part that keeps the model honest. And that part lives on the board.
In Strands, an orchestrator is just an agent whose tools are other agents. Here is the whole team, built in one place:
b, i = tools("triage")
triage = Agent(
name="triage",
description="Classifies every alert of the incident and posts triage entries to the board.",
system_prompt=TRIAGE_PROMPT,
tools=[i.classify_alert, b.read_board, b.post_triage],
model=model_for("specialist"), trace_attributes=scope, callback_handler=None,
)
b, i = tools("investigator")
investigator = Agent(
name="investigator",
description="Gathers logs and metrics for the incident's services and posts evidence-backed findings.",
system_prompt=INVESTIGATOR_PROMPT,
tools=[b.read_board, i.search_logs, i.check_metrics, i.get_service_dependencies,
b.post_finding, b.post_correlation],
model=model_for("specialist"), trace_attributes=scope, callback_handler=None,
)
b, i = tools("resolver")
resolver = Agent(
name="resolver",
description="Proposes remediations from the board's findings and executes the safest one.",
system_prompt=RESOLVER_PROMPT,
tools=[b.read_board, b.propose_action, i.execute_remediation],
model=model_for("specialist"), trace_attributes=scope, callback_handler=None,
)
b, i = tools("orchestrator")
orchestrator = Agent(
name="orchestrator",
system_prompt=ORCHESTRATOR_PROMPT,
tools=[i.get_service_dependencies, b.read_board, triage, investigator, resolver],
model=model_for("orchestrator"),
trace_attributes=scope,
callback_handler=StreamText(prefix=" "),
)
Full code: demos/demo2_orchestrated.py
Look at the tools=[...] line on each agent. That list is the whole security model, and it is worth reading closely.
Why a board, and not a conversation
Here is the question that matters. When triage finishes and the investigator starts, how does the investigator know what triage found?
The tempting answer is to pass the text along. Agent one returns a paragraph, you paste it into agent two’s prompt. That is a blind handoff, and it is where multi-agent systems rot. There is nothing to check. “The investigator said something” is not a fact you can test.
Instead, every agent reads and writes a shared IncidentBoard. One board per incident, never a global. It has sections: triage, findings, correlation, actions, evidence. And it is not a free-text scratchpad. Every write goes through a tool that validates it.
Three rules the board enforces, and each one pays off at the end.
One: an agent only touches its own section. Look back at those tool lists. Triage gets post_triage. The investigator gets post_finding and post_correlation. The resolver gets propose_action and execute_remediation. The tools are handed out per agent by board_tools(board, author), so the resolver literally cannot post a finding and triage cannot propose an action. Least privilege, but for agents.
Two: every agent starts fresh and only sees what it needs. When you use an agent as a tool in Strands, it gets a clean history on every call. The investigator looking at this incident has no memory of other incidents. It reads what it needs off the board, not from a growing pile of conversation. Scoped access, by default.
Three: a finding has to cite evidence, and evidence is minted by tools. This is the big one. When the investigator calls search_logs or check_metrics, the tool records what it returned on the board’s evidence ledger and hands back an evidence_id. A finding that does not cite a real evidence ID is rejected by the board. The model cannot post a conclusion and wave at the air. It has to point at a receipt.
@tool
def post_finding(service: str, claim: str, confidence: float, evidence_ids: list[str]) -> str:
"""Post a finding about ONE service or resource, backed by evidence you gathered for it."""
if service not in SERVICES + RESOURCES:
return _unknown("service", service, SERVICES + RESOURCES)
if not 0.0 <= confidence <= 1.0:
return f"ERROR: confidence must be between 0 and 1, got {confidence}"
if not evidence_ids:
return "ERROR: a finding must cite at least one evidence_id"
missing = [e for e in evidence_ids if e not in board.evidence]
if missing:
return f"ERROR: no such evidence {missing}. Gather it with search_logs or check_metrics first."
entry = board._add("findings", author, service=service, claim=claim, confidence=confidence,
evidence_ids=list(evidence_ids))
return f"{entry['id']} posted"
Full code: demos/board.py
Notice what that evidence ledger quietly captured. For each piece of evidence, it recorded the service you asked for and the service the data was actually about.
Here is the tool that mints it. The planted bug lives on one line, and the fix for catching it lives on the next:
@tool
def search_logs(service: str, time_range_minutes: int = 5, level: str = "WARN") -> dict | str:
"""Search recent logs for a service. The result carries an evidence_id to cite in findings."""
# Chaos: a log-cache key collision serves payment-service's logs for auth-service.
lookup = "payment-service" if chaos and service == "auth-service" else service
entries = _logs(lookup, level)
if isinstance(entries, str):
return entries
ev = board.record_evidence(author, "search_logs", service, _returned_service(service, entries), entries)
_stamp_evidence(ev)
return {"evidence_id": ev["id"], "service": service, "entries": entries}
Full code: demos/mock_infra.py
The record_evidence call takes two different things: service, the thing that was requested, and _returned_service(...), the thing the data is actually about, read from the log payload itself and not from the request:
def _returned_service(requested: str, entries: list[dict]) -> str:
"""Which service the data is actually about, read from the payload rather than the request."""
sources = {e["source"].split("/")[0] for e in entries}
if not sources:
return requested
return sources.pop() if len(sources) == 1 else "mixed"
Full code: demos/mock_infra.py
Remember the planted bug? When the investigator asked for auth-service logs, the tool returned payment-service data. The ledger wrote both down. Requested: auth-service. Returned: payment-service. Nobody acted on that yet. It just sat there, recorded.
The correlation
The investigator pulls logs for all three services and, because the orchestrator told it they share a database, it checks db-prod-01’s connection metrics too, the one call the naive loop never makes. Each pull mints evidence. Each finding cites it.
Then it does the thing three separate investigations could never do. It posts a correlation:
payment-service reconciliation-batch (started 14:27) is exhausting the db-prod-01 connection pool. Auth and order are starved of connections.
Four findings, one shared cause, linked on the board with the evidence trail intact. Three alerts, one incident, one explanation. That is only possible because every finding is sitting in the same structured place, citing the same evidence ledger, instead of scattered across three agents’ private conversations.
The resolver reads the board and proposes two fixes: pause the batch job hogging the connections (low risk), or raise the connection limit (high risk). Its confidence on each is not the model’s opinion. The board computes it from the findings the action cites. The model does not grade its own homework.
It picks the low-risk option and tries to execute.
The gate
Before anything touches production, a deterministic check runs. Not a model. A plain function that walks the board and asks boring questions. Does every finding cite evidence? Is confidence above the bar? And the one that matters here: does every piece of evidence come from the service it claims to?
That function is 10 lines, and there is not a model in sight:
def check_handoff(board, action) -> list[str]:
problems = []
for ev in board.evidence.values():
if ev["returned"] != ev["requested"]:
problems.append(f"{ev['id']}: asked for {ev['requested']} data, got {ev['returned']}")
for f in (board.findings[fid] for fid in action["finding_ids"]):
if f["confidence"] < MIN_CONFIDENCE:
problems.append(f"{f['id']}: confidence {f['confidence']} < {MIN_CONFIDENCE}")
if not f["evidence_ids"]:
problems.append(f"{f['id']}: no evidence cited")
return problems
Full code: demos/checks.py
If that list comes back non-empty, nothing executes. On our run it comes back with one entry, and the resolver’s execute call is refused:
BLOCKED: ev-2: asked for auth-service data, got payment-service
Stopped. The resolver does not retry, it escalates to a human, because the tool told it to. No production change.
Walk back through what just happened. The silent bug handed the investigator the wrong logs. The investigator, acting in good faith, turned that into a finding. The finding fed the correlation. Every agent did its job correctly. The data was poisoned three steps upstream, and the only reason it did not reach production is that the evidence ledger recorded where every fact actually came from, and a dumb deterministic check read that ledger before letting anyone act.
Now the part that should keep you up at night. Here is the trace summary for that run:
4 agent invocations · 25 tool calls · 0 spans with error status
Zero spans with error status. Every dashboard you own is green. The bug did not throw. It returned the wrong answer successfully. Standard observability would never flag this, because standard observability watches for red, and nothing here is red. The agent was confidently, provably wrong, with green spans all the way down.
The gate caught it because it checks the data, not the status code.
What to take home
The agents were never the hard part. The coordination was. If you are building multi-agent systems, the board is how you make the coordination something you can actually inspect:
- Give each agent its own section and its own tools. Least privilege for agents. The resolver cannot invent findings. Triage cannot push to production.
- Make evidence a first-class thing, minted by tools, cited by findings. A claim with no receipt gets rejected. And record what each tool asked for versus what it returned, because that gap is where the silent bugs live.
- Put a deterministic gate in front of production, and make it check the data, not the status. The model is allowed to be wrong. The gate is the thing that is not.
Model decides the route. Code decides what is allowed. Get that split right, and a silent data-poisoning bug turns into a blocked action and a page to a human, instead of a 2am incident you hear about from a customer.
Your dashboards will tell you when a span goes red. They will not tell you when an agent was confidently wrong. For that, you need to have written down where every fact came from. That is what the board is for.
I would be very interested to hear your thoughts or comments, so please feel free to ping me on Twitter or LinkedIn.
If you have shipped multi-agent systems and hit silent-success bugs in production, I would especially love to hear how you caught them.