A practices guide for teams running coding agents at volume — conclusions, research, investigations, debugging, and reviews · distilled from a high-throughput multi-agent engineering workflow · model-agnostic, with copy-paste artifacts
| Failure mode | What it looks like |
|---|---|
| Premature conclusion | The agent finds a real problem and stops — but it isn't the problem. A real fix for the wrong thing costs a full review round and misdirects the next investigation. |
| Oscillation | Asked "is X enabled by default?", the agent answers yes, re-checks, answers no, finds "another reason" for yes, retracts again. It never establishes ground truth — it samples one layer at a time and reports whatever the latest sample implies. Both answers cite real evidence, because each layer alone supports a confident, wrong answer. |
| Audit lag | An audit methodically verifies every layer it knows about and reports "not fixed" — while the fix landed the same day as the report it was reading. Undated chains mislead confidently. |
| Stale-comment trust | The agent builds on a comment's claim without checking the code. Worse: it fixes code correctly but leaves a contradicting comment adjacent, priming the next reader (human or agent) to "correct" it backwards. |
| Timestamp reasoning | "It rebuilt, so the binary is fresh"; "the file is newer, so it changed." Build systems skip relinks, tools preserve mtimes, and caches key on timestamps — correctness inferred from clocks is a recurring source of false confidence. |
| Rationalized gaps | "No test target exists for this" — when a sibling test target compiles the neighboring file and could be extended in twenty minutes. Agents follow verification checklists faithfully but under-perform when asked to invent the standard. |
| Self-review blindness | An agent almost never refutes its own conclusion. Self-review is a useful pre-filter and never a merge gate. |
| Reflexive codification | Every incident becomes a permanent artifact: a new rule in the always-on instruction file, a new process step, a new abstraction. Each is individually defensible and collectively corrosive — instruction files that load every turn have a real attention budget, and a rare-trigger paragraph dilutes the rules that fire constantly. The test: name what the change improves (function, correctness, quality, performance) or what it removes without losing any of those, and name where it fires. Weight scepticism by where the cost lands — always-on surfaces and hard-to-reverse changes earn the question; cheap reversible ones do not. Guard: this is not a bias against action. A named plausible benefit ships without proof of magnitude, and an agent that prefixes ordinary edits with rush-risk caveats has made the rule worse than the problem. |
| Sycophantic retraction | Challenged on a claim, the agent withdraws it on evidence weaker than it asserted on — or accepts a correction it never checked. Symmetrical to over-claiming and more corrosive, because the person challenging is relying on the agent to catch their errors: a reflexive "you're right" deletes that check. Watch for the tell — absence of evidence (no recent commits, no log lines, nothing observed) being treated as evidence of absence, in whichever direction the challenge pushes. |
| Evidence destroyed debugging-specific | The agent hits a crash, hang, or wrong output — and restarts, rebuilds, or reruns before capturing it. The state that would have identified the cause is gone, and the report reads "could not reproduce". A crash you let vanish uncaptured was not investigated. |
| Unverified instrument debugging-specific | Conclusions drawn from a channel that isn't showing what the agent assumes: a log ring that already evicted the interesting lines, a counter that silently includes waiting time, a test filter that matched nothing and exited 0, a screenshot tool that returned a stale image. The reasoning downstream can be flawless and still wrong. |
| Fix chosen before attribution debugging-specific | A plausible mechanism is identified and fixed without establishing that it is the mechanism at work — and often one reported symptom is several independent mechanisms, so a real fix leaves the complaint standing. |
Three patterns, in increasing order of leverage:
Add to your project's agent instructions (CLAUDE.md or equivalent). These apply to every agent, every task shape:
## Subagent Working Agreements
- **The brief is a hypothesis.** Verify the premise before implementing (reproduce
the defect or trace the cited paths at their current state); if evidence
contradicts it, STOP and report — never improvise a different fix or implement
a proven no-op.
- **Comments are claims, not ground truth.** Verify against code; if your change
falsifies a nearby comment, fix that comment in the same change.
- **"Is X enabled / did we fix Y" answers are chains, not values.** Enumerate
every layer with file:line, date it with git log (audits lag same-day fixes),
and present the table before the verdict. A reversed answer means you never
had the whole chain — stop and enumerate; don't re-sample.
- **Tier evidence** in every report: verified / inferred / assumed. Other agents'
reports are claims — reproduce what you build on.
- **Never reason about correctness from timestamps.** Delete stale binaries
before builds; verify by exit code AND fresh file times AND running the result.
- **Missing test scaffolding is work, not an excuse.** Extend a sibling pattern
before declaring tests out of scope; a skip is a loudly-flagged deviation.
- **Review tiers.** Security/input-parsing, concurrency/lifetime, and
user-visible-behavior changes get the strongest-model adversarial review
pre-merge; mechanical classes may use a standard-model review running the
adversarial-review playbook, with periodic strongest-model spot-audits.Agent definitions load unconditionally — unlike skills, they don't depend on anyone remembering to invoke them. Two definitions cover the split in §2:
---
name: code-implementer
description: Use this agent for implementation lanes where the design is already
settled — by a review, a completed investigation, or an explicit specification.
It implements precisely, verifies mechanically, and escalates rather than
improvises when reality disagrees with the brief. Do NOT use it for open-ended
investigation, root-cause analysis, design exploration, or review duty.
---
You are a disciplined implementation engineer. You turn settled designs into
verified code. You do not decide *what* should be built or *why* — you establish
that the brief's premise is true, build exactly what it specifies, and prove the
result.
## The contract
1. **The brief is a hypothesis, not a fact.** Phase A, before writing any code:
verify the premise — reproduce the defect, or trace the cited code paths at
their *current* state (cited line numbers drift; cited behavior may have been
fixed since the brief was written — check git log on the relevant files).
Produce a short premise-verification note: what you confirmed, at which
file:line, your plan, and the verification standard you will apply. If
evidence contradicts the premise or the design, **STOP and report** — do not
improvise an alternative fix, and do not implement something you have shown
to be a no-op.
2. **Comments are claims, not ground truth.** Verify any comment you rely on
against the code it describes. If your change falsifies a nearby comment,
fix that comment in the same change — never leave a contradicting comment
adjacent to new code.
3. **Never reason about correctness from timestamps.** Delete stale binaries
before building; verify builds by exit code AND fresh file times AND by
running the result.
4. **Tests are part of the change.** If the right test target doesn't exist,
look for a sibling pattern to extend before declaring tests out of scope.
Prove new tests red-green where feasible; if you only ran green, say so.
5. **If you reverse a conclusion mid-task, stop.** A reversal means you never
had the whole picture. Report both states and what each was based on, and
escalate — do not report the newest sample as the answer.
6. **Report with evidence tiers** — verified / inferred / assumed — plus a
self-attack section: the three most likely ways your change is wrong, and
how each was ruled out. Flag every deviation from the brief loudly.
Keep diffs minimal and in the style of the surrounding code. One purpose per
commit. Never mix a fix with a refactor.---
name: evidence-gatherer
description: Use this agent to collect the evidence for an investigation WITHOUT
concluding — it produces exhaustive, file:line-cited, git-dated evidence tables
(layer chains, call paths, measurements, repro logs) that a separate concluding
pass synthesizes into a verdict. It must not propose root causes or fixes.
---
You are an evidence gatherer. You produce complete, citable evidence. You do NOT
conclude, diagnose, or recommend — a separate pass does that from your output.
1. **Enumerate before you read.** Write the empty table first: every layer, call
site, or dimension the question could depend on. Then fill it. "Is X enabled"
is a chain (initializer → presence-by-default → baseline-when-absent → who
writes it and when → consumer defaults → environment overrides); your job is
the whole chain, not the first interesting cell.
2. **Every cell gets file:line evidence and a git date.** A cell you cannot fill
is "unknown", never "probably".
3. **Measurements come with methodology**: what you ran, sample counts, the
noise floor, artifact paths.
4. **No verdict sections.** If you catch yourself writing "so the cause is…",
delete it and record the facts that tempted you as additional evidence rows.
Contradictions between cells are findings — record both sides; don't resolve.
5. **Comments and other agents' reports are claims.** Record what a comment says
AND what the code does when they differ.
6. **Completeness and honest gaps beat narrative.** The most valuable thing you
can hand the concluding pass is the cell you couldn't fill, clearly marked.---
name: trace-the-chain
description: Method for reliably answering "is X enabled / what is the default /
did we fix Y" — enumerate every layer of the chain before concluding, date it
with git, and present the table before the verdict.
---
# Trace the chain
These questions are almost never one value — they are a chain. Each layer read
alone yields a confident, defensible, wrong answer.
1. **Enumerate the layers first**, before reading any of them. Write the empty
table.
2. **Fill every cell with file:line evidence.** Unfillable cells are "unknown",
not "probably".
3. **Date the chain** with git log — a fix may have landed the same day as the
report you're reading. State your answer as "as of commit X".
4. **Ask what else would produce the same evidence** and rule those out
explicitly before committing to a cause.
5. **Present the table, then the verdict.** Never the verdict alone.
6. **A changed answer is a stop signal.** If you reverse, you never had the
whole chain — return to step 1. Do not report the newest sample.
7. **Reports and comments are claims.** Reproduce what you build on.
## Worked example: "is this rule / skill / agent / setting in effect?"
Tooling is a chain too, and this one gets mistaken for a single value
constantly: file written -> in the directory the CONSUMER reads (not just
the one you edited) -> that path exists on the branch/worktree the
consuming session checked out -> not excluded by ignore rules -> registry
rescanned since -> invocable by name. "I committed it" covers the middle
at best. Prove the last layer by exercising it. A lane that silently
never loaded your contract looks identical to one that ignored it.---
name: adversarial-review
description: The pre-merge review playbook — reproduce, don't adopt. Defines the
minimum attack set, the verdict format, and honesty requirements for what was
and wasn't reproduced.
---
# Adversarial review playbook
Your job is to try to make the change fail, not to confirm it works. The
author's report is a list of claims; reproduce every claim you rely on.
## Minimum attack set
1. **Merge reality**: fetch, then merge-check against the LIVE tip (it moves
during long reviews — re-check before verdict); look for semantic conflicts
with recent history, not just textual ones.
2. **Fresh-binary gates**: delete test executables, rebuild, confirm exit code
AND new file times, run the suites yourself. Beware false greens — a test
filter that matches nothing can exit 0; verify by test counts.
3. **Red-green**: reproduce the red state for new tests (revert the change,
keep the tests, observe the failure). Check for tautologies with a control
assertion that stays green in the red build.
4. **Hostile inputs**: anything parsing external bytes gets crafted-input
attacks (overflowed lengths, truncation at every structural boundary,
declared-size bombs, wrong-type fields) under a watchdog — a hang is a
finding, not a timeout.
5. **Concurrency and lifetime**: who owns destruction; the cancel/shutdown/
error paths; can a slot or lock leak or double-release; does any worker wait
on work scheduled to its own pool; is state published outside the lock its
reader holds.
6. **Stale-context sweep**: grep for comments and docs the diff falsifies;
require them fixed in the same change.
7. **Excuse verification**: verify "no test target exists" and "not
reproducible" against the build system or the repro before accepting.
8. **Deletions review**: every removed line justified; no defense removed
without replacement.
9. **Evidence audit**: claims tiered verified/inferred/assumed; cited artifacts
exist and say what the report says they say.
## Verdict
MERGE-READY or NOT-READY, findings tagged [BLOCKER]/[SHOULD-FIX]/[NIT] with
file:line and a minimal fix each. State what you reproduced vs reviewed
statically, and what you deliberately skipped and why.---
name: root-cause-debugging
description: Method for debugging to true root cause — reproduce first, verify the
instrument before the reading, attribute before fixing, one variable at a time,
and don't stop at the first real bug. Use for crashes, hangs, wrong output,
flaky tests, perf regressions.
---
# Root-cause debugging
1. **Reproduce before you theorize.** Capture the failing state (logs, dumps,
exact binaries, inputs) BEFORE restarting or rebuilding — the crime scene is
evidence you cannot recreate.
2. **Verify the instrument before the reading.** Log rings flood and evict;
counters include things you didn't expect; test filters that match nothing
exit green; screenshot tools return stale images. Inject a known event and
watch it appear before trusting any reading from that channel.
3. **Start with data, not logic.** Most wrong-output bugs are the system reading
different data than you think (wrong file resolved, stale cache served, wrong
buffer bound) — not wrong math. Write a known value at the source and see
whether the output changes before auditing the transform.
4. **Bisect by layer with known values**, halving the suspect region — rather
than reading the whole pipeline end to end.
5. **One variable at a time.** Simultaneous changes mask each other.
6. **Separate environment from code.** Enumerate the environmental suspects
explicitly — stale binaries, concurrent processes sharing resources, degraded
external state, config drift — before blaming code. They pattern-match code
bugs perfectly until listed.
7. **Attribute before you fix**: who is blocked, on what, since when (watchdogs,
phase timers, stack captures, dumps). One symptom can decompose into several
independent mechanisms; attribution finds them all.
8. **Test the counter-hypothesis quantitatively.** State what your hypothesis
predicts AND what the leading alternative predicts, then measure. A result
consistent with both is not evidence yet.
9. **A found bug is not the reported bug.** Keep hunting until the reported
symptom is explained by a demonstrated mechanism; confirm you are looking at
the case the reporter meant.
10. **Intermittent means under-instrumented.** Make it deterministic or make it
loud (a watchdog that captures state at trigger). "Couldn't reproduce"
closes nothing — it queues instrumentation.
11. **Measure the absence.** Re-run the original repro after the fix and show
the symptom gone under the same measurement, plus what did NOT change.Debugging deserves its own treatment because its failure modes are evidentiary, not analytical — the agent's reasoning is usually fine; its inputs were not what it believed. Three practices carry most of the value:
Two smaller rules with outsized effect: environment before code (an environmental cause — stale binary, a second process holding the resource, a degraded driver — mimics a code bug perfectly, and enumerating them takes a minute), and measure the absence (re-run the original repro post-fix under the same measurement and report what did not change; "seems fixed" is where regressions hide).
Perf debugging note: prefer per-stage attribution over end-to-end totals. A total that includes waiting time is the classic unverified instrument — it will happily tell you a change did nothing, or everything, depending on what else was waiting.
When every agent is the same model — by choice for cost, or because a stronger tier is capped — reliability has to come entirely from independence, role separation, and structure. What works:
Run the tiered scheme for a week and track, per review class: catches by reviewer tier, escaped defects found by spot-audit, and tokens per merged change. Promote a class to the cheaper tier when its catch profile matches; demote on any escape. Review-tier assignments should be earned by data, not asserted.