Agents you
can check

01Field note with data

Rules written
after failures

Operating multi-agent coding agents at a small company. Version 0.1, October 2026.

43 active days of retained transcripts, mostly mid-September to October 17

• 1,542sessions
• 2,370subagent transcripts
• 166,638tool calls, 68% inside subagents
• 1,684typed prompts, about 99 tool calls each
• 140peak concurrent agents in a 10-minute window

The taxonomy

Before and after one policy change

On one day the operator rewrote the global instruction file and the harness deny rules: test suites only on request, no worktrees in interactive sessions, commits with explicit paths, never stash. The comparison is not causal; workload and projects changed too, and the “after” window is about 11 days.

67 of the 99 worktree commands did not begin with the git command, so a deny rule matched on a command prefix would not have seen them. More than half of the stash runs before the change came from subagents, while a memory note in the main repository already forbade stashing.

Controls the rules turned into

• Policy hook

  Allow or deny each tool call with a rule id: force-push, hard reset, stash in a shared checkout, credential-shaped writes into memory or handoff files, deploys without a fresh operator approval.

• Ancestry check

  An “already fixed in <sha>” claim is checked with git merge-base --is-ancestor before anyone believes it.

• Confirm-gated writes

  The agent can only relay a status rendered from stored state.

• Rule-based merge verdict

  Replaced a model judge that flipped between pass and block on identical input at low temperature.

• Run caps and diff guards

  Caps on unattended runs and guards on the size and shape of their diffs.

Known gaps of these patterns are part of the release: prefix deny rules versus compound commands, prompt-only rules that may not reach subagents, drift between layers that express the same rule, and the limits of lexical matching.

Limitations

One operator at one company. Transcripts mostly from the last five weeks, because the harness deletes older ones, against a four-month prompt history. Heuristic definitions inferred from harness files that change between versions. Noisy lexical measures for shell commands. No ground truth yet: the taxonomy has not been applied by independent labelers.

02Research agenda

What I want
to measure next

1. i

   Is done done?

   Extract every “done”, “verified”, “passing” or “deployed” claim. Check it against the agent's own trajectory (did the matching check run after the last edit, pass, and execute a nonzero number of tests?) and against later outcomes (operator contradiction, revert, a fix on top). Trace subagent claims through the orchestrator to the human, and release the extractor as a local tool.

2. ii

   Monitors on real traffic

   Compare deployed rule-based hooks with model-based monitors run offline on the same traffic, at an alert budget one operator can absorb: true and false positive rates, lead time over the operator's own interrupt, whether the same action is reached another way after a block, and test-retest stability.

3. iii

   Rule persistence

   Which rules survive a compaction, reach a subagent, or are weakened by a later memory note. Replay real incidents in a sandbox with stand-in deploy and push commands.

4. iv

   Honest status for non-experts

   In an offline replay harness with frozen fixtures, measure calibration, omissions, answer flips under pushback and provenance accuracy when a bot answers status questions from people who cannot check the work themselves.

03AI in biotech

Lineage, chemotype,
chromatogram

In progress