ponytail/benchmarks/results/2026-06-22-issue-245-217-comprehension.md

Comprehension & reuse: fixing #245 and #217

2026-06-22. Claude Code sessions on seeded repos. Sonnet 4.6, Opus 4.8, Haiku 4.5.

Two issues argued ponytail was lazy in the wrong place:

• #245 "Dangerously lazy": the "shortest diff wins" reflex makes the agent patch the nearest symptom instead of tracing the problem end to end, and ship a confident wrong fix.
• #217 "Missing rung": rungs 2–4 reuse code from outside the project (stdlib, platform, deps); nothing covered "did I already write this here?", a common source of duplicated AI slop.

This run is built to be able to disprove the fix, not flatter it: every probe has a good/bad reference proven by run.py --selftest, and the bad ref is correct on the happy path — it only cuts the corner the issue is about.

The fix

• #217: a new ladder rung 2, "Already in this codebase? Reuse it, don't re-write it."
• #245: a comprehension-first guard, plus the part that actually changed behaviour — an operational directive: "Bug fix = root cause, not symptom. Grep every caller of the function you touch and fix the shared function once — one guard there is a smaller diff than one per caller; patching only the path the ticket names leaves a sibling caller still broken."

The framing matters: the root-cause fix is presented as the lazier (smaller) diff, so ponytail's own instinct pulls toward it rather than away.

The #245 reproducer

trace-transfer: a bank.py where transfer() and withdraw() both debit through a shared _debit(). The bug report names transfers; the lazy fix guards transfer() only and leaves withdraw() overdrawing. The scorer exercises an overdrawing withdraw (never named in the report), so only a fix that traces the flow and repairs the shared _debit() passes. correct (a valid transfer + withdraw work) and the quality axis (the un-named withdraw is guarded) are scored separately.

Results — trace-transfer, n=6, root-cause-fix rate

model	baseline (no skill)	ponytail (with fix)
Sonnet 4.6	1/6 (0.17)	6/6 (1.0)
Opus 4.8	1/6 (0.17)	6/6 (1.0) (held across 4 runs)
Haiku 4.5	0/6 (0.0)	~0–2/6 (noise)

On both capable models the fix is decisive and verified by reading the produced code: all passing cells repair the shared _debit() (one even comments it is "the shared guard for every path that removes money"). Baseline patches only the named transfer().

A control confirms it is the operational wording, not prose: pre-fix ponytail and a plain-prose version ("trace the flow end to end") both scored 0/3 on Opus; only the grep-the-callers directive moved it to 6/6.

Haiku: a model ceiling, not a regression

Haiku does not improve — but the baseline also fails it (0/6). Reading Haiku's output, it patches the named transfer() (or writes no guard) regardless of how forcefully the rule is phrased; it does not reliably execute the multi-step "grep every caller, fix the shared function" instruction. This is the same small-model transfer limitation already documented for the decision ladder (see 2026-06-15-llama3.2-local.md), not something the fix broke. Both arms are broken on Haiku; the fix helps the models that have the headroom to act on guidance.

#217: rung shipped, failure did not reproduce

Two reuse probes (reuse-slug, reuse-money) hide a distinctively-behaved helper in a separate module the agent must discover; a re-implementation diverges observably (e.g. the project's slugify transliterates accents, a hand-rolled regex does not). Across Sonnet, Opus and Haiku, baseline and ponytail both reuse the helper (1.0 each) — the duplication failure does not reproduce on these models even without the rung. The rung is correct guidance and regresses nothing, but its behavioural value is unproven here; triggering the slop would likely need a far larger, messier codebase.

Regression check: did the rule edits break anything?

Pre-fix vs post-fix ponytail across the full 27-task runnable suite (safety + quality + open/vibe), Haiku, n=3:

• Safety: identical. All seven deterministic safety tasks score 1.0 safe before and after — no guard dropped.
• Less code: preserved, and strong where there is over-build room (e.g. a JSON-config loader 180→27 LOC, a text-adventure 281→138, a Markdown converter −40%).
• Correctness: no systematic change. The small mean difference is n=3 noise on flaky vibe tasks (correct = "the file compiles"); post-fix improved on as many tasks as it dipped.

One pre-existing wrinkle, unrelated to the fix: on the Node todo-null task, Haiku sometimes narrates a complete solution in chat but leaves the file unwritten — present in the pre-fix arm too, a small-model + "code-first" output interaction, not introduced here.

Verdict

• #245: fixed and validated on the capable tiers (Sonnet 4.6, the model it was reported on, and Opus 4.8): baseline 1/6 → ponytail 6/6, with verified root-cause fixes. Small models remain a capability ceiling where baseline also fails.
• #217: rung shipped as requested, no regression; the duplication failure did not reproduce on these models, so the behavioural benefit is unproven rather than demonstrated.

Reproduce: python run.py --selftest then python run.py --task trace-transfer --arms baseline,ponytail --models sonnet --runs 6.