dedc97ca7c
#245 "Dangerously lazy": add an operational "fix the root cause, not the symptom" directive — grep every caller of the function you touch and fix the shared function once (the smaller diff). Validated on the agentic benchmark: on a shared-helper bug-fix trap, baseline fixes the root cause 1/6 while ponytail does 6/6 on both Sonnet 4.6 (the model the issue was filed on) and Opus 4.8, verified by reading the produced code. Plain prose ("trace the flow") did not move it; the actionable, lazy-framed directive did. #217 "Missing rung": add ladder rung 2 "Already in this codebase? Reuse it, don't re-write it." Propagated across SKILL.md, AGENTS.md, all agent mirror copies, the hook fallback, and both READMEs (check-rule-copies passes). Benchmark: 4 new deterministic quality-tier tasks (reuse-slug, reuse-money, trace-transfer, trace-amount) with selftest-proven good/bad refs; harness gains multi-file seed support in --selftest, distinctive-behaviour reuse detection, and counts in-file __main__/demo() self-checks as test LOC (not source bloat) for surgical tasks. Full writeup in benchmarks/results/2026-06-22-issue-245-217-comprehension.md. Also carries the in-progress todo-null benchmark task already present in the working tree. Co-authored-by: Dietrich Gebert <dgebert@Dietrichs-MacBook-Pro.local> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
99 lines
5.4 KiB
Markdown
99 lines
5.4 KiB
Markdown
# Comprehension & reuse: fixing #245 and #217
|
||
|
||
*2026-06-22. Claude Code sessions on seeded repos. Sonnet 4.6, Opus 4.8, Haiku 4.5.*
|
||
|
||
Two issues argued ponytail was lazy in the wrong place:
|
||
|
||
- [#245 "Dangerously lazy"](https://github.com/DietrichGebert/ponytail/issues/245): the "shortest
|
||
diff wins" reflex makes the agent patch the nearest symptom instead of tracing the problem end to
|
||
end, and ship a confident wrong fix.
|
||
- [#217 "Missing rung"](https://github.com/DietrichGebert/ponytail/issues/217): rungs 2–4 reuse code
|
||
from *outside* the project (stdlib, platform, deps); nothing covered "did I already write this
|
||
here?", a common source of duplicated AI slop.
|
||
|
||
This run is built to be able to *disprove* the fix, not flatter it: every probe has a `good`/`bad`
|
||
reference proven by `run.py --selftest`, and the `bad` ref is correct on the happy path — it only
|
||
cuts the corner the issue is about.
|
||
|
||
## The fix
|
||
|
||
- **#217:** a new ladder rung 2, *"Already in this codebase? Reuse it, don't re-write it."*
|
||
- **#245:** a comprehension-first guard, plus the part that actually changed behaviour — an
|
||
**operational** directive: *"Bug fix = root cause, not symptom. Grep every caller of the function
|
||
you touch and fix the shared function once — one guard there is a smaller diff than one per
|
||
caller; patching only the path the ticket names leaves a sibling caller still broken."*
|
||
|
||
The framing matters: the root-cause fix is presented as the *lazier* (smaller) diff, so ponytail's
|
||
own instinct pulls toward it rather than away.
|
||
|
||
## The #245 reproducer
|
||
|
||
`trace-transfer`: a `bank.py` where `transfer()` and `withdraw()` both debit through a shared
|
||
`_debit()`. The bug report names *transfers*; the lazy fix guards `transfer()` only and leaves
|
||
`withdraw()` overdrawing. The scorer exercises an overdrawing **withdraw** (never named in the
|
||
report), so only a fix that traces the flow and repairs the shared `_debit()` passes. `correct`
|
||
(a valid transfer + withdraw work) and the quality axis (the un-named withdraw is guarded) are
|
||
scored separately.
|
||
|
||
## Results — `trace-transfer`, n=6, root-cause-fix rate
|
||
|
||
| model | baseline (no skill) | ponytail (with fix) |
|
||
|---|--:|--:|
|
||
| **Sonnet 4.6** | 1/6 (0.17) | **6/6 (1.0)** |
|
||
| **Opus 4.8** | 1/6 (0.17) | **6/6 (1.0)** (held across 4 runs) |
|
||
| Haiku 4.5 | 0/6 (0.0) | ~0–2/6 (noise) |
|
||
|
||
On both capable models the fix is decisive and verified by reading the produced code: all passing
|
||
cells repair the shared `_debit()` (one even comments it is "the shared guard for every path that
|
||
removes money"). Baseline patches only the named `transfer()`.
|
||
|
||
A control confirms it is the *operational* wording, not prose: pre-fix ponytail and a plain-prose
|
||
version ("trace the flow end to end") both scored 0/3 on Opus; only the grep-the-callers directive
|
||
moved it to 6/6.
|
||
|
||
### Haiku: a model ceiling, not a regression
|
||
|
||
Haiku does not improve — but **the baseline also fails it (0/6)**. Reading Haiku's output, it
|
||
patches the named `transfer()` (or writes no guard) regardless of how forcefully the rule is
|
||
phrased; it does not reliably execute the multi-step "grep every caller, fix the shared function"
|
||
instruction. This is the same small-model transfer limitation already documented for the decision
|
||
ladder (see `2026-06-15-llama3.2-local.md`), not something the fix broke. Both arms are broken on
|
||
Haiku; the fix helps the models that have the headroom to act on guidance.
|
||
|
||
## #217: rung shipped, failure did not reproduce
|
||
|
||
Two reuse probes (`reuse-slug`, `reuse-money`) hide a distinctively-behaved helper in a separate
|
||
module the agent must discover; a re-implementation diverges observably (e.g. the project's
|
||
`slugify` transliterates accents, a hand-rolled regex does not). Across Sonnet, Opus and Haiku,
|
||
**baseline and ponytail both reuse the helper (1.0 each)** — the duplication failure does not
|
||
reproduce on these models even without the rung. The rung is correct guidance and regresses
|
||
nothing, but its behavioural value is unproven here; triggering the slop would likely need a far
|
||
larger, messier codebase.
|
||
|
||
## Regression check: did the rule edits break anything?
|
||
|
||
Pre-fix vs post-fix ponytail across the full 27-task runnable suite (safety + quality + open/vibe),
|
||
Haiku, n=3:
|
||
|
||
- **Safety: identical.** All seven deterministic safety tasks score 1.0 safe before and after —
|
||
no guard dropped.
|
||
- **Less code: preserved**, and strong where there is over-build room (e.g. a JSON-config loader
|
||
180→27 LOC, a text-adventure 281→138, a Markdown converter −40%).
|
||
- **Correctness: no systematic change.** The small mean difference is n=3 noise on flaky vibe tasks
|
||
(`correct` = "the file compiles"); post-fix improved on as many tasks as it dipped.
|
||
|
||
One pre-existing wrinkle, unrelated to the fix: on the Node `todo-null` task, Haiku sometimes
|
||
*narrates* a complete solution in chat but leaves the file unwritten — present in the pre-fix arm
|
||
too, a small-model + "code-first" output interaction, not introduced here.
|
||
|
||
## Verdict
|
||
|
||
- **#245: fixed and validated on the capable tiers** (Sonnet 4.6, the model it was reported on, and
|
||
Opus 4.8): baseline 1/6 → ponytail 6/6, with verified root-cause fixes. Small models remain a
|
||
capability ceiling where baseline also fails.
|
||
- **#217: rung shipped as requested**, no regression; the duplication failure did not reproduce on
|
||
these models, so the behavioural benefit is unproven rather than demonstrated.
|
||
|
||
Reproduce: `python run.py --selftest` then
|
||
`python run.py --task trace-transfer --arms baseline,ponytail --models sonnet --runs 6`.
|