Rebuild the benchmark to the standard #126 asked for: real headless Claude Code
sessions (not a bare model) editing a real public repo
(tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline,
caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is
the git diff; the safety tasks execute the produced code against adversarial
input.
Results: ponytail -54% LOC mean (up to -94% on over-build features like the
date/color picker), -22% tokens, -20% cost, -27% time, and never more than
baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal
guard once). caveman writes less code but spends more tokens.
Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook
fired on every arm; now isolated with --setting-sources project,local + per-arm
--plugin-dir) and a Windows subprocess-timeout hang.
Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a
labelled "isolated generation" note; supersede the contaminated 2026-06-17
writeup. Dead react-app fixture left untracked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: correct cost claim to 42-75% from 30-rep re-verification
Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus):
ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%.
The direction holds, both ends came in a few points lower. Updates the README
headline and body, the benchmark chart subtitle, and the benchmarks/README cost
table, and adds a dated results doc with full method.
Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On
OpenAI reasoning models ponytail costs more, not less, so the claim stays
Claude-scoped. Gemini run pending a fresh-quota day.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: scope the body claim to Claude models
"on every model" read as cross-provider, but the 30-rep verification shows
the cost win reverses on OpenAI reasoning models. Match the caption and
benchmarks/README, which already say Claude.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: reframe the pitch as the discipline, not token savings
The cost/code/latency numbers vary by model and on some (terse reasoning
models like GPT-5.5) ponytail costs more, so leading with them as a universal
win was misleading. Adds model-variance to the headline caption and a paragraph
making the stated point the mental model: write only what the task needs,
safety kept, maintainable code. Savings are a model-dependent side effect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: name the ladder's reasoning cost
The ladder is a deliberation step: on reasoning models the agent spends
thinking tokens working through the rungs before it saves any output, which
together with the always-on ruleset can outweigh the shorter code. Makes the
GPT-5.5 cost increase legible rather than just stating it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: state the single-shot limitation honestly
The benchmark is single-shot (one prompt, one completion); it does not measure
a real multi-turn agent session, where the ruleset re-injects and the ladder
deliberates every turn. Adds that caveat to the README, and corrects the
benchmarks/README note that claimed caching widens the gap "in ponytail's
favor" (unverified, and a measured agentic A/B in #121 found the opposite can
happen). Per-session cost can land either way.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: fix run count in caption (cost is 30 runs, not 10)
Cost was re-verified at 30 reps; code and latency are still the original 10.
The headline caption said "10 runs" across the board, which undersold the cost
verification. Now states the split, matching benchmarks/README.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(benchmarks): correctness gate scores unfenced code; fix debounce task
The `correct` gate under-reported correctness for terse models, the likely
source of "Ponytail degrades models" reports (issue #65):
- extractBlocks() only matched fenced code blocks, so bare/unfenced code
scored an automatic fail even when correct. Now falls back to the whole
response as one block (and tolerates CRLF). Debounce detection also accepts
unfenced arrow functions.
- The debounce task asked to "add debounce to a search input" but the check
expected a reusable debounce(fn, delay) util, failing correct inline answers.
Task reworded to the deliverable the check verifies.
Adds correctness.test.js (regression guard) and a GPT-mini repro config plus
results writeup: on a clean n=20 run, the reported gpt-4.1-mini drop (10/15)
does not reproduce (100/100). The LOC win (~halved) holds.
README repro fixed: promptfoo needs --env-file ../.env (reads cwd, not root).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(benchmarks): add robustness audit — ponytail vs baseline on edge cases
Answers the real question behind #65: does ponytail's push for the shortest
solution make weak models produce wrong code on edge cases?
robustness-audit.js: 16 self-verifying tasks (12 algorithmic edge-case traps +
4 validators). Each check ships a known-good and known-lazy-wrong reference that
must pass/fail before any model output is scored (--selftest, 16/16).
Findings (gpt-4.1-mini + gpt-5.4-mini, baseline vs ponytail): parity on every
edge-case trap on both models. The one measured soft spot is gpt-5.4-mini email
(~4-5%, reaches for parseaddr). A sharpened SKILL.md validation rule had no
reliable effect in an n=100 A/B (96% vs 95%), so it was not shipped — the
tendency is model-level, not skill-level. Full writeup in results/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(benchmarks): email slip is provider-specific — 100% on Claude
High-n cross-provider follow-up to the robustness audit. The one ponytail
soft spot (email validation via parseaddr) splits by provider, not model size:
- Claude (haiku/sonnet/opus): 100% under ponytail, n=40 each — and ponytail
beats baseline (unconstrained Sonnet over-engineers into an always-truthy
dict, 0/40; ponytail writes a clean validator).
- OpenAI (gpt-4.1-mini..gpt-5.5): slips at every size under ponytail
(~79-98%), baseline ~100%. The parseaddr reflex lives in OpenAI training.
Not fixable by skill text: 8 distinct SKILL.md edits (incl. an n=100 A/B,
96% vs 95%) all scored <= current, several worse, all bloated LOC. Nothing
shipped. SKILL.md unchanged.
Conclusion: on ponytail's target platform (Claude) email is 100%; the GPT
slip is a documented cross-provider transfer quirk. Adds model-email.js /
claude-email.js to reproduce the tables. Writeup updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(benchmarks): correct misleading Sonnet baseline 0 percent
The Sonnet baseline 0/40 on email is a return-type artifact, not a logic
failure: unconstrained Sonnet returns a dict {is_valid, message} instead of a
bool, so the bool-contract gate scores every case as accepted. Read dict-aware
via is_valid, its logic is ~75% correct (9/12). Reframed honestly so we are not
presenting 0 vs 100 as a clean win; ponytail still wins (clean 100% bool) but
the point is over-engineered return type, not total failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the local benchmark LOC counter (counted only fenced code, scored bare output 0), makes summary output ASCII-safe (a Unicode arrow crashed the script on Windows cp1252), gitignores generated artifacts, and refreshes the llama3.2 writeup with n=5 data showing the LOC effect is within the noise floor. Follow-up to #63. Verified live.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Six no-skill control arms re-run through the same harness so all three
arms share one model. README Numbers section and chart now cite the
complete dataset: -47% tokens, 3x faster, 490 vs 3,629 LOC, extension
96 vs 1,115 lines, probes green everywhere. Em dashes removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probes 8/8 + 6/6 both arms, LOC 490 vs 1440, extension cost 41/55 vs
156/257, all six v4 arms ship a runnable check with no bloat creep.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Skill preaching minimalism was 2x caveman length. Smaller file cuts
per-read and per-session-injection cost. Benchmark: beats caveman on
all areas now — 135.7k vs 138.4k tokens, 127s vs 136s, 47 vs 117 loc.
v1 lost to caveman on tokens/time despite minimal code: it wrote
essays defending each simplification. v2 caps explanation at three
lines and ships the lazy version instead of stalling on necessity
questions. Benchmark: 136.6k tok vs caveman 138.4k, code 47 vs 117
lines across 5 tasks.
