feat(benchmarks): agentic LOC + safety benchmark answering #126

Rebuild the benchmark to the standard #126 asked for: real headless Claude Code sessions (not a bare model) editing a real public repo (tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline, caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is the git diff; the safety tasks execute the produced code against adversarial input. Results: ponytail -54% LOC mean (up to -94% on over-build features like the date/color picker), -22% tokens, -20% cost, -27% time, and never more than baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal guard once). caveman writes less code but spends more tokens. Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook fired on every arm; now isolated with --setting-sources project,local + per-arm --plugin-dir) and a Windows subprocess-timeout hang. Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a labelled "isolated generation" note; supersede the contaminated 2026-06-17 writeup. Dead react-app fixture left untracked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(examples): replace hand-written examples with real benchmark output
2026-06-18 16:13:51 +02:00 · 2026-06-17 04:31:52 +02:00 · 2026-06-17 04:22:47 +02:00 · 2026-06-17 04:20:19 +02:00 · 2026-06-17 04:17:44 +02:00 · 2026-06-17 04:16:49 +02:00
2 changed files with 2 additions and 68 deletions
@@ -19,8 +19,8 @@
 </p>
 <p align="center">
-  <strong>~54% less code (up to 94%) &middot; ~20% cheaper &middot; ~27% faster &middot; 100% safe</strong><br>
+  <strong>~54% less code &middot; ~20% cheaper &middot; ~27% faster &middot; 100% safe</strong><br>
-  <sub>Measured on real Claude Code sessions editing a real open-source repo (FastAPI + React), against the same agent with no skill. ~54% is the mean across 12 feature tasks (Haiku 4.5, n=4); it reaches 94% where an agent over-builds (a date picker) and is near zero where the code is already minimal. ponytail keeps every safety guard while a bare "write one-liners" prompt drops one. (The earlier single-shot benchmark reported 80-94% as a flat figure; against a fair agentic baseline that is the per-task ceiling, not the average.) <a href="benchmarks/results/2026-06-18-agentic.md">Full writeup</a> &middot; <a href="benchmarks/">reproduce it</a>.</sub>
+  <sub>Measured on real Claude Code sessions editing a real open-source repo (FastAPI + React), against the same agent with no skill. Mean across 12 feature tasks (Haiku 4.5, n=4). ponytail keeps every safety guard while a bare "write one-liners" prompt drops one. (An older single-shot test showed a larger 80-94% gap, but that counted a chatty model's prose; this is the honest multi-turn number.) <a href="benchmarks/results/2026-06-18-agentic.md">Full writeup</a> &middot; <a href="benchmarks/">reproduce it</a>.</sub>
 </p>
 ---
@@ -46,10 +46,6 @@ More survivors in [examples/](examples/).
 The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5.
 <p align="center">
  <img src="assets/benchmark-agentic.svg" width="860" alt="Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%. Safety, separate adversarial tier: baseline, caveman and ponytail 100%, yagni-oneliner 95%.">
 </p>
 | vs no-skill baseline | LOC | tokens | cost | time | safe |
 |---|--:|--:|--:|--:|--:|
 | **ponytail** | **-54%** | **-22%** | **-20%** | **-27%** | **100%** |
@@ -1,62 +0,0 @@
 <svg viewBox="0 0 860 488" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
  <title>Each arm vs the no-skill baseline across every metric, plus safety, Claude Code on Haiku 4.5</title>
  <text x="430" y="24" font-size="15" font-weight="600" fill="#8b949e" text-anchor="middle">Every metric vs the no-skill baseline (Claude Code, Haiku 4.5, 12 tasks)</text>
  <rect x="212" y="38" width="12" height="12" rx="2" fill="#8b949e"/><text x="229" y="48" font-size="12" fill="#8b949e">baseline</text>
  <rect x="300" y="38" width="12" height="12" rx="2" fill="#d9822b"/><text x="317" y="48" font-size="12" fill="#8b949e">caveman</text>
  <rect x="392" y="38" width="12" height="12" rx="2" fill="#2da44e"/><text x="409" y="48" font-size="12" fill="#8b949e">ponytail</text>
  <rect x="478" y="38" width="12" height="12" rx="2" fill="#8957e5"/><text x="495" y="48" font-size="12" fill="#8b949e">yagni-oneliner</text>
  <text x="32" y="248" font-size="12" fill="#8b949e" text-anchor="middle" transform="rotate(-90 32 248)">% of baseline (lower is leaner)</text>
  <line x1="85" y1="360" x2="815" y2="360" stroke="#8b949e" stroke-opacity="0.55"/>
  <line x1="85" y1="305" x2="815" y2="305" stroke="#8b949e" stroke-opacity="0.16"/>
  <line x1="85" y1="250" x2="815" y2="250" stroke="#8b949e" stroke-opacity="0.16"/>
  <line x1="85" y1="195" x2="815" y2="195" stroke="#8b949e" stroke-opacity="0.16"/>
  <line x1="85" y1="140" x2="815" y2="140" stroke="#8b949e" stroke-opacity="0.45" stroke-dasharray="4 4"/>
  <text x="78" y="364" font-size="11" fill="#8b949e" text-anchor="end">0%</text>
  <text x="78" y="309" font-size="11" fill="#8b949e" text-anchor="end">25%</text>
  <text x="78" y="254" font-size="11" fill="#8b949e" text-anchor="end">50%</text>
  <text x="78" y="199" font-size="11" fill="#8b949e" text-anchor="end">75%</text>
  <text x="78" y="144" font-size="11" fill="#8b949e" text-anchor="end">100%</text>
  <!-- LOC -->
  <rect x="108" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="123" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
  <rect x="146" y="184" width="30" height="176" rx="2" fill="#d9822b"/><text x="161" y="179" font-size="10" fill="#d9822b" text-anchor="middle">80%</text>
  <rect x="184" y="259" width="30" height="101" rx="2" fill="#2da44e"/><text x="199" y="254" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">46%</text>
  <rect x="222" y="213" width="30" height="147" rx="2" fill="#8957e5"/><text x="237" y="208" font-size="10" fill="#8957e5" text-anchor="middle">67%</text>
  <text x="180" y="380" font-size="13" fill="#8b949e" text-anchor="middle">LOC</text>
  <text x="180" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 191</text>
  <!-- tokens -->
  <rect x="288" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="303" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
  <rect x="326" y="125" width="30" height="235" rx="2" fill="#d9822b"/><text x="341" y="120" font-size="10" fill="#d9822b" text-anchor="middle">107%</text>
  <rect x="364" y="188" width="30" height="172" rx="2" fill="#2da44e"/><text x="379" y="183" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">78%</text>
  <rect x="402" y="171" width="30" height="189" rx="2" fill="#8957e5"/><text x="417" y="166" font-size="10" fill="#8957e5" text-anchor="middle">86%</text>
  <text x="360" y="380" font-size="13" fill="#8b949e" text-anchor="middle">tokens</text>
  <text x="360" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 349k</text>
  <!-- cost -->
  <rect x="468" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="483" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
  <rect x="506" y="136" width="30" height="224" rx="2" fill="#d9822b"/><text x="521" y="131" font-size="10" fill="#d9822b" text-anchor="middle">102%</text>
  <rect x="544" y="184" width="30" height="176" rx="2" fill="#2da44e"/><text x="559" y="179" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">80%</text>
  <rect x="582" y="188" width="30" height="172" rx="2" fill="#8957e5"/><text x="597" y="183" font-size="10" fill="#8957e5" text-anchor="middle">78%</text>
  <text x="540" y="380" font-size="13" fill="#8b949e" text-anchor="middle">cost</text>
  <text x="540" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base $0.10</text>
  <!-- time -->
  <rect x="648" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="663" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
  <rect x="686" y="136" width="30" height="224" rx="2" fill="#d9822b"/><text x="701" y="131" font-size="10" fill="#d9822b" text-anchor="middle">102%</text>
  <rect x="724" y="199" width="30" height="161" rx="2" fill="#2da44e"/><text x="739" y="194" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">73%</text>
  <rect x="762" y="206" width="30" height="154" rx="2" fill="#8957e5"/><text x="777" y="201" font-size="10" fill="#8957e5" text-anchor="middle">70%</text>
  <text x="720" y="380" font-size="13" fill="#8b949e" text-anchor="middle">time</text>
  <text x="720" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 69s</text>
  <text x="20" y="418" font-size="11" fill="#8b949e" opacity="0.8">Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4.</text>
  <line x1="20" y1="438" x2="815" y2="438" stroke="#8b949e" stroke-opacity="0.25"/>
  <text x="20" y="460" font-size="11" fill="#8b949e" opacity="0.9">Safety, separate 6-task adversarial tier (path-traversal, SQLi, token forgery, malformed input, rate-limit). Higher is safer:</text>
  <text x="90" y="478" font-size="12" fill="#8b949e">baseline 100%</text>
  <text x="230" y="478" font-size="12" fill="#d9822b">caveman 100%</text>
  <text x="370" y="478" font-size="12" font-weight="600" fill="#2da44e">ponytail 100%</text>
  <text x="510" y="478" font-size="12" fill="#8957e5">yagni-oneliner <tspan fill="#cf222e" font-weight="600">95%</tspan> (dropped a guard once)</text>
 </svg>
Author	SHA1	Message	Date
EmerikoandClaude Opus 4.8	1f130793d4	feat(benchmarks): agentic LOC + safety benchmark answering #126 Rebuild the benchmark to the standard #126 asked for: real headless Claude Code sessions (not a bare model) editing a real public repo (tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline, caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is the git diff; the safety tasks execute the produced code against adversarial input. Results: ponytail -54% LOC mean (up to -94% on over-build features like the date/color picker), -22% tokens, -20% cost, -27% time, and never more than baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal guard once). caveman writes less code but spends more tokens. Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook fired on every arm; now isolated with --setting-sources project,local + per-arm --plugin-dir) and a Windows subprocess-timeout hang. Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a labelled "isolated generation" note; supersede the contaminated 2026-06-17 writeup. Dead react-app fixture left untracked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 16:13:51 +02:00
EmerikoandClaude Opus 4.8	08440027f2	fix(examples): replace hand-written examples with real benchmark output The examples/ before/after blocks were authored by hand, not produced by a model. Issue #127 correctly noted that nobody hand-rolls quicksort for "sort this array" - every model just calls .sort(). Regenerate all examples verbatim from a real benchmark run (Claude Haiku 4.5, no-skill arm vs ponytail arm, benchmarks/output.json) so the before/after is reproducible, not authored: email 75->3, debounce 116->10, csv 20->3, countdown 267->9, rate-limit 128->10 LOC - Delete sorting.md (pure strawman) plus the other hand-written caricatures (api-endpoint, caching, date-picker) - Add benchmarks/generate-examples.mjs to regenerate examples from any run - examples/README.md indexes the set and documents how to reproduce Closes #127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 04:31:52 +02:00
EmerikoandClaude Opus 4.8	f72c1459dd	docs: fix run count in caption (cost is 30 runs, not 10) Cost was re-verified at 30 reps; code and latency are still the original 10. The headline caption said "10 runs" across the board, which undersold the cost verification. Now states the split, matching benchmarks/README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 04:22:47 +02:00
EmerikoandClaude Opus 4.8	399b1dedd5	docs: state the single-shot limitation honestly The benchmark is single-shot (one prompt, one completion); it does not measure a real multi-turn agent session, where the ruleset re-injects and the ladder deliberates every turn. Adds that caveat to the README, and corrects the benchmarks/README note that claimed caching widens the gap "in ponytail's favor" (unverified, and a measured agentic A/B in #121 found the opposite can happen). Per-session cost can land either way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 04:20:19 +02:00
EmerikoandClaude Opus 4.8	b4c3659f69	docs: name the ladder's reasoning cost The ladder is a deliberation step: on reasoning models the agent spends thinking tokens working through the rungs before it saves any output, which together with the always-on ruleset can outweigh the shorter code. Makes the GPT-5.5 cost increase legible rather than just stating it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 04:17:44 +02:00
EmerikoandClaude Opus 4.8	334df3d7c7	docs: reframe the pitch as the discipline, not token savings The cost/code/latency numbers vary by model and on some (terse reasoning models like GPT-5.5) ponytail costs more, so leading with them as a universal win was misleading. Adds model-variance to the headline caption and a paragraph making the stated point the mental model: write only what the task needs, safety kept, maintainable code. Savings are a model-dependent side effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 04:16:49 +02:00
EmerikoandClaude Opus 4.8	69bf5967a5	docs: scope the body claim to Claude models "on every model" read as cross-provider, but the 30-rep verification shows the cost win reverses on OpenAI reasoning models. Match the caption and benchmarks/README, which already say Claude. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 03:56:00 +02:00
EmerikoandClaude Opus 4.8	0bf152a987	docs: correct cost claim to 42-75% from 30-rep re-verification Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus): ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%. The direction holds, both ends came in a few points lower. Updates the README headline and body, the benchmark chart subtitle, and the benchmarks/README cost table, and adds a dated results doc with full method. Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On OpenAI reasoning models ponytail costs more, not less, so the claim stays Claude-scoped. Gemini run pending a fresh-quota day. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 03:47:36 +02:00