docs(readme): add the safety tier to the benchmark chart

Separate strip below the metric clusters showing the 6-task adversarial safe rate (higher is safer): baseline, caveman and ponytail 100%, yagni-oneliner 95% (flagged). Kept apart from the lower-is-better % cluster so the opposite polarity and different task set are not conflated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs(readme): make the benchmark chart multi-metric
2026-06-18 17:01:28 +02:00 · 2026-06-18 16:56:37 +02:00 · 2026-06-18 16:48:20 +02:00 · 2026-06-18 16:19:48 +02:00
1 changed files with 2 additions and 2 deletions
@@ -19,8 +19,8 @@
 </p>

 <p align="center">
-  <strong>~54% less code (up to 94%) &middot; ~20% cheaper &middot; ~27% faster &middot; 100% safe</strong><br>
-  <sub>Measured on real Claude Code sessions editing a real open-source repo (FastAPI + React), against the same agent with no skill. ~54% is the mean across 12 feature tasks (Haiku 4.5, n=4); it reaches 94% where an agent over-builds (a date picker) and is near zero where the code is already minimal. ponytail keeps every safety guard while a bare "write one-liners" prompt drops one. (The earlier single-shot benchmark reported 80-94% as a flat figure; against a fair agentic baseline that is the per-task ceiling, not the average.) <a href="benchmarks/results/2026-06-18-agentic.md">Full writeup</a> &middot; <a href="benchmarks/">reproduce it</a>.</sub>
+  <strong>~54% less code &middot; ~20% cheaper &middot; ~27% faster &middot; 100% safe</strong><br>
+  <sub>Measured on real Claude Code sessions editing a real open-source repo (FastAPI + React), against the same agent with no skill. Mean across 12 feature tasks (Haiku 4.5, n=4). ponytail keeps every safety guard while a bare "write one-liners" prompt drops one. (An older single-shot test showed a larger 80-94% gap, but that counted a chatty model's prose; this is the honest multi-turn number.) <a href="benchmarks/results/2026-06-18-agentic.md">Full writeup</a> &middot; <a href="benchmarks/">reproduce it</a>.</sub>
 </p>

 ---
Author	SHA1	Message	Date
EmerikoandClaude Opus 4.8	b742c9c07c	docs(readme): add the safety tier to the benchmark chart Separate strip below the metric clusters showing the 6-task adversarial safe rate (higher is safer): baseline, caveman and ponytail 100%, yagni-oneliner 95% (flagged). Kept apart from the lower-is-better % cluster so the opposite polarity and different task set are not conflated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 17:01:28 +02:00
EmerikoandClaude Opus 4.8	f0dad86a52	docs(readme): make the benchmark chart multi-metric Grouped bars showing LOC, tokens, cost and time, each as a % of the no-skill baseline (baseline = 100%, lower is leaner/cheaper/faster), instead of LOC alone. Makes ponytail's win on every metric visible and shows caveman rising above 100% on tokens/cost/time. Same system-gray palette so it reads on both GitHub themes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 16:56:37 +02:00
EmerikoandClaude Opus 4.8	1a9689af1d	docs(readme): add agentic benchmark chart (LOC per arm) Bar chart of mean LOC per task by arm (Haiku 4.5) in the Numbers section, same system-gray style as the single-shot chart so it reads on both GitHub themes. The old single-shot chart stays in the collapsed "isolated generation" section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 16:48:20 +02:00
EmerikoandClaude Opus 4.8	75fea8f743	feat(benchmarks): agentic LOC + safety benchmark answering #126 Rebuild the benchmark to the standard #126 asked for: real headless Claude Code sessions (not a bare model) editing a real public repo (tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline, caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is the git diff; the safety tasks execute the produced code against adversarial input. Results: ponytail -54% LOC mean (up to -94% on over-build features like the date/color picker), -22% tokens, -20% cost, -27% time, and never more than baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal guard once). caveman writes less code but spends more tokens. Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook fired on every arm; now isolated with --setting-sources project,local + per-arm --plugin-dir) and a Windows subprocess-timeout hang. Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a labelled "isolated generation" note; supersede the contaminated 2026-06-17 writeup. Dead react-app fixture left untracked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 16:19:48 +02:00