Compare commits

..
Author SHA1 Message Date
EmerikoandClaude Opus 4.8 1a0b0a1b52 docs(readme): note the up-to-94% peak in the hero line
The hero showed only ~54% (the mean); the rigorous agentic run also reaches 94%
on the over-build tasks (the date picker), so the headline now reads
"~54% (up to 94%)". The sub-line is reworded so 80-94% reads as the per-task
ceiling against a fair baseline, not the old single-shot figure, which would
otherwise contradict the hero.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 18:30:57 +02:00
DietrichGebertandClaude Opus 4.8 8d5037d9e5 docs(readme): add the agentic benchmark chart (#160)
Grouped bars of LOC, tokens, cost and time as a % of the no-skill baseline
(lower is leaner/cheaper/faster), plus a separate safety strip (baseline,
caveman and ponytail 100%; yagni-oneliner 95%). System-gray palette so it reads
on both GitHub themes. The chart commits landed after #158 had already
squash-merged, so this brings the chart onto main.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-18 17:09:29 +02:00
+2 -2
View File
@@ -19,8 +19,8 @@
</p>
<p align="center">
<strong>~54% less code &middot; ~20% cheaper &middot; ~27% faster &middot; 100% safe</strong><br>
<sub>Measured on real Claude Code sessions editing a real open-source repo (FastAPI + React), against the same agent with no skill. Mean across 12 feature tasks (Haiku 4.5, n=4). ponytail keeps every safety guard while a bare "write one-liners" prompt drops one. (An older single-shot test showed a larger 80-94% gap, but that counted a chatty model's prose; this is the honest multi-turn number.) <a href="benchmarks/results/2026-06-18-agentic.md">Full writeup</a> &middot; <a href="benchmarks/">reproduce it</a>.</sub>
<strong>~54% less code (up to 94%) &middot; ~20% cheaper &middot; ~27% faster &middot; 100% safe</strong><br>
<sub>Measured on real Claude Code sessions editing a real open-source repo (FastAPI + React), against the same agent with no skill. ~54% is the mean across 12 feature tasks (Haiku 4.5, n=4); it reaches 94% where an agent over-builds (a date picker) and is near zero where the code is already minimal. ponytail keeps every safety guard while a bare "write one-liners" prompt drops one. (The earlier single-shot benchmark reported 80-94% as a flat figure; against a fair agentic baseline that is the per-task ceiling, not the average.) <a href="benchmarks/results/2026-06-18-agentic.md">Full writeup</a> &middot; <a href="benchmarks/">reproduce it</a>.</sub>
</p>
---