diff --git a/README.md b/README.md index 238c1be..c22df88 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ More survivors in [examples/](examples/). The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5.

- Mean lines of code per task by arm on the agentic benchmark (Haiku 4.5): baseline 191, caveman 152, yagni-oneliner 128, ponytail 87 + Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%.

| vs no-skill baseline | LOC | tokens | cost | time | safe | diff --git a/assets/benchmark-agentic.svg b/assets/benchmark-agentic.svg index 6903706..4336227 100644 --- a/assets/benchmark-agentic.svg +++ b/assets/benchmark-agentic.svg @@ -1,23 +1,55 @@ - - Mean lines of code per task by arm, agentic benchmark on Haiku 4.5 - Lines of code per task (12 real-repo features), Claude Code on Haiku 4.5 - Mean over 12 frontend + backend tickets, n=4. Lower is leaner. ponytail writes 54% less than a no-skill agent and stays 100% safe. + + Each arm vs the no-skill baseline across every metric, Claude Code on Haiku 4.5 + Every metric vs the no-skill baseline (Claude Code, Haiku 4.5, 12 tasks) - baseline - - 191 + baseline + caveman + ponytail + yagni-oneliner - caveman - - 152 (-20%) + % of baseline (lower is leaner) + + + + + + 0% + 25% + 50% + 75% + 100% - yagni-oneliner - - 128 (-33%) + + 100% + 80% + 46% + 67% + LOC + base 191 - ponytail - - 87 (-54%) + + 100% + 107% + 78% + 86% + tokens + base 349k - Range by task: -94% on over-build features (date/color picker, native input vs custom component) to ~0% on already-minimal backend code. LOC = git diff added lines. Writeup: benchmarks/results/2026-06-18-agentic.md + + 100% + 102% + 80% + 78% + cost + base $0.10 + + + 100% + 102% + 73% + 70% + time + base 69s + + Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4. Writeup: benchmarks/results/2026-06-18-agentic.md