wangjunxiao/ponytail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(readme): make the benchmark chart multi-metric

Browse Source 
Grouped bars showing LOC, tokens, cost and time, each as a % of the no-skill
baseline (baseline = 100%, lower is leaner/cheaper/faster), instead of LOC
alone. Makes ponytail's win on every metric visible and shows caveman rising
above 100% on tokens/cost/time. Same system-gray palette so it reads on both
GitHub themes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Emeriko
2026-06-18 16:56:37 +02:00
co-authored by Claude Opus 4.8
parent 1a9689af1d
commit f0dad86a52
2 changed files with 50 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1 -1
View File
@@ -47,7 +47,7 @@ More survivors in [examples/](examples/).
The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5. The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5.
<p align="center"> <p align="center">
<img src="assets/benchmark-agentic.svg" width="860" alt="Mean lines of code per task by arm on the agentic benchmark (Haiku 4.5): baseline 191, caveman 152, yagni-oneliner 128, ponytail 87"> <img src="assets/benchmark-agentic.svg" width="860" alt="Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%.">
</p> </p>
| vs no-skill baseline | LOC | tokens | cost | time | safe | | vs no-skill baseline | LOC | tokens | cost | time | safe |
assets/benchmark-agentic.svg
+49 -17
View File
@@ -1,23 +1,55 @@
<svg viewBox="0 0 860 268" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif"> <svg viewBox="0 0 860 430" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
<title>Mean lines of code per task by arm, agentic benchmark on Haiku 4.5</title> <title>Each arm vs the no-skill baseline across every metric, Claude Code on Haiku 4.5</title>
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Lines of code per task (12 real-repo features), Claude Code on Haiku 4.5</text> <text x="430" y="24" font-size="15" font-weight="600" fill="#8b949e" text-anchor="middle">Every metric vs the no-skill baseline (Claude Code, Haiku 4.5, 12 tasks)</text>
<text x="20" y="46" font-size="12" fill="#8b949e" opacity="0.85">Mean over 12 frontend + backend tickets, n=4. Lower is leaner. ponytail writes 54% less than a no-skill agent and stays 100% safe.</text>
<text x="132" y="91" font-size="13" fill="#8b949e" text-anchor="end">baseline</text> <rect x="212" y="38" width="12" height="12" rx="2" fill="#8b949e"/><text x="229" y="48" font-size="12" fill="#8b949e">baseline</text>
<rect x="140" y="78" width="520" height="18" rx="3" fill="#8b949e"/> <rect x="300" y="38" width="12" height="12" rx="2" fill="#d9822b"/><text x="317" y="48" font-size="12" fill="#8b949e">caveman</text>
<text x="668" y="91" font-size="12" fill="#8b949e">191</text> <rect x="392" y="38" width="12" height="12" rx="2" fill="#2da44e"/><text x="409" y="48" font-size="12" fill="#8b949e">ponytail</text>
<rect x="478" y="38" width="12" height="12" rx="2" fill="#8957e5"/><text x="495" y="48" font-size="12" fill="#8b949e">yagni-oneliner</text>
<text x="132" y="129" font-size="13" fill="#8b949e" text-anchor="end">caveman</text> <text x="32" y="248" font-size="12" fill="#8b949e" text-anchor="middle" transform="rotate(-90 32 248)">% of baseline (lower is leaner)</text>
<rect x="140" y="116" width="413" height="18" rx="3" fill="#d9822b"/> <line x1="85" y1="360" x2="815" y2="360" stroke="#8b949e" stroke-opacity="0.55"/>
<text x="561" y="129" font-size="12" fill="#d9822b">152 (-20%)</text> <line x1="85" y1="305" x2="815" y2="305" stroke="#8b949e" stroke-opacity="0.16"/>
<line x1="85" y1="250" x2="815" y2="250" stroke="#8b949e" stroke-opacity="0.16"/>
<line x1="85" y1="195" x2="815" y2="195" stroke="#8b949e" stroke-opacity="0.16"/>
<line x1="85" y1="140" x2="815" y2="140" stroke="#8b949e" stroke-opacity="0.45" stroke-dasharray="4 4"/>
<text x="78" y="364" font-size="11" fill="#8b949e" text-anchor="end">0%</text>
<text x="78" y="309" font-size="11" fill="#8b949e" text-anchor="end">25%</text>
<text x="78" y="254" font-size="11" fill="#8b949e" text-anchor="end">50%</text>
<text x="78" y="199" font-size="11" fill="#8b949e" text-anchor="end">75%</text>
<text x="78" y="144" font-size="11" fill="#8b949e" text-anchor="end">100%</text>
<text x="132" y="167" font-size="13" fill="#8b949e" text-anchor="end">yagni-oneliner</text> <!-- LOC -->
<rect x="140" y="154" width="348" height="18" rx="3" fill="#8957e5"/> <rect x="108" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="123" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
<text x="496" y="167" font-size="12" fill="#8957e5">128 (-33%)</text> <rect x="146" y="184" width="30" height="176" rx="2" fill="#d9822b"/><text x="161" y="179" font-size="10" fill="#d9822b" text-anchor="middle">80%</text>
<rect x="184" y="259" width="30" height="101" rx="2" fill="#2da44e"/><text x="199" y="254" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">46%</text>
<rect x="222" y="213" width="30" height="147" rx="2" fill="#8957e5"/><text x="237" y="208" font-size="10" fill="#8957e5" text-anchor="middle">67%</text>
<text x="180" y="380" font-size="13" fill="#8b949e" text-anchor="middle">LOC</text>
<text x="180" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 191</text>
<text x="132" y="205" font-size="13" font-weight="600" fill="#2da44e" text-anchor="end">ponytail</text> <!-- tokens -->
<rect x="140" y="192" width="237" height="18" rx="3" fill="#2da44e"/> <rect x="288" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="303" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
<text x="385" y="205" font-size="12" font-weight="600" fill="#2da44e">87 (-54%)</text> <rect x="326" y="125" width="30" height="235" rx="2" fill="#d9822b"/><text x="341" y="120" font-size="10" fill="#d9822b" text-anchor="middle">107%</text>
<rect x="364" y="188" width="30" height="172" rx="2" fill="#2da44e"/><text x="379" y="183" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">78%</text>
<rect x="402" y="171" width="30" height="189" rx="2" fill="#8957e5"/><text x="417" y="166" font-size="10" fill="#8957e5" text-anchor="middle">86%</text>
<text x="360" y="380" font-size="13" fill="#8b949e" text-anchor="middle">tokens</text>
<text x="360" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 349k</text>
<text x="20" y="248" font-size="11" fill="#8b949e" opacity="0.8">Range by task: -94% on over-build features (date/color picker, native input vs custom component) to ~0% on already-minimal backend code. LOC = git diff added lines. Writeup: benchmarks/results/2026-06-18-agentic.md</text> <!-- cost -->
<rect x="468" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="483" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
<rect x="506" y="136" width="30" height="224" rx="2" fill="#d9822b"/><text x="521" y="131" font-size="10" fill="#d9822b" text-anchor="middle">102%</text>
<rect x="544" y="184" width="30" height="176" rx="2" fill="#2da44e"/><text x="559" y="179" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">80%</text>
<rect x="582" y="188" width="30" height="172" rx="2" fill="#8957e5"/><text x="597" y="183" font-size="10" fill="#8957e5" text-anchor="middle">78%</text>
<text x="540" y="380" font-size="13" fill="#8b949e" text-anchor="middle">cost</text>
<text x="540" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base $0.10</text>
<!-- time -->
<rect x="648" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="663" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
<rect x="686" y="136" width="30" height="224" rx="2" fill="#d9822b"/><text x="701" y="131" font-size="10" fill="#d9822b" text-anchor="middle">102%</text>
<rect x="724" y="199" width="30" height="161" rx="2" fill="#2da44e"/><text x="739" y="194" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">73%</text>
<rect x="762" y="206" width="30" height="154" rx="2" fill="#8957e5"/><text x="777" y="201" font-size="10" fill="#8957e5" text-anchor="middle">70%</text>
<text x="720" y="380" font-size="13" fill="#8b949e" text-anchor="middle">time</text>
<text x="720" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 69s</text>
<text x="20" y="420" font-size="11" fill="#8b949e" opacity="0.8">Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4. Writeup: benchmarks/results/2026-06-18-agentic.md</text>
</svg> </svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.8 KiB

After

Width:  |  Height:  |  Size: 5.5 KiB