wangjunxiao/ponytail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: benchmark chart — ponytail vs caveman vs control

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
dgebert
2026-06-12 12:18:18 +02:00
co-authored by Claude Fable 5
parent 983255e2a1
commit b609e019a0
2 changed files with 78 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+9 -1
View File
@@ -73,7 +73,15 @@ You know exactly why.
## Numbers
5 coding tasks, same agent with and without ponytail: **−16% tokens, ~4× faster, 293 → 47 lines.** The 246 lines nobody wrote have never caused an incident. Data: [benchmarks/](benchmarks/).
5 coding tasks, same agent with and without ponytail: **−16% tokens, ~4× faster, 293 → 47 lines.** The 246 lines nobody wrote have never caused an incident.
Then six harder tasks — streaming parsers, atomic file sync, auth, a concurrent money ledger — against a no-skill control and the [caveman](https://github.com/JuliusBrussee/caveman) skill, with adversarial security and concurrency probes:
<p align="center">
<img src="assets/benchmark-loc.svg" width="860" alt="Lines of code per task: ponytail 490 total vs caveman 1,440 vs control 2,943 — all passing the same adversarial probes">
</p>
Everyone passes every probe. Ponytail does it with a third of caveman's code, a sixth of control's, and was ~5× cheaper when the surprise feature request landed (96 lines changed vs 487 for control). Every shortcut is marked in-code with its upgrade path. Data: [benchmarks/](benchmarks/).
## License
assets/benchmark-loc.svg
+69
View File
@@ -0,0 +1,69 @@
<svg viewBox="0 0 860 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
<title>Non-blank lines of code per task — control vs caveman vs ponytail</title>
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Six tasks. Adversarial probes: everyone passes. Lines of code: not everyone.</text>
<!-- legend -->
<rect x="20" y="42" width="12" height="12" rx="2" fill="#8b949e"/>
<text x="38" y="53" font-size="13" fill="#8b949e">Control (no skill) · 2,943 total</text>
<rect x="250" y="42" width="12" height="12" rx="2" fill="#d9822b"/>
<text x="268" y="53" font-size="13" fill="#8b949e">Caveman · 1,440</text>
<rect x="420" y="42" width="12" height="12" rx="2" fill="#2da44e"/>
<text x="438" y="53" font-size="13" fill="#8b949e">Ponytail · 490</text>
<!-- Task A -->
<text x="112" y="117" font-size="12" fill="#8b949e" text-anchor="end">log-analysis CLI</text>
<rect x="120" y="90" width="679" height="13" rx="2" fill="#8b949e"/>
<text x="805" y="101" font-size="11" fill="#8b949e">970</text>
<rect x="120" y="106" width="198" height="13" rx="2" fill="#d9822b"/>
<text x="324" y="117" font-size="11" fill="#d9822b">283</text>
<rect x="120" y="122" width="102" height="13" rx="2" fill="#2da44e"/>
<text x="228" y="133" font-size="11" fill="#2da44e" font-weight="600">145</text>
<!-- Task B -->
<text x="112" y="177" font-size="12" fill="#8b949e" text-anchor="end">file sync</text>
<rect x="120" y="150" width="411" height="13" rx="2" fill="#8b949e"/>
<text x="537" y="161" font-size="11" fill="#8b949e">587</text>
<rect x="120" y="166" width="160" height="13" rx="2" fill="#d9822b"/>
<text x="286" y="177" font-size="11" fill="#d9822b">228</text>
<rect x="120" y="182" width="69" height="13" rx="2" fill="#2da44e"/>
<text x="195" y="193" font-size="11" fill="#2da44e" font-weight="600">99</text>
<!-- Task C -->
<text x="112" y="237" font-size="12" fill="#8b949e" text-anchor="end">notification dispatcher</text>
<rect x="120" y="210" width="508" height="13" rx="2" fill="#8b949e"/>
<text x="634" y="221" font-size="11" fill="#8b949e">726</text>
<rect x="120" y="226" width="277" height="13" rx="2" fill="#d9822b"/>
<text x="403" y="237" font-size="11" fill="#d9822b">396</text>
<rect x="120" y="242" width="51" height="13" rx="2" fill="#2da44e"/>
<text x="177" y="253" font-size="11" fill="#2da44e" font-weight="600">73</text>
<!-- Task D -->
<text x="112" y="297" font-size="12" fill="#8b949e" text-anchor="end">validation engine</text>
<rect x="120" y="270" width="240" height="13" rx="2" fill="#8b949e"/>
<text x="366" y="281" font-size="11" fill="#8b949e">343</text>
<rect x="120" y="286" width="153" height="13" rx="2" fill="#d9822b"/>
<text x="279" y="297" font-size="11" fill="#d9822b">218</text>
<rect x="120" y="302" width="49" height="13" rx="2" fill="#2da44e"/>
<text x="175" y="313" font-size="11" fill="#2da44e" font-weight="600">70</text>
<!-- Task E -->
<text x="112" y="357" font-size="12" fill="#8b949e" text-anchor="end">auth module</text>
<rect x="120" y="330" width="109" height="13" rx="2" fill="#8b949e"/>
<text x="235" y="341" font-size="11" fill="#8b949e">155</text>
<rect x="120" y="346" width="104" height="13" rx="2" fill="#d9822b"/>
<text x="230" y="357" font-size="11" fill="#d9822b">148</text>
<rect x="120" y="362" width="34" height="13" rx="2" fill="#2da44e"/>
<text x="160" y="373" font-size="11" fill="#2da44e" font-weight="600">49</text>
<!-- Task F -->
<text x="112" y="417" font-size="12" fill="#8b949e" text-anchor="end">money ledger</text>
<rect x="120" y="390" width="113" height="13" rx="2" fill="#8b949e"/>
<text x="239" y="401" font-size="11" fill="#8b949e">162</text>
<rect x="120" y="406" width="117" height="13" rx="2" fill="#d9822b"/>
<text x="243" y="417" font-size="11" fill="#d9822b">167</text>
<rect x="120" y="422" width="38" height="13" rx="2" fill="#2da44e"/>
<text x="164" y="433" font-size="11" fill="#2da44e" font-weight="600">54</text>
<text x="120" y="458" font-size="11" fill="#8b949e" opacity="0.8">Non-blank LOC, AST-counted. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12.</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 4.2 KiB