wangjunxiao/ponytail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: reproducible promptfoo benchmark + 3-model results

Browse Source 
Commit a promptfoo harness (config + arm prompts + LOC metric + vendored
caveman SKILL) so anyone can re-run the comparison: no-skill vs caveman vs
ponytail, across Haiku / Sonnet / Opus, 10 runs per cell, median reported.

Replace the old unreproducible 6-task chart with assets/benchmark-3model.svg
from this run, and reframe the README to the reproducible numbers: ponytail
writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a
no-skill agent on every model. benchmarks/README.md carries the median tables
and the reproduce command. Drops nothing that is not measured.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Emeriko
2026-06-13 05:08:55 +02:00
co-authored by Claude Opus 4.8
parent 93f3ac1d76
commit 321a59c82f
12 changed files with 221 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
assets/benchmark-3model.svg
+21
View File
@@ -0,0 +1,21 @@
<svg viewBox="0 0 860 336" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
<title>Median lines of code per arm across three models</title>
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Median lines of code. 10 runs per cell. Lower is leaner.</text>
<text x="20" y="45" font-size="12" fill="#8b949e" opacity="0.85">Ponytail writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a no-skill agent.</text>
<rect x="20" y="58" width="12" height="12" rx="2" fill="#8b949e"/><text x="38" y="69" font-size="13" fill="#8b949e">baseline (no skill)</text>
<rect x="190" y="58" width="12" height="12" rx="2" fill="#d9822b"/><text x="208" y="69" font-size="13" fill="#8b949e">caveman</text>
<rect x="300" y="58" width="12" height="12" rx="2" fill="#2da44e"/><text x="318" y="69" font-size="13" fill="#8b949e">ponytail</text>
<text x="112" y="119" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Haiku</text>
<rect x="120" y="92" width="508" height="14" rx="2" fill="#8b949e"/><text x="634" y="103" font-size="11" fill="#8b949e">518</text>
<rect x="120" y="110" width="114" height="14" rx="2" fill="#d9822b"/><text x="240" y="121" font-size="11" fill="#d9822b">116</text>
<rect x="120" y="128" width="38" height="14" rx="2" fill="#2da44e"/><text x="164" y="139" font-size="11" fill="#2da44e" font-weight="600">39</text>
<text x="112" y="193" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Sonnet</text>
<rect x="120" y="166" width="680" height="14" rx="2" fill="#8b949e"/><text x="806" y="177" font-size="11" fill="#8b949e">693</text>
<rect x="120" y="184" width="118" height="14" rx="2" fill="#d9822b"/><text x="244" y="195" font-size="11" fill="#d9822b">120</text>
<rect x="120" y="202" width="43" height="14" rx="2" fill="#2da44e"/><text x="169" y="213" font-size="11" fill="#2da44e" font-weight="600">44</text>
<text x="112" y="267" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Opus</text>
<rect x="120" y="240" width="251" height="14" rx="2" fill="#8b949e"/><text x="377" y="251" font-size="11" fill="#8b949e">256</text>
<rect x="120" y="258" width="66" height="14" rx="2" fill="#d9822b"/><text x="192" y="269" font-size="11" fill="#d9822b">67</text>
<rect x="120" y="276" width="50" height="14" rx="2" fill="#2da44e"/><text x="176" y="287" font-size="11" fill="#2da44e" font-weight="600">51</text>
<text x="120" y="324" font-size="11" fill="#8b949e" opacity="0.8">Median of 10 runs/cell, default temperature. 5 tasks (email, debounce, CSV sum, countdown, rate-limit), same model per group. Reproduce: npx promptfoo eval -c benchmarks/promptfooconfig.yaml</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.7 KiB

assets/benchmark-loc.svg
-69
View File
@@ -1,69 +0,0 @@
<svg viewBox="0 0 860 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
<title>Non-blank lines of code per task: control vs caveman vs ponytail</title>
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Six tasks. Adversarial probes: everyone passes. Lines of code: not everyone.</text>
<!-- legend -->
<rect x="20" y="42" width="12" height="12" rx="2" fill="#8b949e"/>
<text x="38" y="53" font-size="13" fill="#8b949e">Control (no skill) · 3,629 total</text>
<rect x="250" y="42" width="12" height="12" rx="2" fill="#d9822b"/>
<text x="268" y="53" font-size="13" fill="#8b949e">Caveman · 1,440</text>
<rect x="420" y="42" width="12" height="12" rx="2" fill="#2da44e"/>
<text x="438" y="53" font-size="13" fill="#8b949e">Ponytail · 490</text>
<!-- Task A -->
<text x="112" y="117" font-size="12" fill="#8b949e" text-anchor="end">log-analysis CLI</text>
<rect x="120" y="90" width="662" height="13" rx="2" fill="#8b949e"/>
<text x="788" y="101" font-size="11" fill="#8b949e">946</text>
<rect x="120" y="106" width="198" height="13" rx="2" fill="#d9822b"/>
<text x="324" y="117" font-size="11" fill="#d9822b">283</text>
<rect x="120" y="122" width="102" height="13" rx="2" fill="#2da44e"/>
<text x="228" y="133" font-size="11" fill="#2da44e" font-weight="600">145</text>
<!-- Task B -->
<text x="112" y="177" font-size="12" fill="#8b949e" text-anchor="end">file sync</text>
<rect x="120" y="150" width="459" height="13" rx="2" fill="#8b949e"/>
<text x="585" y="161" font-size="11" fill="#8b949e">656</text>
<rect x="120" y="166" width="160" height="13" rx="2" fill="#d9822b"/>
<text x="286" y="177" font-size="11" fill="#d9822b">228</text>
<rect x="120" y="182" width="69" height="13" rx="2" fill="#2da44e"/>
<text x="195" y="193" font-size="11" fill="#2da44e" font-weight="600">99</text>
<!-- Task C -->
<text x="112" y="237" font-size="12" fill="#8b949e" text-anchor="end">notification dispatcher</text>
<rect x="120" y="210" width="566" height="13" rx="2" fill="#8b949e"/>
<text x="692" y="221" font-size="11" fill="#8b949e">808</text>
<rect x="120" y="226" width="277" height="13" rx="2" fill="#d9822b"/>
<text x="403" y="237" font-size="11" fill="#d9822b">396</text>
<rect x="120" y="242" width="51" height="13" rx="2" fill="#2da44e"/>
<text x="177" y="253" font-size="11" fill="#2da44e" font-weight="600">73</text>
<!-- Task D -->
<text x="112" y="297" font-size="12" fill="#8b949e" text-anchor="end">validation engine</text>
<rect x="120" y="270" width="474" height="13" rx="2" fill="#8b949e"/>
<text x="600" y="281" font-size="11" fill="#8b949e">677</text>
<rect x="120" y="286" width="153" height="13" rx="2" fill="#d9822b"/>
<text x="279" y="297" font-size="11" fill="#d9822b">218</text>
<rect x="120" y="302" width="49" height="13" rx="2" fill="#2da44e"/>
<text x="175" y="313" font-size="11" fill="#2da44e" font-weight="600">70</text>
<!-- Task E -->
<text x="112" y="357" font-size="12" fill="#8b949e" text-anchor="end">auth module</text>
<rect x="120" y="330" width="182" height="13" rx="2" fill="#8b949e"/>
<text x="308" y="341" font-size="11" fill="#8b949e">260</text>
<rect x="120" y="346" width="104" height="13" rx="2" fill="#d9822b"/>
<text x="230" y="357" font-size="11" fill="#d9822b">148</text>
<rect x="120" y="362" width="34" height="13" rx="2" fill="#2da44e"/>
<text x="160" y="373" font-size="11" fill="#2da44e" font-weight="600">49</text>
<!-- Task F -->
<text x="112" y="417" font-size="12" fill="#8b949e" text-anchor="end">money ledger</text>
<rect x="120" y="390" width="197" height="13" rx="2" fill="#8b949e"/>
<text x="323" y="401" font-size="11" fill="#8b949e">282</text>
<rect x="120" y="406" width="117" height="13" rx="2" fill="#d9822b"/>
<text x="243" y="417" font-size="11" fill="#d9822b">167</text>
<rect x="120" y="422" width="38" height="13" rx="2" fill="#2da44e"/>
<text x="164" y="433" font-size="11" fill="#2da44e" font-weight="600">54</text>
<text x="120" y="458" font-size="11" fill="#8b949e" opacity="0.8">Non-blank LOC, AST-counted. Same model, same specs, one fresh agent per arm. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12.</text>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 4.2 KiB