wangjunxiao/ponytail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: same-model control arm, refresh numbers and chart

Browse Source 
Six no-skill control arms re-run through the same harness so all three
arms share one model. README Numbers section and chart now cite the
complete dataset: -47% tokens, 3x faster, 490 vs 3,629 LOC, extension
96 vs 1,115 lines, probes green everywhere. Em dashes removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
dgebert
2026-06-12 12:44:30 +02:00
co-authored by Claude Fable 5
parent b609e019a0
commit 9c99843725
3 changed files with 42 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
assets/benchmark-loc.svg
+15 -15
View File
@@ -1,11 +1,11 @@
<svg viewBox="0 0 860 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
<title>Non-blank lines of code per task — control vs caveman vs ponytail</title>
<title>Non-blank lines of code per task: control vs caveman vs ponytail</title>
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Six tasks. Adversarial probes: everyone passes. Lines of code: not everyone.</text>
<!-- legend -->
<rect x="20" y="42" width="12" height="12" rx="2" fill="#8b949e"/>
<text x="38" y="53" font-size="13" fill="#8b949e">Control (no skill) · 2,943 total</text>
<text x="38" y="53" font-size="13" fill="#8b949e">Control (no skill) · 3,629 total</text>
<rect x="250" y="42" width="12" height="12" rx="2" fill="#d9822b"/>
<text x="268" y="53" font-size="13" fill="#8b949e">Caveman · 1,440</text>
<rect x="420" y="42" width="12" height="12" rx="2" fill="#2da44e"/>
@@ -13,8 +13,8 @@
<!-- Task A -->
<text x="112" y="117" font-size="12" fill="#8b949e" text-anchor="end">log-analysis CLI</text>
<rect x="120" y="90" width="679" height="13" rx="2" fill="#8b949e"/>
<text x="805" y="101" font-size="11" fill="#8b949e">970</text>
<rect x="120" y="90" width="662" height="13" rx="2" fill="#8b949e"/>
<text x="788" y="101" font-size="11" fill="#8b949e">946</text>
<rect x="120" y="106" width="198" height="13" rx="2" fill="#d9822b"/>
<text x="324" y="117" font-size="11" fill="#d9822b">283</text>
<rect x="120" y="122" width="102" height="13" rx="2" fill="#2da44e"/>
@@ -22,8 +22,8 @@
<!-- Task B -->
<text x="112" y="177" font-size="12" fill="#8b949e" text-anchor="end">file sync</text>
<rect x="120" y="150" width="411" height="13" rx="2" fill="#8b949e"/>
<text x="537" y="161" font-size="11" fill="#8b949e">587</text>
<rect x="120" y="150" width="459" height="13" rx="2" fill="#8b949e"/>
<text x="585" y="161" font-size="11" fill="#8b949e">656</text>
<rect x="120" y="166" width="160" height="13" rx="2" fill="#d9822b"/>
<text x="286" y="177" font-size="11" fill="#d9822b">228</text>
<rect x="120" y="182" width="69" height="13" rx="2" fill="#2da44e"/>
@@ -31,8 +31,8 @@
<!-- Task C -->
<text x="112" y="237" font-size="12" fill="#8b949e" text-anchor="end">notification dispatcher</text>
<rect x="120" y="210" width="508" height="13" rx="2" fill="#8b949e"/>
<text x="634" y="221" font-size="11" fill="#8b949e">726</text>
<rect x="120" y="210" width="566" height="13" rx="2" fill="#8b949e"/>
<text x="692" y="221" font-size="11" fill="#8b949e">808</text>
<rect x="120" y="226" width="277" height="13" rx="2" fill="#d9822b"/>
<text x="403" y="237" font-size="11" fill="#d9822b">396</text>
<rect x="120" y="242" width="51" height="13" rx="2" fill="#2da44e"/>
@@ -40,8 +40,8 @@
<!-- Task D -->
<text x="112" y="297" font-size="12" fill="#8b949e" text-anchor="end">validation engine</text>
<rect x="120" y="270" width="240" height="13" rx="2" fill="#8b949e"/>
<text x="366" y="281" font-size="11" fill="#8b949e">343</text>
<rect x="120" y="270" width="474" height="13" rx="2" fill="#8b949e"/>
<text x="600" y="281" font-size="11" fill="#8b949e">677</text>
<rect x="120" y="286" width="153" height="13" rx="2" fill="#d9822b"/>
<text x="279" y="297" font-size="11" fill="#d9822b">218</text>
<rect x="120" y="302" width="49" height="13" rx="2" fill="#2da44e"/>
@@ -49,8 +49,8 @@
<!-- Task E -->
<text x="112" y="357" font-size="12" fill="#8b949e" text-anchor="end">auth module</text>
<rect x="120" y="330" width="109" height="13" rx="2" fill="#8b949e"/>
<text x="235" y="341" font-size="11" fill="#8b949e">155</text>
<rect x="120" y="330" width="182" height="13" rx="2" fill="#8b949e"/>
<text x="308" y="341" font-size="11" fill="#8b949e">260</text>
<rect x="120" y="346" width="104" height="13" rx="2" fill="#d9822b"/>
<text x="230" y="357" font-size="11" fill="#d9822b">148</text>
<rect x="120" y="362" width="34" height="13" rx="2" fill="#2da44e"/>
@@ -58,12 +58,12 @@
<!-- Task F -->
<text x="112" y="417" font-size="12" fill="#8b949e" text-anchor="end">money ledger</text>
<rect x="120" y="390" width="113" height="13" rx="2" fill="#8b949e"/>
<text x="239" y="401" font-size="11" fill="#8b949e">162</text>
<rect x="120" y="390" width="197" height="13" rx="2" fill="#8b949e"/>
<text x="323" y="401" font-size="11" fill="#8b949e">282</text>
<rect x="120" y="406" width="117" height="13" rx="2" fill="#d9822b"/>
<text x="243" y="417" font-size="11" fill="#d9822b">167</text>
<rect x="120" y="422" width="38" height="13" rx="2" fill="#2da44e"/>
<text x="164" y="433" font-size="11" fill="#2da44e" font-weight="600">54</text>
<text x="120" y="458" font-size="11" fill="#8b949e" opacity="0.8">Non-blank LOC, AST-counted. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12.</text>
<text x="120" y="458" font-size="11" fill="#8b949e" opacity="0.8">Non-blank LOC, AST-counted. Same model, same specs, one fresh agent per arm. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12.</text>
</svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 4.2 KiB

After

Width:  |  Height:  |  Size: 4.2 KiB