docs: same-model control arm, refresh numbers and chart
Six no-skill control arms re-run through the same harness so all three arms share one model. README Numbers section and chart now cite the complete dataset: -47% tokens, 3x faster, 490 vs 3,629 LOC, extension 96 vs 1,115 lines, probes green everywhere. Em dashes removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
b609e019a0
commit
9c99843725
@@ -1,5 +1,5 @@
|
|||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="assets/logo.png" width="220" alt="Ponytail — the lazy senior dev">
|
<img src="assets/logo.png" width="220" alt="Ponytail, the lazy senior dev">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<h1 align="center">Ponytail</h1>
|
<h1 align="center">Ponytail</h1>
|
||||||
@@ -63,7 +63,7 @@ Cursor, Windsurf, Cline, Copilot, Aider: copy the matching rules file from this
|
|||||||
No.
|
No.
|
||||||
|
|
||||||
**What if I really need the 120-line cache class?**
|
**What if I really need the 120-line cache class?**
|
||||||
You don't. Insist anyway and he'll build it — slowly, correctly, while looking at you.
|
You don't. Insist anyway and he'll build it. Slowly. Correctly. While looking at you.
|
||||||
|
|
||||||
**Does it scale?**
|
**Does it scale?**
|
||||||
The code you never wrote scales infinitely. Zero bugs, zero CVEs, 100% uptime since forever.
|
The code you never wrote scales infinitely. Zero bugs, zero CVEs, 100% uptime since forever.
|
||||||
@@ -73,15 +73,13 @@ You know exactly why.
|
|||||||
|
|
||||||
## Numbers
|
## Numbers
|
||||||
|
|
||||||
5 coding tasks, same agent with and without ponytail: **−16% tokens, ~4× faster, 293 → 47 lines.** The 246 lines nobody wrote have never caused an incident.
|
Six tasks: streaming log parser, atomic file sync, notification dispatcher, validation engine, auth module, concurrent money ledger. One spec each, one fresh agent per arm, same model. Three arms: no skill, the [caveman](https://github.com/JuliusBrussee/caveman) skill, and ponytail. Every arm passes the same adversarial security and concurrency probes. Then the agreement ends:
|
||||||
|
|
||||||
Then six harder tasks — streaming parsers, atomic file sync, auth, a concurrent money ledger — against a no-skill control and the [caveman](https://github.com/JuliusBrussee/caveman) skill, with adversarial security and concurrency probes:
|
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="assets/benchmark-loc.svg" width="860" alt="Lines of code per task: ponytail 490 total vs caveman 1,440 vs control 2,943 — all passing the same adversarial probes">
|
<img src="assets/benchmark-loc.svg" width="860" alt="Lines of code per task: ponytail 490 total vs caveman 1,440 vs no-skill control 3,629, all passing the same adversarial probes">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
Everyone passes every probe. Ponytail does it with a third of caveman's code, a sixth of control's, and was ~5× cheaper when the surprise feature request landed (96 lines changed vs 487 for control). Every shortcut is marked in-code with its upgrade path. Data: [benchmarks/](benchmarks/).
|
**47% fewer tokens than the no-skill agent. 3× faster. A seventh of the code.** The 3,139 lines nobody wrote have never caused an incident. When a surprise feature request hit two of the tasks, ponytail extended in 96 changed lines; caveman needed 413, the no-skill agent 1,115. Every shortcut ponytail took is marked in the code with a `ponytail:` comment naming its upgrade path. Data: [benchmarks/](benchmarks/).
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
+15
-15
@@ -1,11 +1,11 @@
|
|||||||
<svg viewBox="0 0 860 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
<svg viewBox="0 0 860 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
||||||
<title>Non-blank lines of code per task — control vs caveman vs ponytail</title>
|
<title>Non-blank lines of code per task: control vs caveman vs ponytail</title>
|
||||||
|
|
||||||
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Six tasks. Adversarial probes: everyone passes. Lines of code: not everyone.</text>
|
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Six tasks. Adversarial probes: everyone passes. Lines of code: not everyone.</text>
|
||||||
|
|
||||||
<!-- legend -->
|
<!-- legend -->
|
||||||
<rect x="20" y="42" width="12" height="12" rx="2" fill="#8b949e"/>
|
<rect x="20" y="42" width="12" height="12" rx="2" fill="#8b949e"/>
|
||||||
<text x="38" y="53" font-size="13" fill="#8b949e">Control (no skill) · 2,943 total</text>
|
<text x="38" y="53" font-size="13" fill="#8b949e">Control (no skill) · 3,629 total</text>
|
||||||
<rect x="250" y="42" width="12" height="12" rx="2" fill="#d9822b"/>
|
<rect x="250" y="42" width="12" height="12" rx="2" fill="#d9822b"/>
|
||||||
<text x="268" y="53" font-size="13" fill="#8b949e">Caveman · 1,440</text>
|
<text x="268" y="53" font-size="13" fill="#8b949e">Caveman · 1,440</text>
|
||||||
<rect x="420" y="42" width="12" height="12" rx="2" fill="#2da44e"/>
|
<rect x="420" y="42" width="12" height="12" rx="2" fill="#2da44e"/>
|
||||||
@@ -13,8 +13,8 @@
|
|||||||
|
|
||||||
<!-- Task A -->
|
<!-- Task A -->
|
||||||
<text x="112" y="117" font-size="12" fill="#8b949e" text-anchor="end">log-analysis CLI</text>
|
<text x="112" y="117" font-size="12" fill="#8b949e" text-anchor="end">log-analysis CLI</text>
|
||||||
<rect x="120" y="90" width="679" height="13" rx="2" fill="#8b949e"/>
|
<rect x="120" y="90" width="662" height="13" rx="2" fill="#8b949e"/>
|
||||||
<text x="805" y="101" font-size="11" fill="#8b949e">970</text>
|
<text x="788" y="101" font-size="11" fill="#8b949e">946</text>
|
||||||
<rect x="120" y="106" width="198" height="13" rx="2" fill="#d9822b"/>
|
<rect x="120" y="106" width="198" height="13" rx="2" fill="#d9822b"/>
|
||||||
<text x="324" y="117" font-size="11" fill="#d9822b">283</text>
|
<text x="324" y="117" font-size="11" fill="#d9822b">283</text>
|
||||||
<rect x="120" y="122" width="102" height="13" rx="2" fill="#2da44e"/>
|
<rect x="120" y="122" width="102" height="13" rx="2" fill="#2da44e"/>
|
||||||
@@ -22,8 +22,8 @@
|
|||||||
|
|
||||||
<!-- Task B -->
|
<!-- Task B -->
|
||||||
<text x="112" y="177" font-size="12" fill="#8b949e" text-anchor="end">file sync</text>
|
<text x="112" y="177" font-size="12" fill="#8b949e" text-anchor="end">file sync</text>
|
||||||
<rect x="120" y="150" width="411" height="13" rx="2" fill="#8b949e"/>
|
<rect x="120" y="150" width="459" height="13" rx="2" fill="#8b949e"/>
|
||||||
<text x="537" y="161" font-size="11" fill="#8b949e">587</text>
|
<text x="585" y="161" font-size="11" fill="#8b949e">656</text>
|
||||||
<rect x="120" y="166" width="160" height="13" rx="2" fill="#d9822b"/>
|
<rect x="120" y="166" width="160" height="13" rx="2" fill="#d9822b"/>
|
||||||
<text x="286" y="177" font-size="11" fill="#d9822b">228</text>
|
<text x="286" y="177" font-size="11" fill="#d9822b">228</text>
|
||||||
<rect x="120" y="182" width="69" height="13" rx="2" fill="#2da44e"/>
|
<rect x="120" y="182" width="69" height="13" rx="2" fill="#2da44e"/>
|
||||||
@@ -31,8 +31,8 @@
|
|||||||
|
|
||||||
<!-- Task C -->
|
<!-- Task C -->
|
||||||
<text x="112" y="237" font-size="12" fill="#8b949e" text-anchor="end">notification dispatcher</text>
|
<text x="112" y="237" font-size="12" fill="#8b949e" text-anchor="end">notification dispatcher</text>
|
||||||
<rect x="120" y="210" width="508" height="13" rx="2" fill="#8b949e"/>
|
<rect x="120" y="210" width="566" height="13" rx="2" fill="#8b949e"/>
|
||||||
<text x="634" y="221" font-size="11" fill="#8b949e">726</text>
|
<text x="692" y="221" font-size="11" fill="#8b949e">808</text>
|
||||||
<rect x="120" y="226" width="277" height="13" rx="2" fill="#d9822b"/>
|
<rect x="120" y="226" width="277" height="13" rx="2" fill="#d9822b"/>
|
||||||
<text x="403" y="237" font-size="11" fill="#d9822b">396</text>
|
<text x="403" y="237" font-size="11" fill="#d9822b">396</text>
|
||||||
<rect x="120" y="242" width="51" height="13" rx="2" fill="#2da44e"/>
|
<rect x="120" y="242" width="51" height="13" rx="2" fill="#2da44e"/>
|
||||||
@@ -40,8 +40,8 @@
|
|||||||
|
|
||||||
<!-- Task D -->
|
<!-- Task D -->
|
||||||
<text x="112" y="297" font-size="12" fill="#8b949e" text-anchor="end">validation engine</text>
|
<text x="112" y="297" font-size="12" fill="#8b949e" text-anchor="end">validation engine</text>
|
||||||
<rect x="120" y="270" width="240" height="13" rx="2" fill="#8b949e"/>
|
<rect x="120" y="270" width="474" height="13" rx="2" fill="#8b949e"/>
|
||||||
<text x="366" y="281" font-size="11" fill="#8b949e">343</text>
|
<text x="600" y="281" font-size="11" fill="#8b949e">677</text>
|
||||||
<rect x="120" y="286" width="153" height="13" rx="2" fill="#d9822b"/>
|
<rect x="120" y="286" width="153" height="13" rx="2" fill="#d9822b"/>
|
||||||
<text x="279" y="297" font-size="11" fill="#d9822b">218</text>
|
<text x="279" y="297" font-size="11" fill="#d9822b">218</text>
|
||||||
<rect x="120" y="302" width="49" height="13" rx="2" fill="#2da44e"/>
|
<rect x="120" y="302" width="49" height="13" rx="2" fill="#2da44e"/>
|
||||||
@@ -49,8 +49,8 @@
|
|||||||
|
|
||||||
<!-- Task E -->
|
<!-- Task E -->
|
||||||
<text x="112" y="357" font-size="12" fill="#8b949e" text-anchor="end">auth module</text>
|
<text x="112" y="357" font-size="12" fill="#8b949e" text-anchor="end">auth module</text>
|
||||||
<rect x="120" y="330" width="109" height="13" rx="2" fill="#8b949e"/>
|
<rect x="120" y="330" width="182" height="13" rx="2" fill="#8b949e"/>
|
||||||
<text x="235" y="341" font-size="11" fill="#8b949e">155</text>
|
<text x="308" y="341" font-size="11" fill="#8b949e">260</text>
|
||||||
<rect x="120" y="346" width="104" height="13" rx="2" fill="#d9822b"/>
|
<rect x="120" y="346" width="104" height="13" rx="2" fill="#d9822b"/>
|
||||||
<text x="230" y="357" font-size="11" fill="#d9822b">148</text>
|
<text x="230" y="357" font-size="11" fill="#d9822b">148</text>
|
||||||
<rect x="120" y="362" width="34" height="13" rx="2" fill="#2da44e"/>
|
<rect x="120" y="362" width="34" height="13" rx="2" fill="#2da44e"/>
|
||||||
@@ -58,12 +58,12 @@
|
|||||||
|
|
||||||
<!-- Task F -->
|
<!-- Task F -->
|
||||||
<text x="112" y="417" font-size="12" fill="#8b949e" text-anchor="end">money ledger</text>
|
<text x="112" y="417" font-size="12" fill="#8b949e" text-anchor="end">money ledger</text>
|
||||||
<rect x="120" y="390" width="113" height="13" rx="2" fill="#8b949e"/>
|
<rect x="120" y="390" width="197" height="13" rx="2" fill="#8b949e"/>
|
||||||
<text x="239" y="401" font-size="11" fill="#8b949e">162</text>
|
<text x="323" y="401" font-size="11" fill="#8b949e">282</text>
|
||||||
<rect x="120" y="406" width="117" height="13" rx="2" fill="#d9822b"/>
|
<rect x="120" y="406" width="117" height="13" rx="2" fill="#d9822b"/>
|
||||||
<text x="243" y="417" font-size="11" fill="#d9822b">167</text>
|
<text x="243" y="417" font-size="11" fill="#d9822b">167</text>
|
||||||
<rect x="120" y="422" width="38" height="13" rx="2" fill="#2da44e"/>
|
<rect x="120" y="422" width="38" height="13" rx="2" fill="#2da44e"/>
|
||||||
<text x="164" y="433" font-size="11" fill="#2da44e" font-weight="600">54</text>
|
<text x="164" y="433" font-size="11" fill="#2da44e" font-weight="600">54</text>
|
||||||
|
|
||||||
<text x="120" y="458" font-size="11" fill="#8b949e" opacity="0.8">Non-blank LOC, AST-counted. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12.</text>
|
<text x="120" y="458" font-size="11" fill="#8b949e" opacity="0.8">Non-blank LOC, AST-counted. Same model, same specs, one fresh agent per arm. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12.</text>
|
||||||
</svg>
|
</svg>
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 4.2 KiB After Width: | Height: | Size: 4.2 KiB |
@@ -91,6 +91,28 @@ locks at 3× the LOC).
|
|||||||
with its ceiling; Caveman's code marks only spec-allowed simulated transports,
|
with its ceiling; Caveman's code marks only spec-allowed simulated transports,
|
||||||
and its design trade-offs live in the chat report, invisible to a later reviewer.
|
and its design trade-offs live in the chat report, invisible to a later reviewer.
|
||||||
|
|
||||||
|
## Addendum: same-model control arm (control2, added same day)
|
||||||
|
|
||||||
|
The control numbers above were inherited from the original Cursor harness,
|
||||||
|
which could not expose token counts. Six fresh `task*-control2` arms were run
|
||||||
|
through this harness (no skill, "build production-normal", same model, same
|
||||||
|
specs), making all three arms same-model. Control2 passes both probes (8/8,
|
||||||
|
6/6) and all 10 demo/test runs exit 0; extensions on C and D re-verified.
|
||||||
|
|
||||||
|
| Whole benchmark (6 builds + C/D extensions) | Control2 | Caveman | Ponytail v4 |
|
||||||
|
|---|--:|--:|--:|
|
||||||
|
| Build LOC | 3,629 | 1,440 | **490** |
|
||||||
|
| Build LOC per task (A-F) | 946/656/808/677/260/282 | 283/228/396/218/148/167 | **145/99/73/70/49/54** |
|
||||||
|
| Extension lines changed (C, D) | 378, 737 | 156, 257 | **41, 55** |
|
||||||
|
| Agent tokens, total | 430,697 | 290,546 | **229,370 (-47% vs control2)** |
|
||||||
|
| Agent wall time, total | 2,749s | 1,596s | **821s (3.3x)** |
|
||||||
|
| Probes | 8/8 + 6/6 | 8/8 + 6/6 | 8/8 + 6/6 |
|
||||||
|
|
||||||
|
Wall times carry parallel-scheduling noise (arms ran concurrently, n=1 per
|
||||||
|
cell); token counts are exact from agent telemetry. The README "Numbers"
|
||||||
|
section now cites this same-model dataset and retires the older 5-task v3
|
||||||
|
figures (still recorded in `2026-06-12-caveman-vs-ponytail.md`).
|
||||||
|
|
||||||
## Residual (honest notes)
|
## Residual (honest notes)
|
||||||
|
|
||||||
- A's spike stats still use observed-hours-only mean+3σ rather than a
|
- A's spike stats still use observed-hours-only mean+3σ rather than a
|
||||||
|
|||||||
Reference in New Issue
Block a user