diff --git a/README.md b/README.md index 5bfba32..713db6f 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@

- Ponytail — the lazy senior dev + Ponytail, the lazy senior dev

Ponytail

@@ -63,7 +63,7 @@ Cursor, Windsurf, Cline, Copilot, Aider: copy the matching rules file from this No. **What if I really need the 120-line cache class?** -You don't. Insist anyway and he'll build it — slowly, correctly, while looking at you. +You don't. Insist anyway and he'll build it. Slowly. Correctly. While looking at you. **Does it scale?** The code you never wrote scales infinitely. Zero bugs, zero CVEs, 100% uptime since forever. @@ -73,15 +73,13 @@ You know exactly why. ## Numbers -5 coding tasks, same agent with and without ponytail: **−16% tokens, ~4× faster, 293 → 47 lines.** The 246 lines nobody wrote have never caused an incident. - -Then six harder tasks — streaming parsers, atomic file sync, auth, a concurrent money ledger — against a no-skill control and the [caveman](https://github.com/JuliusBrussee/caveman) skill, with adversarial security and concurrency probes: +Six tasks: streaming log parser, atomic file sync, notification dispatcher, validation engine, auth module, concurrent money ledger. One spec each, one fresh agent per arm, same model. Three arms: no skill, the [caveman](https://github.com/JuliusBrussee/caveman) skill, and ponytail. Every arm passes the same adversarial security and concurrency probes. Then the agreement ends:

- Lines of code per task: ponytail 490 total vs caveman 1,440 vs control 2,943 — all passing the same adversarial probes + Lines of code per task: ponytail 490 total vs caveman 1,440 vs no-skill control 3,629, all passing the same adversarial probes

-Everyone passes every probe. Ponytail does it with a third of caveman's code, a sixth of control's, and was ~5× cheaper when the surprise feature request landed (96 lines changed vs 487 for control). Every shortcut is marked in-code with its upgrade path. Data: [benchmarks/](benchmarks/). +**47% fewer tokens than the no-skill agent. 3× faster. A seventh of the code.** The 3,139 lines nobody wrote have never caused an incident. When a surprise feature request hit two of the tasks, ponytail extended in 96 changed lines; caveman needed 413, the no-skill agent 1,115. Every shortcut ponytail took is marked in the code with a `ponytail:` comment naming its upgrade path. Data: [benchmarks/](benchmarks/). ## License diff --git a/assets/benchmark-loc.svg b/assets/benchmark-loc.svg index 5da8e8b..be81d52 100644 --- a/assets/benchmark-loc.svg +++ b/assets/benchmark-loc.svg @@ -1,11 +1,11 @@ - Non-blank lines of code per task — control vs caveman vs ponytail + Non-blank lines of code per task: control vs caveman vs ponytail Six tasks. Adversarial probes: everyone passes. Lines of code: not everyone. - Control (no skill) · 2,943 total + Control (no skill) · 3,629 total Caveman · 1,440 @@ -13,8 +13,8 @@ log-analysis CLI - - 970 + + 946 283 @@ -22,8 +22,8 @@ file sync - - 587 + + 656 228 @@ -31,8 +31,8 @@ notification dispatcher - - 726 + + 808 396 @@ -40,8 +40,8 @@ validation engine - - 343 + + 677 218 @@ -49,8 +49,8 @@ auth module - - 155 + + 260 148 @@ -58,12 +58,12 @@ money ledger - - 162 + + 282 167 54 - Non-blank LOC, AST-counted. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12. + Non-blank LOC, AST-counted. Same model, same specs, one fresh agent per arm. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12. diff --git a/benchmarks/results/2026-06-12-v4-hardening-vs-caveman.md b/benchmarks/results/2026-06-12-v4-hardening-vs-caveman.md index bd2196d..49f8d8e 100644 --- a/benchmarks/results/2026-06-12-v4-hardening-vs-caveman.md +++ b/benchmarks/results/2026-06-12-v4-hardening-vs-caveman.md @@ -91,6 +91,28 @@ locks at 3× the LOC). with its ceiling; Caveman's code marks only spec-allowed simulated transports, and its design trade-offs live in the chat report, invisible to a later reviewer. +## Addendum: same-model control arm (control2, added same day) + +The control numbers above were inherited from the original Cursor harness, +which could not expose token counts. Six fresh `task*-control2` arms were run +through this harness (no skill, "build production-normal", same model, same +specs), making all three arms same-model. Control2 passes both probes (8/8, +6/6) and all 10 demo/test runs exit 0; extensions on C and D re-verified. + +| Whole benchmark (6 builds + C/D extensions) | Control2 | Caveman | Ponytail v4 | +|---|--:|--:|--:| +| Build LOC | 3,629 | 1,440 | **490** | +| Build LOC per task (A-F) | 946/656/808/677/260/282 | 283/228/396/218/148/167 | **145/99/73/70/49/54** | +| Extension lines changed (C, D) | 378, 737 | 156, 257 | **41, 55** | +| Agent tokens, total | 430,697 | 290,546 | **229,370 (-47% vs control2)** | +| Agent wall time, total | 2,749s | 1,596s | **821s (3.3x)** | +| Probes | 8/8 + 6/6 | 8/8 + 6/6 | 8/8 + 6/6 | + +Wall times carry parallel-scheduling noise (arms ran concurrently, n=1 per +cell); token counts are exact from agent telemetry. The README "Numbers" +section now cites this same-model dataset and retires the older 5-task v3 +figures (still recorded in `2026-06-12-caveman-vs-ponytail.md`). + ## Residual (honest notes) - A's spike stats still use observed-hours-only mean+3σ rather than a