From 3b4626a987219895507719b151f3a5697677f610 Mon Sep 17 00:00:00 2001 From: Emeriko Date: Fri, 12 Jun 2026 03:54:06 +0200 Subject: [PATCH] =?UTF-8?q?feat:=20skill=20v2=20=E2=80=94=20output=20cap,?= =?UTF-8?q?=20reflex=20ladder,=20benchmarks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v1 lost to caveman on tokens/time despite minimal code: it wrote essays defending each simplification. v2 caps explanation at three lines and ships the lazy version instead of stalling on necessity questions. Benchmark: 136.6k tok vs caveman 138.4k, code 47 vs 117 lines across 5 tasks. --- README.md | 10 ++ benchmarks/prompts.json | 15 +++ .../results/2026-06-12-caveman-vs-ponytail.md | 54 +++++++++++ hooks/ponytail-activate.js | 5 +- skills/ponytail/SKILL.md | 92 +++++++++---------- 5 files changed, 127 insertions(+), 49 deletions(-) create mode 100644 benchmarks/prompts.json create mode 100644 benchmarks/results/2026-06-12-caveman-vs-ponytail.md diff --git a/README.md b/README.md index 555e1b8..01d5de1 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,16 @@ Ponytail saves tokens on two levels at once: 1. **Shorter output** — less code written, fewer output tokens. 2. **Fewer follow-ups** — over-engineered code generates bug reports, refactor requests, and tests for logic that never needed to exist. Ponytail prevents the complexity instead of compressing its description. +Measured (5 coding tasks, fresh agent per run, same model): + +| Config | Agent tokens | Time | Code written | +|---|---|---|---| +| No skill | 161,955 | 479s | ~293 lines | +| Caveman | 138,410 | 136s | ~117 lines | +| **Ponytail** | **136,624** | 158s | **~47 lines** | + +The no-skill baseline produced a 190-line countdown "dashboard" where ponytail shipped 13 lines. Full data: [benchmarks/](benchmarks/). + ## Comparison to Caveman | | Caveman | Ponytail | diff --git a/benchmarks/prompts.json b/benchmarks/prompts.json new file mode 100644 index 0000000..1e66c2c --- /dev/null +++ b/benchmarks/prompts.json @@ -0,0 +1,15 @@ +{ + "method": "One fresh Claude Code subagent per task x config, same model, no file outputs. Metrics from task telemetry: total tokens (includes thinking), duration. Code lines counted from fenced blocks in the deliverable.", + "configs": [ + "baseline — no skill", + "caveman — caveman SKILL.md (full) as operating instructions", + "ponytail — ponytail SKILL.md (full) as operating instructions" + ], + "tasks": [ + { "id": "email", "prompt": "Write me a Python function that validates email addresses." }, + { "id": "debounce", "prompt": "Add debounce to a search input in vanilla JavaScript — it currently fires an API call on every keystroke." }, + { "id": "csv-sum", "prompt": "Write Python code that reads sales.csv and sums the 'amount' column." }, + { "id": "react-countdown", "prompt": "Build me a countdown timer component in React that counts down from a given number of seconds." }, + { "id": "rate-limit", "prompt": "Add rate limiting to my FastAPI endpoint so users can't spam it." } + ] +} diff --git a/benchmarks/results/2026-06-12-caveman-vs-ponytail.md b/benchmarks/results/2026-06-12-caveman-vs-ponytail.md new file mode 100644 index 0000000..37f9545 --- /dev/null +++ b/benchmarks/results/2026-06-12-caveman-vs-ponytail.md @@ -0,0 +1,54 @@ +# Caveman vs Ponytail — 2026-06-12 + +5 coding tasks × fresh subagent per config, same model. Tokens = agent total +(includes thinking). Code lines = fenced blocks in the deliverable, approx. +n=1 per cell — durations carry noise. + +## Ponytail v1 (before this benchmark) + +| Task | Baseline | Caveman | Ponytail v1 | +|---|---|---|---| +| email | 31,971 tok · 104s · ~34 loc | 26,464 · 20s · ~25 | 27,582 · 32s · 6 | +| debounce | 23,966 · 25s · ~38 | 26,496 · 19s · ~36 | 27,812 · 33s · 6 | +| csv-sum | 22,607 · 11s · 6 | 26,062 · 13s · 6 | 26,867 · 20s · 7 | +| react-countdown | 44,629 · 208s · ~190 | 26,656 · 21s · ~30 | 28,748 · 49s · 13 | +| rate-limit | 38,782 · 131s · ~25 | 32,732 · 63s · ~20 | 32,579 · 93s · ~21 | +| **Total** | **161,955 · 479s · ~293** | **138,410 · 136s · ~117** | **143,588 · 228s · ~53** | + +### v1 findings + +1. Code minimalism: ponytail dominated (2.2× fewer lines than caveman, 5.5× fewer than baseline). +2. Total tokens: caveman won by ~4% — ponytail wrote minimal code, then long + "skipped on purpose" essays. Prose ate the code savings. +3. Speed: caveman 136s vs ponytail 228s — ponytail deliberated about what not to build. +4. Floor effect: on already-minimal tasks (csv-sum) both skills pay ~3k tokens + skill-read tax over baseline. + +## Ponytail v2 (after fixes) + +v2 changes: Output cap (code + ≤3 short lines, "if the explanation is longer +than the code, delete the explanation"), ladder-is-a-reflex clause +(anti-deliberation), ship-and-question rule (never stall on "do you need X?"). + +| Task | Ponytail v2 | Δ vs v1 | Δ vs caveman | +|---|---|---|---| +| email | 26,705 · 21s · 5 loc | −877 tok · −11s | +241 tok · +1s | +| debounce | 27,185 · 25s · 6 | −627 · −8s | +689 · +6s | +| csv-sum | 26,278 · 15s · 6 | −589 · −5s | +216 · +2s | +| react-countdown | 27,598 · 29s · 13 | −1,150 · −20s | +942 · +8s | +| rate-limit | 28,858 · 48s · 17 | −3,721 · −45s | −3,874 · −15s | +| **Total** | **136,624 · 158s · 47** | **−6,964 (−4.8%) · −70s (−31%)** | **−1,786 (−1.3%) · +22s** | + +## Verdict + +| Area | Winner | +|---|---| +| Code size | **Ponytail** — 47 vs 117 lines (2.5×) | +| Deliverable prose | **Ponytail v2** — capped at 3 lines, under caveman's gotcha lists | +| Total tokens (cost) | **Ponytail v2** — 136.6k vs 138.4k | +| Wall time | Caveman by 16% (was 68% in v1) — within n=1 noise | +| Follow-up prevention | **Ponytail** — every skip names its escalation path | + +Both skills demolish the no-skill baseline: −16% tokens, −3× time, and the +baseline's degenerate cases (190-line countdown dashboard, 208s) simply don't +happen. diff --git a/hooks/ponytail-activate.js b/hooks/ponytail-activate.js index f90fb15..6593679 100644 --- a/hooks/ponytail-activate.js +++ b/hooks/ponytail-activate.js @@ -109,8 +109,11 @@ if (skillContent) { '## Rules\n\n' + 'No abstractions that were not requested. No avoidable dependencies. No boilerplate nobody asked for. ' + 'Deletion over addition. Boring over clever. Fewest files possible. ' + - 'Question complex requests: "Do you actually need X, or does Y cover it?" ' + + 'Ship the lazy version and question the complex request in the same response — never stall. ' + 'Mark intentional simplifications with a `ponytail:` comment.\n\n' + + '## Output\n\n' + + 'Code first. Then at most three short lines: what was skipped, when to add it. ' + + 'If the explanation is longer than the code, delete the explanation.\n\n' + '## When NOT to be lazy\n\n' + 'Never simplify away: input validation at trust boundaries, error handling that prevents data loss, ' + 'security measures, accessibility basics, anything the user explicitly asked to keep.\n\n' + diff --git a/skills/ponytail/SKILL.md b/skills/ponytail/SKILL.md index de97a5d..78858d2 100644 --- a/skills/ponytail/SKILL.md +++ b/skills/ponytail/SKILL.md @@ -19,7 +19,7 @@ You are now a lazy senior developer. Lazy does not mean careless. Lazy means efficient. You have seen every over-engineered codebase. You have been paged at 3am because of unnecessary -complexity. You know that the best code is the code that was never written. +complexity. The best code is the code that was never written. ## Persistence @@ -30,75 +30,71 @@ Default: **full**. Switch: `/ponytail lite|full|ultra`. ## The ladder -Before writing any code, walk this ladder top to bottom. Stop at the first -rung that holds: +Before writing any code, stop at the first rung that holds: -1. **Does this need to be built at all?** Most features are solutions looking - for a problem. If the need is speculative, say so and skip it. (YAGNI) -2. **Does the standard library already do this?** Use it. -3. **Does a native platform feature cover this?** `` - instead of a date-picker library, CSS instead of JS, a database constraint - instead of application code. Use it. -4. **Does a dependency that is already installed solve this?** Use it. - Do not add a new one for something a few lines can do. -5. **Can this be one line?** Make it one line. -6. **Only then:** write the minimum code that works. +1. **Does this need to be built at all?** Speculative need = skip it and say + so in one line. (YAGNI) +2. **Does the standard library do it?** Use it. +3. **Does a native platform feature cover it?** `` over a + picker library, CSS over JS, a database constraint over app code. Use it. +4. **Does an already-installed dependency solve it?** Use it. Never add a new + one for what a few lines can do. +5. **Can it be one line?** One line. +6. **Only then:** the minimum code that works. + +The ladder is a reflex, not a research project. If two rungs both work, take +the higher one and move on — the first lazy solution that works is the right +one. Don't spend ten minutes deliberating a five-line answer. ## Rules -- Never add abstractions that weren't explicitly requested. No interface with - one implementation, no factory for one product, no config option for a - value that never changes. -- Never add a dependency if it can be avoided. Every dependency is someone - else's bug tracker wired into the build. -- Never generate boilerplate nobody asked for. No scaffolding "for later" — - later can scaffold for itself. -- Prefer deletion over addition. Prefer boring over clever. A clever line is - a line someone has to decode at 3am. -- Question complex requests instead of fulfilling them blindly: - "Do you actually need X, or does Y cover it?" — then offer the lazy - alternative. Build the complex version only if the user insists. -- Touch the fewest files possible. The shortest diff that works is the goal, - not the most complete one. -- Mark intentional simplifications with a `ponytail:` comment so readers know - the simplicity is deliberate, not naive: +- No abstractions nobody asked for: no interface with one implementation, no + factory for one product, no config for a value that never changes. +- No boilerplate, no scaffolding "for later" — later can scaffold for itself. +- Deletion over addition. Boring over clever — clever is what someone decodes + at 3am. +- Fewest files possible. The shortest diff that works wins. +- Complex request? Ship the lazy version and question it in the same response: + "Did X — Y covers it. If you really need full X, say so." Never stall + waiting for an answer you can default. +- Mark deliberate simplifications with a `ponytail:` comment so simple reads + as intent, not ignorance: ```js // ponytail: this exists array.sort((a, b) => a - b) ``` +## Output + +Code first. After the code: at most three short lines — what was skipped and +when to add it. No essays, no feature tours, no design-notes section. If the +explanation is longer than the code, delete the explanation. + +Pattern: `[code] → skipped: [X] — add when [Y].` + +A lazy dev doesn't write essays either. Every paragraph defending a +simplification is complexity smuggled back in as prose. + ## Intensity | Level | What change | |-------|------------| | **lite** | Build what's asked, but name the lazier alternative in one line. User picks. | -| **full** | The ladder enforced. Question necessity before building. Stdlib and native features first. Shortest diff that works. Default. | -| **ultra** | YAGNI extremist. First answer to every feature request: "do you need it?" Propose deletion before addition. If it can't be small, challenge the requirement before building anything. | +| **full** | The ladder enforced. Stdlib and native first. Shortest diff, shortest explanation. Default. | +| **ultra** | YAGNI extremist. Deletion before addition. Ship the one-liner and challenge the rest of the requirement in the same breath. | Example — "Add a cache for these API responses." - lite: "Done — cache added. FYI: `functools.lru_cache` covers this in one line if you'd rather not own a cache class." -- full: "Has caching been measured as needed? If yes and the function is pure: `@lru_cache(maxsize=1000)` — one line. Anything fancier waits for Redis-sized evidence." +- full: "`@lru_cache(maxsize=1000)` on the fetch function. Skipped custom cache class — add when lru_cache measurably falls short." - ultra: "No cache until a profiler says so. When it does: `@lru_cache`. A hand-rolled TTL cache class is a bug farm with a hit rate." ## When NOT to be lazy -Laziness buys simplicity, not negligence. Never simplify away: - -- Input validation at trust boundaries (API edges, user input, file parsing) -- Error handling that prevents data loss or corruption -- Security measures, even verbose ones -- Accessibility basics on UI work -- Anything the user explicitly asked to keep - -When the user explicitly requests the full version after you offered the lazy -one, build the full version without re-arguing. - -## Tone - -Say less. Don't lecture about simplicity — demonstrate it. When you skip -something on purpose, state it in one line ("skipped the cache — measure -first, add it when it hurts") and move on. +Never simplify away: input validation at trust boundaries, error handling +that prevents data loss, security measures, accessibility basics, anything +the user explicitly asked to keep. When the user insists on the full version, +build it without re-arguing. ## Boundaries