The benchmark is single-shot (one prompt, one completion); it does not measure
a real multi-turn agent session, where the ruleset re-injects and the ladder
deliberates every turn. Adds that caveat to the README, and corrects the
benchmarks/README note that claimed caching widens the gap "in ponytail's
favor" (unverified, and a measured agentic A/B in #121 found the opposite can
happen). Per-session cost can land either way.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus):
ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%.
The direction holds, both ends came in a few points lower. Updates the README
headline and body, the benchmark chart subtitle, and the benchmarks/README cost
table, and adds a dated results doc with full method.
Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On
OpenAI reasoning models ponytail costs more, not less, so the claim stays
Claude-scoped. Gemini run pending a fresh-quota day.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(benchmarks): correctness gate scores unfenced code; fix debounce task
The `correct` gate under-reported correctness for terse models, the likely
source of "Ponytail degrades models" reports (issue #65):
- extractBlocks() only matched fenced code blocks, so bare/unfenced code
scored an automatic fail even when correct. Now falls back to the whole
response as one block (and tolerates CRLF). Debounce detection also accepts
unfenced arrow functions.
- The debounce task asked to "add debounce to a search input" but the check
expected a reusable debounce(fn, delay) util, failing correct inline answers.
Task reworded to the deliverable the check verifies.
Adds correctness.test.js (regression guard) and a GPT-mini repro config plus
results writeup: on a clean n=20 run, the reported gpt-4.1-mini drop (10/15)
does not reproduce (100/100). The LOC win (~halved) holds.
README repro fixed: promptfoo needs --env-file ../.env (reads cwd, not root).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(benchmarks): add robustness audit — ponytail vs baseline on edge cases
Answers the real question behind #65: does ponytail's push for the shortest
solution make weak models produce wrong code on edge cases?
robustness-audit.js: 16 self-verifying tasks (12 algorithmic edge-case traps +
4 validators). Each check ships a known-good and known-lazy-wrong reference that
must pass/fail before any model output is scored (--selftest, 16/16).
Findings (gpt-4.1-mini + gpt-5.4-mini, baseline vs ponytail): parity on every
edge-case trap on both models. The one measured soft spot is gpt-5.4-mini email
(~4-5%, reaches for parseaddr). A sharpened SKILL.md validation rule had no
reliable effect in an n=100 A/B (96% vs 95%), so it was not shipped — the
tendency is model-level, not skill-level. Full writeup in results/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(benchmarks): email slip is provider-specific — 100% on Claude
High-n cross-provider follow-up to the robustness audit. The one ponytail
soft spot (email validation via parseaddr) splits by provider, not model size:
- Claude (haiku/sonnet/opus): 100% under ponytail, n=40 each — and ponytail
beats baseline (unconstrained Sonnet over-engineers into an always-truthy
dict, 0/40; ponytail writes a clean validator).
- OpenAI (gpt-4.1-mini..gpt-5.5): slips at every size under ponytail
(~79-98%), baseline ~100%. The parseaddr reflex lives in OpenAI training.
Not fixable by skill text: 8 distinct SKILL.md edits (incl. an n=100 A/B,
96% vs 95%) all scored <= current, several worse, all bloated LOC. Nothing
shipped. SKILL.md unchanged.
Conclusion: on ponytail's target platform (Claude) email is 100%; the GPT
slip is a documented cross-provider transfer quirk. Adds model-email.js /
claude-email.js to reproduce the tables. Writeup updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(benchmarks): correct misleading Sonnet baseline 0 percent
The Sonnet baseline 0/40 on email is a return-type artifact, not a logic
failure: unconstrained Sonnet returns a dict {is_valid, message} instead of a
bool, so the bool-contract gate scores every case as accepted. Read dict-aware
via is_valid, its logic is ~75% correct (9/12). Reframed honestly so we are not
presenting 0 vs 100 as a clean win; ponytail still wins (clean 100% bool) but
the point is over-engineered return type, not total failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the local benchmark LOC counter (counted only fenced code, scored bare output 0), makes summary output ASCII-safe (a Unicode arrow crashed the script on Windows cp1252), gitignores generated artifacts, and refreshes the llama3.2 writeup with n=5 data showing the LOC effect is within the noise floor. Follow-up to #63. Verified live.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The benchmark harness hardcoded `python`, which is missing on macOS and
many Linux images. Probe python3 first, add npm test, and run checks in
GitHub Actions so regressions are caught on every PR.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: refine ruleset from a full-project field review
A reviewer ran ponytail across a 9-phase rewrite (protocol, PC app, simulator,
RPi daemon, ESP32 firmware) and flagged three gaps. All three land in SKILL.md
and propagate to AGENTS.md + the rule copies:
- Promote the one-runnable-check rule to a headline ("Lazy code without its
check is unfinished"), enforced as a check-rule-copies invariant.
- Hardware carve-out in "When NOT to be lazy": a real device is never the spec
ideal (clock drift, sensor offset), leave the calibration knob.
- Clarify the Output rule: explanation the user explicitly asked for is not
debt, only unrequested prose is.
Fallback instructions kept in sync. Rule-copy check + tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add a behavior gate proving the refinements actually fire
The refinements were verified as injected text, but injected != behavioral.
This adds a behavior eval that probes each refined rule on a task that should
trigger it:
- hardware -> does the output leave a calibration knob?
- explanation -> when a write-up is explicitly requested, is it given in full?
- onecheck -> is a runnable check left behind?
benchmarks/behavior.yaml runs the probes (baseline vs ponytail arm); the
grader benchmarks/behavior.js is proven by tests/behavior.test.js (8 cases,
RED/GREEN, no API key, runs in CI). Live-confirmed: the model under the
current ruleset passes all three gates, graded by the same grader.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(benchmarks): add correctness assertion - proves less code is not broken code
The existing benchmark measures lines-of-code (loc.js) but never checks
whether the generated code actually works. This adds a functional
correctness gate (correctness.js) that extracts code from fenced blocks
and runs per-task checks:
- email validator: spawns Python, asserts accept/reject on 5 inputs
- debounce: spawns Node, asserts delayed execution + reset on re-call
- csv sum: spawns Python with a test CSV, asserts correct total (351)
- countdown (React): structural check (useState + useEffect + decrement)
- rate limiter (FastAPI): structural check (limit logic + framework usage)
12 unit tests (node:test) cover good/bad outputs for every task plus the
unknown-task edge case. Existing tests and rule-copy checks unaffected.
* fix: address review feedback
- csv check: use regex lookaround instead of substring match to prevent
false positives (e.g. 13510 containing '351')
- ratelimit: fix operator precedence in block finder by adding parens
around the || inside the !b.lang guard
- README: note that React/FastAPI checks are structural only, add
prerequisites section (Python 3, pandas, Node.js 18+)
- test: add regression test for csv substring false positive
Swap em dashes for commas/colons/periods in the README, skills, AGENTS.md and
its five rule copies, examples, command files, and benchmark README. Rule
copies stay in sync (same edit applied to all) and the invariant guard passes.
Left untouched on purpose: the vendored caveman SKILL.md (verbatim third-party
text), the dated benchmark writeups in results/ (historical records), and
.js code comments.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Commit a promptfoo harness (config + arm prompts + LOC metric + vendored
caveman SKILL) so anyone can re-run the comparison: no-skill vs caveman vs
ponytail, across Haiku / Sonnet / Opus, 10 runs per cell, median reported.
Replace the old unreproducible 6-task chart with assets/benchmark-3model.svg
from this run, and reframe the README to the reproducible numbers: ponytail
writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a
no-skill agent on every model. benchmarks/README.md carries the median tables
and the reproduce command. Drops nothing that is not measured.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Six no-skill control arms re-run through the same harness so all three
arms share one model. README Numbers section and chart now cite the
complete dataset: -47% tokens, 3x faster, 490 vs 3,629 LOC, extension
96 vs 1,115 lines, probes green everywhere. Em dashes removed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probes 8/8 + 6/6 both arms, LOC 490 vs 1440, extension cost 41/55 vs
156/257, all six v4 arms ship a runnable check with no bloat creep.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Skill preaching minimalism was 2x caveman length. Smaller file cuts
per-read and per-session-injection cost. Benchmark: beats caveman on
all areas now — 135.7k vs 138.4k tokens, 127s vs 136s, 47 vs 117 loc.
v1 lost to caveman on tokens/time despite minimal code: it wrote
essays defending each simplification. v2 caps explanation at three
lines and ships the lazy version instead of stalling on necessity
questions. Benchmark: 136.6k tok vs caveman 138.4k, code 47 vs 117
lines across 5 tasks.
