* feat(benchmarks): add correctness assertion - proves less code is not broken code
The existing benchmark measures lines-of-code (loc.js) but never checks
whether the generated code actually works. This adds a functional
correctness gate (correctness.js) that extracts code from fenced blocks
and runs per-task checks:
- email validator: spawns Python, asserts accept/reject on 5 inputs
- debounce: spawns Node, asserts delayed execution + reset on re-call
- csv sum: spawns Python with a test CSV, asserts correct total (351)
- countdown (React): structural check (useState + useEffect + decrement)
- rate limiter (FastAPI): structural check (limit logic + framework usage)
12 unit tests (node:test) cover good/bad outputs for every task plus the
unknown-task edge case. Existing tests and rule-copy checks unaffected.
* fix: address review feedback
- csv check: use regex lookaround instead of substring match to prevent
false positives (e.g. 13510 containing '351')
- ratelimit: fix operator precedence in block finder by adding parens
around the || inside the !b.lang guard
- README: note that React/FastAPI checks are structural only, add
prerequisites section (Python 3, pandas, Node.js 18+)
- test: add regression test for csv substring false positive
Commit a promptfoo harness (config + arm prompts + LOC metric + vendored
caveman SKILL) so anyone can re-run the comparison: no-skill vs caveman vs
ponytail, across Haiku / Sonnet / Opus, 10 runs per cell, median reported.
Replace the old unreproducible 6-task chart with assets/benchmark-3model.svg
from this run, and reframe the README to the reproducible numbers: ponytail
writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a
no-skill agent on every model. benchmarks/README.md carries the median tables
and the reproduce command. Drops nothing that is not measured.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
at384