wangjunxiao/ponytail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(benchmarks): add correctness assertion (#31)

Browse Source 
* feat(benchmarks): add correctness assertion - proves less code is not broken code

The existing benchmark measures lines-of-code (loc.js) but never checks
whether the generated code actually works. This adds a functional
correctness gate (correctness.js) that extracts code from fenced blocks
and runs per-task checks:

- email validator: spawns Python, asserts accept/reject on 5 inputs
- debounce: spawns Node, asserts delayed execution + reset on re-call
- csv sum: spawns Python with a test CSV, asserts correct total (351)
- countdown (React): structural check (useState + useEffect + decrement)
- rate limiter (FastAPI): structural check (limit logic + framework usage)

12 unit tests (node:test) cover good/bad outputs for every task plus the
unknown-task edge case. Existing tests and rule-copy checks unaffected.

* fix: address review feedback

- csv check: use regex lookaround instead of substring match to prevent
  false positives (e.g. 13510 containing '351')
- ratelimit: fix operator precedence in block finder by adding parens
  around the || inside the !b.lang guard
- README: note that React/FastAPI checks are structural only, add
  prerequisites section (Python 3, pandas, Node.js 18+)
- test: add regression test for csv substring false positive
This commit is contained in:
at384
2026-06-14 23:42:01 +02:00
committed by GitHub
parent 94d231cd32
commit 6d990f8c54
4 changed files with 472 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/README.md
+15
View File
@@ -40,6 +40,21 @@ Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limi
Versus baseline, ponytail writes **80-94% less code**, costs **47-77% less**, and runs **3-6x faster**, on every model.
## Metrics
| File | Metric | Behavior |
|------|--------|----------|
| `loc.js` | `loc` | Measurement - always passes, records line count |
| `correctness.js` | `correct` | Gate - fails if generated code doesn't work |
`correctness.js` extracts fenced code blocks and runs per-task checks (spawns Python/Node for email, debounce, CSV; structural regex for React and FastAPI). A broken one-liner that scores great on LOC will fail on correctness.
> **Note:** The React countdown and FastAPI rate-limit checks are keyword/structural only (no runtime execution), so they verify plausible structure rather than full correctness. The email, debounce, and CSV checks execute the code.
### Prerequisites
Running the benchmark requires **Python 3**, **pandas**, and **Node.js** (18+).
## Notes
- Caveman is a prose-compression skill (it leaves code "normal"), so it lands between baseline and ponytail on code size and wins mainly on prose tokens.