feat(benchmarks): add correctness assertion (#31)
* feat(benchmarks): add correctness assertion - proves less code is not broken code The existing benchmark measures lines-of-code (loc.js) but never checks whether the generated code actually works. This adds a functional correctness gate (correctness.js) that extracts code from fenced blocks and runs per-task checks: - email validator: spawns Python, asserts accept/reject on 5 inputs - debounce: spawns Node, asserts delayed execution + reset on re-call - csv sum: spawns Python with a test CSV, asserts correct total (351) - countdown (React): structural check (useState + useEffect + decrement) - rate limiter (FastAPI): structural check (limit logic + framework usage) 12 unit tests (node:test) cover good/bad outputs for every task plus the unknown-task edge case. Existing tests and rule-copy checks unaffected. * fix: address review feedback - csv check: use regex lookaround instead of substring match to prevent false positives (e.g. 13510 containing '351') - ratelimit: fix operator precedence in block finder by adding parens around the || inside the !b.lang guard - README: note that React/FastAPI checks are structural only, add prerequisites section (Python 3, pandas, Node.js 18+) - test: add regression test for csv substring false positive
This commit is contained in:
@@ -29,6 +29,9 @@ defaultTest:
|
||||
- type: javascript
|
||||
value: file://loc.js
|
||||
metric: code_loc
|
||||
- type: javascript
|
||||
value: file://correctness.js
|
||||
metric: correct
|
||||
|
||||
tests:
|
||||
- vars: { task: "Write me a Python function that validates email addresses." }
|
||||
|
||||
Reference in New Issue
Block a user