at384
6d990f8c54
* feat(benchmarks): add correctness assertion - proves less code is not broken code The existing benchmark measures lines-of-code (loc.js) but never checks whether the generated code actually works. This adds a functional correctness gate (correctness.js) that extracts code from fenced blocks and runs per-task checks: - email validator: spawns Python, asserts accept/reject on 5 inputs - debounce: spawns Node, asserts delayed execution + reset on re-call - csv sum: spawns Python with a test CSV, asserts correct total (351) - countdown (React): structural check (useState + useEffect + decrement) - rate limiter (FastAPI): structural check (limit logic + framework usage) 12 unit tests (node:test) cover good/bad outputs for every task plus the unknown-task edge case. Existing tests and rule-copy checks unaffected. * fix: address review feedback - csv check: use regex lookaround instead of substring match to prevent false positives (e.g. 13510 containing '351') - ratelimit: fix operator precedence in block finder by adding parens around the || inside the !b.lang guard - README: note that React/FastAPI checks are structural only, add prerequisites section (Python 3, pandas, Node.js 18+) - test: add regression test for csv substring false positive
42 lines
1.7 KiB
YAML
42 lines
1.7 KiB
YAML
# Ponytail benchmark: code size + cost across three arms, same model, same tasks.
|
|
#
|
|
# Run: npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml
|
|
# View: npx promptfoo@latest view
|
|
# Share: npx promptfoo@latest share (publishes a hosted report URL)
|
|
#
|
|
# Needs ANTHROPIC_API_KEY in the environment or a .env file (see benchmarks/README.md).
|
|
# Caveman arm uses JuliusBrussee/caveman SKILL.md (MIT), vendored at arms/caveman-SKILL.md.
|
|
description: "Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and tokens/cost (API telemetry)."
|
|
|
|
providers:
|
|
- id: anthropic:messages:claude-haiku-4-5-20251001
|
|
config: { max_tokens: 8192, temperature: 1 }
|
|
- id: anthropic:messages:claude-sonnet-4-6
|
|
config: { max_tokens: 8192, temperature: 1 }
|
|
- id: anthropic:messages:claude-opus-4-8
|
|
config: { max_tokens: 8192, temperature: 1 }
|
|
|
|
prompts:
|
|
- id: file://arms/baseline.js
|
|
label: baseline (no skill)
|
|
- id: file://arms/caveman.js
|
|
label: caveman
|
|
- id: file://arms/ponytail.js
|
|
label: ponytail
|
|
|
|
defaultTest:
|
|
assert:
|
|
- type: javascript
|
|
value: file://loc.js
|
|
metric: code_loc
|
|
- type: javascript
|
|
value: file://correctness.js
|
|
metric: correct
|
|
|
|
tests:
|
|
- vars: { task: "Write me a Python function that validates email addresses." }
|
|
- vars: { task: "Add debounce to a search input in vanilla JavaScript. It currently fires an API call on every keystroke." }
|
|
- vars: { task: "Write Python code that reads sales.csv and sums the 'amount' column." }
|
|
- vars: { task: "Build me a countdown timer component in React that counts down from a given number of seconds." }
|
|
- vars: { task: "Add rate limiting to my FastAPI endpoint so users can't spam it." }
|