at384
6d990f8c54
* feat(benchmarks): add correctness assertion - proves less code is not broken code The existing benchmark measures lines-of-code (loc.js) but never checks whether the generated code actually works. This adds a functional correctness gate (correctness.js) that extracts code from fenced blocks and runs per-task checks: - email validator: spawns Python, asserts accept/reject on 5 inputs - debounce: spawns Node, asserts delayed execution + reset on re-call - csv sum: spawns Python with a test CSV, asserts correct total (351) - countdown (React): structural check (useState + useEffect + decrement) - rate limiter (FastAPI): structural check (limit logic + framework usage) 12 unit tests (node:test) cover good/bad outputs for every task plus the unknown-task edge case. Existing tests and rule-copy checks unaffected. * fix: address review feedback - csv check: use regex lookaround instead of substring match to prevent false positives (e.g. 13510 containing '351') - ratelimit: fix operator precedence in block finder by adding parens around the || inside the !b.lang guard - README: note that React/FastAPI checks are structural only, add prerequisites section (Python 3, pandas, Node.js 18+) - test: add regression test for csv substring false positive
63 lines
2.6 KiB
Markdown
63 lines
2.6 KiB
Markdown
# Benchmark
|
|
|
|
Three arms (no skill, [caveman](https://github.com/JuliusBrussee/caveman), ponytail), three models, five everyday tasks, **10 runs per cell, median reported**. Code LOC is counted from fenced code blocks; tokens, cost, and latency come straight from the API.
|
|
|
|
## Reproduce
|
|
|
|
```bash
|
|
cp ../.env.example ../.env # add your ANTHROPIC_API_KEY
|
|
npx promptfoo@latest eval -c promptfooconfig.yaml --repeat 10
|
|
npx promptfoo@latest view
|
|
```
|
|
|
|
Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limit (see `promptfooconfig.yaml`). Single-shot completions, default temperature.
|
|
|
|
## Median results (10 runs, 2026-06-13)
|
|
|
|
**Code (lines)**
|
|
|
|
| arm | Haiku | Sonnet | Opus |
|
|
|---|--:|--:|--:|
|
|
| baseline (no skill) | 518 | 693 | 256 |
|
|
| caveman | 116 | 120 | 67 |
|
|
| **ponytail** | **39** | **44** | **51** |
|
|
|
|
**Cost (USD, 5 tasks)**
|
|
|
|
| arm | Haiku | Sonnet | Opus |
|
|
|---|--:|--:|--:|
|
|
| baseline (no skill) | 0.032 | 0.141 | 0.135 |
|
|
| caveman | 0.014 | 0.045 | 0.075 |
|
|
| **ponytail** | **0.010** | **0.032** | **0.071** |
|
|
|
|
**Latency (seconds, 5 tasks)**
|
|
|
|
| arm | Haiku | Sonnet | Opus |
|
|
|---|--:|--:|--:|
|
|
| baseline (no skill) | 37.7 | 124.1 | 58.7 |
|
|
| caveman | 14.9 | 34.7 | 23.1 |
|
|
| **ponytail** | **9.9** | **20.1** | **18.0** |
|
|
|
|
Versus baseline, ponytail writes **80-94% less code**, costs **47-77% less**, and runs **3-6x faster**, on every model.
|
|
|
|
## Metrics
|
|
|
|
| File | Metric | Behavior |
|
|
|------|--------|----------|
|
|
| `loc.js` | `loc` | Measurement - always passes, records line count |
|
|
| `correctness.js` | `correct` | Gate - fails if generated code doesn't work |
|
|
|
|
`correctness.js` extracts fenced code blocks and runs per-task checks (spawns Python/Node for email, debounce, CSV; structural regex for React and FastAPI). A broken one-liner that scores great on LOC will fail on correctness.
|
|
|
|
> **Note:** The React countdown and FastAPI rate-limit checks are keyword/structural only (no runtime execution), so they verify plausible structure rather than full correctness. The email, debounce, and CSV checks execute the code.
|
|
|
|
### Prerequisites
|
|
|
|
Running the benchmark requires **Python 3**, **pandas**, and **Node.js** (18+).
|
|
|
|
## Notes
|
|
|
|
- Caveman is a prose-compression skill (it leaves code "normal"), so it lands between baseline and ponytail on code size and wins mainly on prose tokens.
|
|
- Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so the cost gap widens further in ponytail's favor.
|
|
- These are everyday tasks. For production-grade specs, where an unconstrained agent bloats much harder, see the writeups in `results/`.
|