EmerikoandClaude Opus 4.8
321a59c82f
Commit a promptfoo harness (config + arm prompts + LOC metric + vendored caveman SKILL) so anyone can re-run the comparison: no-skill vs caveman vs ponytail, across Haiku / Sonnet / Opus, 10 runs per cell, median reported. Replace the old unreproducible 6-task chart with assets/benchmark-3model.svg from this run, and reframe the README to the reproducible numbers: ponytail writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a no-skill agent on every model. benchmarks/README.md carries the median tables and the reproduce command. Drops nothing that is not measured. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
39 lines
1.6 KiB
YAML
39 lines
1.6 KiB
YAML
# Ponytail benchmark: code size + cost across three arms, same model, same tasks.
|
|
#
|
|
# Run: npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml
|
|
# View: npx promptfoo@latest view
|
|
# Share: npx promptfoo@latest share (publishes a hosted report URL)
|
|
#
|
|
# Needs ANTHROPIC_API_KEY in the environment or a .env file (see benchmarks/README.md).
|
|
# Caveman arm uses JuliusBrussee/caveman SKILL.md (MIT), vendored at arms/caveman-SKILL.md.
|
|
description: "Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and tokens/cost (API telemetry)."
|
|
|
|
providers:
|
|
- id: anthropic:messages:claude-haiku-4-5-20251001
|
|
config: { max_tokens: 8192, temperature: 1 }
|
|
- id: anthropic:messages:claude-sonnet-4-6
|
|
config: { max_tokens: 8192, temperature: 1 }
|
|
- id: anthropic:messages:claude-opus-4-8
|
|
config: { max_tokens: 8192, temperature: 1 }
|
|
|
|
prompts:
|
|
- id: file://arms/baseline.js
|
|
label: baseline (no skill)
|
|
- id: file://arms/caveman.js
|
|
label: caveman
|
|
- id: file://arms/ponytail.js
|
|
label: ponytail
|
|
|
|
defaultTest:
|
|
assert:
|
|
- type: javascript
|
|
value: file://loc.js
|
|
metric: code_loc
|
|
|
|
tests:
|
|
- vars: { task: "Write me a Python function that validates email addresses." }
|
|
- vars: { task: "Add debounce to a search input in vanilla JavaScript. It currently fires an API call on every keystroke." }
|
|
- vars: { task: "Write Python code that reads sales.csv and sums the 'amount' column." }
|
|
- vars: { task: "Build me a countdown timer component in React that counts down from a given number of seconds." }
|
|
- vars: { task: "Add rate limiting to my FastAPI endpoint so users can't spam it." }
|