Files

T

EmerikoandClaude Opus 4.8 0bf152a987 docs: correct cost claim to 42-75% from 30-rep re-verification

Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus):
ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%.
The direction holds, both ends came in a few points lower. Updates the README
headline and body, the benchmark chart subtitle, and the benchmarks/README cost
table, and adds a dated results doc with full method.

Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On
OpenAI reasoning models ponytail costs more, not less, so the claim stays
Claude-scoped. Gemini run pending a fresh-quota day.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-17 03:47:36 +02:00

3.6 KiB

Raw Blame History

Benchmark

Three arms (no skill, caveman, ponytail), three models, five everyday tasks, 10 runs per cell, median reported. Code LOC is counted from fenced code blocks; tokens, cost, and latency come straight from the API.

Reproduce

Claude (Haiku / Sonnet / Opus)

Requires an Anthropic API key and Node.js ≥ 22.22.0 (promptfoo's engine constraint — check with node --version and upgrade if needed):

cp ../.env.example ../.env      # add your ANTHROPIC_API_KEY
npx promptfoo@latest eval -c promptfooconfig.yaml --env-file ../.env --repeat 10
npx promptfoo@latest view

--env-file ../.env is required because promptfoo reads .env from the current directory (benchmarks/), not the repo root where the file lives.

Local models via Ollama

No API key or promptfoo required. Runs against any model served by Ollama:

ollama pull llama3.2          # or any other model
python benchmarks/benchmark-local.py --model llama3.2 --repeat 3

See benchmarks/results/2026-06-15-llama3.2-local.md for what to expect: the skill works well on instruction-following models (Claude-class) but transfers poorly to small local models where the multi-step decision ladder isn't reliably followed.

Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limit (see promptfooconfig.yaml). Single-shot completions, default temperature.

Median results (10 runs, 2026-06-13; cost re-verified at 30 runs, 2026-06-17)

Code (lines)

arm	Haiku	Sonnet	Opus
baseline (no skill)	518	693	256
caveman	116	120	67
ponytail	39	44	51

Cost (USD, 5 tasks; 30 runs, 2026-06-17)

arm	Haiku	Sonnet	Opus
baseline (no skill)	0.030	0.137	0.137
caveman	0.014	0.046	0.072
ponytail	0.011	0.035	0.079

Latency (seconds, 5 tasks)

arm	Haiku	Sonnet	Opus
baseline (no skill)	37.7	124.1	58.7
caveman	14.9	34.7	23.1
ponytail	9.9	20.1	18.0

Versus baseline, ponytail writes 80-94% less code, costs 42-75% less, and runs 3-6x faster, on every Claude model. Cost re-verified at 30 reps, with OpenAI and Gemini arms, in results/2026-06-17-cost-verification.md.

Metrics

File	Metric	Behavior
`loc.js`	`loc`	Measurement - always passes, records line count
`correctness.js`	`correct`	Gate - fails if generated code doesn't work

correctness.js extracts fenced code blocks and runs per-task checks (spawns Python/Node for email, debounce, CSV; structural regex for React and FastAPI). A broken one-liner that scores great on LOC will fail on correctness.

Note: The React countdown and FastAPI rate-limit checks are keyword/structural only (no runtime execution), so they verify plausible structure rather than full correctness. The email, debounce, and CSV checks execute the code.

Prerequisites

Running the benchmark requires Python 3, pandas, and Node.js (18+).

Notes

Caveman is a prose-compression skill (it leaves code "normal"), so it lands between baseline and ponytail on code size and wins mainly on prose tokens.
Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so the cost gap widens further in ponytail's favor.
These are everyday tasks. For production-grade specs, where an unconstrained agent bloats much harder, see the writeups in results/.

3.6 KiB Raw Blame History