feat: reproducible promptfoo benchmark + 3-model results
Commit a promptfoo harness (config + arm prompts + LOC metric + vendored caveman SKILL) so anyone can re-run the comparison: no-skill vs caveman vs ponytail, across Haiku / Sonnet / Opus, 10 runs per cell, median reported. Replace the old unreproducible 6-task chart with assets/benchmark-3model.svg from this run, and reframe the README to the reproducible numbers: ponytail writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a no-skill agent on every model. benchmarks/README.md carries the median tables and the reproduce command. Drops nothing that is not measured. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Emeriko
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
93f3ac1d76
commit
321a59c82f
@@ -0,0 +1,47 @@
|
||||
# Benchmark
|
||||
|
||||
Three arms (no skill, [caveman](https://github.com/JuliusBrussee/caveman), ponytail), three models, five everyday tasks, **10 runs per cell, median reported**. Code LOC is counted from fenced code blocks; tokens, cost, and latency come straight from the API.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
cp ../.env.example ../.env # add your ANTHROPIC_API_KEY
|
||||
npx promptfoo@latest eval -c promptfooconfig.yaml --repeat 10
|
||||
npx promptfoo@latest view
|
||||
```
|
||||
|
||||
Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limit (see `promptfooconfig.yaml`). Single-shot completions, default temperature.
|
||||
|
||||
## Median results (10 runs, 2026-06-13)
|
||||
|
||||
**Code (lines)**
|
||||
|
||||
| arm | Haiku | Sonnet | Opus |
|
||||
|---|--:|--:|--:|
|
||||
| baseline (no skill) | 518 | 693 | 256 |
|
||||
| caveman | 116 | 120 | 67 |
|
||||
| **ponytail** | **39** | **44** | **51** |
|
||||
|
||||
**Cost (USD, 5 tasks)**
|
||||
|
||||
| arm | Haiku | Sonnet | Opus |
|
||||
|---|--:|--:|--:|
|
||||
| baseline (no skill) | 0.032 | 0.141 | 0.135 |
|
||||
| caveman | 0.014 | 0.045 | 0.075 |
|
||||
| **ponytail** | **0.010** | **0.032** | **0.071** |
|
||||
|
||||
**Latency (seconds, 5 tasks)**
|
||||
|
||||
| arm | Haiku | Sonnet | Opus |
|
||||
|---|--:|--:|--:|
|
||||
| baseline (no skill) | 37.7 | 124.1 | 58.7 |
|
||||
| caveman | 14.9 | 34.7 | 23.1 |
|
||||
| **ponytail** | **9.9** | **20.1** | **18.0** |
|
||||
|
||||
Versus baseline, ponytail writes **80-94% less code**, costs **47-77% less**, and runs **3-6x faster** — on every model.
|
||||
|
||||
## Notes
|
||||
|
||||
- Caveman is a prose-compression skill (it leaves code "normal"), so it lands between baseline and ponytail on code size and wins mainly on prose tokens.
|
||||
- Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so the cost gap widens further in ponytail's favor.
|
||||
- These are everyday tasks. For production-grade specs, where an unconstrained agent bloats much harder, see the writeups in `results/`.
|
||||
@@ -0,0 +1,2 @@
|
||||
// Baseline arm: no skill, just the task.
|
||||
module.exports = ({ vars }) => [{ role: 'user', content: vars.task }];
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: caveman
|
||||
description: >
|
||||
Ultra-compressed communication mode. Cuts token usage ~75% by speaking like caveman
|
||||
while keeping full technical accuracy. Supports intensity levels: lite, full (default), ultra,
|
||||
wenyan-lite, wenyan-full, wenyan-ultra.
|
||||
Use when user says "caveman mode", "talk like caveman", "use caveman", "less tokens",
|
||||
"be brief", or invokes /caveman. Also auto-triggers when token efficiency is requested.
|
||||
---
|
||||
|
||||
Respond terse like smart caveman. All technical substance stay. Only fluff die.
|
||||
|
||||
## Persistence
|
||||
|
||||
ACTIVE EVERY RESPONSE. No revert after many turns. No filler drift. Still active if unsure. Off only: "stop caveman" / "normal mode".
|
||||
|
||||
Default: **full**. Switch: `/caveman lite|full|ultra`.
|
||||
|
||||
## Rules
|
||||
|
||||
Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact.
|
||||
|
||||
Pattern: `[thing] [action] [reason]. [next step].`
|
||||
|
||||
Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
|
||||
Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
|
||||
|
||||
## Intensity
|
||||
|
||||
| Level | What change |
|
||||
|-------|------------|
|
||||
| **lite** | No filler/hedging. Keep articles + full sentences. Professional but tight |
|
||||
| **full** | Drop articles, fragments OK, short synonyms. Classic caveman |
|
||||
| **ultra** | Abbreviate (DB/auth/config/req/res/fn/impl), strip conjunctions, arrows for causality (X → Y), one word when one word enough |
|
||||
| **wenyan-lite** | Semi-classical. Drop filler/hedging but keep grammar structure, classical register |
|
||||
| **wenyan-full** | Maximum classical terseness. Fully 文言文. 80-90% character reduction. Classical sentence patterns, verbs precede objects, subjects often omitted, classical particles (之/乃/為/其) |
|
||||
| **wenyan-ultra** | Extreme abbreviation while keeping classical Chinese feel. Maximum compression, ultra terse |
|
||||
|
||||
Example — "Why React component re-render?"
|
||||
- lite: "Your component re-renders because you create a new object reference each render. Wrap it in `useMemo`."
|
||||
- full: "New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`."
|
||||
- ultra: "Inline obj prop → new ref → re-render. `useMemo`."
|
||||
- wenyan-lite: "組件頻重繪，以每繪新生對象參照故。以 useMemo 包之。"
|
||||
- wenyan-full: "物出新參照，致重繪。useMemo .Wrap之。"
|
||||
- wenyan-ultra: "新參照→重繪。useMemo Wrap。"
|
||||
|
||||
Example — "Explain database connection pooling."
|
||||
- lite: "Connection pooling reuses open connections instead of creating new ones per request. Avoids repeated handshake overhead."
|
||||
- full: "Pool reuse open DB connections. No new connection per request. Skip handshake overhead."
|
||||
- ultra: "Pool = reuse DB conn. Skip handshake → fast under load."
|
||||
- wenyan-full: "池reuse open connection。不每req新開。skip handshake overhead。"
|
||||
- wenyan-ultra: "池reuse conn。skip handshake → fast。"
|
||||
|
||||
## Auto-Clarity
|
||||
|
||||
Drop caveman for: security warnings, irreversible action confirmations, multi-step sequences where fragment order risks misread, user asks to clarify or repeats question. Resume caveman after clear part done.
|
||||
|
||||
Example — destructive op:
|
||||
> **Warning:** This will permanently delete all rows in the `users` table and cannot be undone.
|
||||
> ```sql
|
||||
> DROP TABLE users;
|
||||
> ```
|
||||
> Caveman resume. Verify backup exist first.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.
|
||||
@@ -0,0 +1,8 @@
|
||||
// Caveman arm: caveman SKILL.md (full) as the system prompt.
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const system = fs.readFileSync(path.join(__dirname, 'caveman-SKILL.md'), 'utf8');
|
||||
module.exports = ({ vars }) => [
|
||||
{ role: 'system', content: system },
|
||||
{ role: 'user', content: vars.task },
|
||||
];
|
||||
@@ -0,0 +1,8 @@
|
||||
// Ponytail arm: the repo's own SKILL.md (full) as the system prompt. Single source of truth.
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const system = fs.readFileSync(path.join(__dirname, '..', '..', 'skills', 'ponytail', 'SKILL.md'), 'utf8');
|
||||
module.exports = ({ vars }) => [
|
||||
{ role: 'system', content: system },
|
||||
{ role: 'user', content: vars.task },
|
||||
];
|
||||
@@ -0,0 +1,12 @@
|
||||
// Deterministic code-size metric: non-blank, non-comment lines inside fenced code blocks.
|
||||
// Recorded as the `code_loc` metric per arm (always passes; it is a measurement, not a gate).
|
||||
module.exports = (output) => {
|
||||
const text = String(output || '');
|
||||
const blocks = [...text.matchAll(/```[a-zA-Z0-9_+-]*\n([\s\S]*?)```/g)].map((m) => m[1]);
|
||||
const code = blocks.join('\n');
|
||||
const loc = code
|
||||
.split('\n')
|
||||
.map((l) => l.trim())
|
||||
.filter((l) => l && !l.startsWith('//') && !l.startsWith('#') && l !== '*/' && !l.startsWith('/*') && !l.startsWith('*')).length;
|
||||
return { pass: true, score: loc, reason: loc + ' code LOC' };
|
||||
};
|
||||
@@ -0,0 +1,38 @@
|
||||
# Ponytail benchmark: code size + cost across three arms, same model, same tasks.
|
||||
#
|
||||
# Run: npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml
|
||||
# View: npx promptfoo@latest view
|
||||
# Share: npx promptfoo@latest share (publishes a hosted report URL)
|
||||
#
|
||||
# Needs ANTHROPIC_API_KEY in the environment or a .env file (see benchmarks/README.md).
|
||||
# Caveman arm uses JuliusBrussee/caveman SKILL.md (MIT), vendored at arms/caveman-SKILL.md.
|
||||
description: "Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and tokens/cost (API telemetry)."
|
||||
|
||||
providers:
|
||||
- id: anthropic:messages:claude-haiku-4-5-20251001
|
||||
config: { max_tokens: 8192, temperature: 1 }
|
||||
- id: anthropic:messages:claude-sonnet-4-6
|
||||
config: { max_tokens: 8192, temperature: 1 }
|
||||
- id: anthropic:messages:claude-opus-4-8
|
||||
config: { max_tokens: 8192, temperature: 1 }
|
||||
|
||||
prompts:
|
||||
- id: file://arms/baseline.js
|
||||
label: baseline (no skill)
|
||||
- id: file://arms/caveman.js
|
||||
label: caveman
|
||||
- id: file://arms/ponytail.js
|
||||
label: ponytail
|
||||
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: javascript
|
||||
value: file://loc.js
|
||||
metric: code_loc
|
||||
|
||||
tests:
|
||||
- vars: { task: "Write me a Python function that validates email addresses." }
|
||||
- vars: { task: "Add debounce to a search input in vanilla JavaScript. It currently fires an API call on every keystroke." }
|
||||
- vars: { task: "Write Python code that reads sales.csv and sums the 'amount' column." }
|
||||
- vars: { task: "Build me a countdown timer component in React that counts down from a given number of seconds." }
|
||||
- vars: { task: "Add rate limiting to my FastAPI endpoint so users can't spam it." }
|
||||
Reference in New Issue
Block a user