feat: reproducible promptfoo benchmark + 3-model results
Commit a promptfoo harness (config + arm prompts + LOC metric + vendored caveman SKILL) so anyone can re-run the comparison: no-skill vs caveman vs ponytail, across Haiku / Sonnet / Opus, 10 runs per cell, median reported. Replace the old unreproducible 6-task chart with assets/benchmark-3model.svg from this run, and reframe the README to the reproducible numbers: ponytail writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a no-skill agent on every model. benchmarks/README.md carries the median tables and the reproduce command. Drops nothing that is not measured. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Emeriko
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
93f3ac1d76
commit
321a59c82f
@@ -0,0 +1,2 @@
|
|||||||
|
# Copy to .env (gitignored) and fill in. promptfoo reads this automatically.
|
||||||
|
ANTHROPIC_API_KEY=sk-ant-...
|
||||||
+11
@@ -0,0 +1,11 @@
|
|||||||
|
# Secrets — never commit API keys
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
|
||||||
|
# Dependencies
|
||||||
|
node_modules/
|
||||||
|
|
||||||
|
# promptfoo eval artifacts
|
||||||
|
.promptfoo/
|
||||||
|
benchmarks/output*
|
||||||
@@ -16,8 +16,8 @@
|
|||||||
</p>
|
</p>
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<strong>47% fewer tokens · 3× faster · one-seventh the code</strong><br>
|
<strong>80-94% less code · 3-6× faster · 47-77% cheaper</strong><br>
|
||||||
<sub>Same six tasks, same model, same adversarial security and concurrency probes. <a href="benchmarks/">See the benchmark.</a></sub>
|
<sub>Median of 10 runs across Haiku, Sonnet, and Opus. <a href="benchmarks/">Reproduce it yourself.</a></sub>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -41,13 +41,13 @@ More survivors in [examples/](examples/).
|
|||||||
|
|
||||||
## Numbers
|
## Numbers
|
||||||
|
|
||||||
Six tasks: streaming log parser, atomic file sync, notification dispatcher, validation engine, auth module, concurrent money ledger. One spec each, one fresh agent per arm, same model. Three arms: no skill, the [caveman](https://github.com/JuliusBrussee/caveman) skill, and ponytail. Every arm passes the same adversarial security and concurrency probes. Then the agreement ends:
|
Five everyday tasks (email validator, debounce, CSV sum, countdown timer, rate limiter), three models, three arms: no skill, the [caveman](https://github.com/JuliusBrussee/caveman) skill, and ponytail. Ten runs per cell, median reported.
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="assets/benchmark-loc.svg" width="860" alt="Lines of code per task: ponytail 490 total vs caveman 1,440 vs no-skill control 3,629, all passing the same adversarial probes">
|
<img src="assets/benchmark-3model.svg" width="860" alt="Median lines of code per arm across Haiku, Sonnet and Opus; ponytail writes 80-94% less code than the no-skill baseline">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
**47% fewer tokens than the no-skill agent. 3× faster. A seventh of the code.** The 3,139 lines nobody wrote have never caused an incident. When a surprise feature request hit two of the tasks, ponytail extended in 96 changed lines; caveman needed 413, the no-skill agent 1,115. Every shortcut ponytail took is marked in the code with a `ponytail:` comment naming its upgrade path. Data: [benchmarks/](benchmarks/).
|
**80-94% less code, 47-77% less cost, and 3-6× faster than a no-skill agent, on every model.** Every shortcut ponytail takes is marked in the code with a `ponytail:` comment naming its upgrade path. Reproduce it yourself: `npx promptfoo eval -c benchmarks/promptfooconfig.yaml`. Method and raw numbers: [benchmarks/](benchmarks/). Production-grade tasks, where an unconstrained agent bloats far more, are written up in [benchmarks/results/](benchmarks/results/).
|
||||||
|
|
||||||
## How it works
|
## How it works
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,21 @@
|
|||||||
|
<svg viewBox="0 0 860 336" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
||||||
|
<title>Median lines of code per arm across three models</title>
|
||||||
|
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Median lines of code. 10 runs per cell. Lower is leaner.</text>
|
||||||
|
<text x="20" y="45" font-size="12" fill="#8b949e" opacity="0.85">Ponytail writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a no-skill agent.</text>
|
||||||
|
<rect x="20" y="58" width="12" height="12" rx="2" fill="#8b949e"/><text x="38" y="69" font-size="13" fill="#8b949e">baseline (no skill)</text>
|
||||||
|
<rect x="190" y="58" width="12" height="12" rx="2" fill="#d9822b"/><text x="208" y="69" font-size="13" fill="#8b949e">caveman</text>
|
||||||
|
<rect x="300" y="58" width="12" height="12" rx="2" fill="#2da44e"/><text x="318" y="69" font-size="13" fill="#8b949e">ponytail</text>
|
||||||
|
<text x="112" y="119" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Haiku</text>
|
||||||
|
<rect x="120" y="92" width="508" height="14" rx="2" fill="#8b949e"/><text x="634" y="103" font-size="11" fill="#8b949e">518</text>
|
||||||
|
<rect x="120" y="110" width="114" height="14" rx="2" fill="#d9822b"/><text x="240" y="121" font-size="11" fill="#d9822b">116</text>
|
||||||
|
<rect x="120" y="128" width="38" height="14" rx="2" fill="#2da44e"/><text x="164" y="139" font-size="11" fill="#2da44e" font-weight="600">39</text>
|
||||||
|
<text x="112" y="193" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Sonnet</text>
|
||||||
|
<rect x="120" y="166" width="680" height="14" rx="2" fill="#8b949e"/><text x="806" y="177" font-size="11" fill="#8b949e">693</text>
|
||||||
|
<rect x="120" y="184" width="118" height="14" rx="2" fill="#d9822b"/><text x="244" y="195" font-size="11" fill="#d9822b">120</text>
|
||||||
|
<rect x="120" y="202" width="43" height="14" rx="2" fill="#2da44e"/><text x="169" y="213" font-size="11" fill="#2da44e" font-weight="600">44</text>
|
||||||
|
<text x="112" y="267" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Opus</text>
|
||||||
|
<rect x="120" y="240" width="251" height="14" rx="2" fill="#8b949e"/><text x="377" y="251" font-size="11" fill="#8b949e">256</text>
|
||||||
|
<rect x="120" y="258" width="66" height="14" rx="2" fill="#d9822b"/><text x="192" y="269" font-size="11" fill="#d9822b">67</text>
|
||||||
|
<rect x="120" y="276" width="50" height="14" rx="2" fill="#2da44e"/><text x="176" y="287" font-size="11" fill="#2da44e" font-weight="600">51</text>
|
||||||
|
<text x="120" y="324" font-size="11" fill="#8b949e" opacity="0.8">Median of 10 runs/cell, default temperature. 5 tasks (email, debounce, CSV sum, countdown, rate-limit), same model per group. Reproduce: npx promptfoo eval -c benchmarks/promptfooconfig.yaml</text>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 2.7 KiB |
@@ -1,69 +0,0 @@
|
|||||||
<svg viewBox="0 0 860 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
|
||||||
<title>Non-blank lines of code per task: control vs caveman vs ponytail</title>
|
|
||||||
|
|
||||||
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Six tasks. Adversarial probes: everyone passes. Lines of code: not everyone.</text>
|
|
||||||
|
|
||||||
<!-- legend -->
|
|
||||||
<rect x="20" y="42" width="12" height="12" rx="2" fill="#8b949e"/>
|
|
||||||
<text x="38" y="53" font-size="13" fill="#8b949e">Control (no skill) · 3,629 total</text>
|
|
||||||
<rect x="250" y="42" width="12" height="12" rx="2" fill="#d9822b"/>
|
|
||||||
<text x="268" y="53" font-size="13" fill="#8b949e">Caveman · 1,440</text>
|
|
||||||
<rect x="420" y="42" width="12" height="12" rx="2" fill="#2da44e"/>
|
|
||||||
<text x="438" y="53" font-size="13" fill="#8b949e">Ponytail · 490</text>
|
|
||||||
|
|
||||||
<!-- Task A -->
|
|
||||||
<text x="112" y="117" font-size="12" fill="#8b949e" text-anchor="end">log-analysis CLI</text>
|
|
||||||
<rect x="120" y="90" width="662" height="13" rx="2" fill="#8b949e"/>
|
|
||||||
<text x="788" y="101" font-size="11" fill="#8b949e">946</text>
|
|
||||||
<rect x="120" y="106" width="198" height="13" rx="2" fill="#d9822b"/>
|
|
||||||
<text x="324" y="117" font-size="11" fill="#d9822b">283</text>
|
|
||||||
<rect x="120" y="122" width="102" height="13" rx="2" fill="#2da44e"/>
|
|
||||||
<text x="228" y="133" font-size="11" fill="#2da44e" font-weight="600">145</text>
|
|
||||||
|
|
||||||
<!-- Task B -->
|
|
||||||
<text x="112" y="177" font-size="12" fill="#8b949e" text-anchor="end">file sync</text>
|
|
||||||
<rect x="120" y="150" width="459" height="13" rx="2" fill="#8b949e"/>
|
|
||||||
<text x="585" y="161" font-size="11" fill="#8b949e">656</text>
|
|
||||||
<rect x="120" y="166" width="160" height="13" rx="2" fill="#d9822b"/>
|
|
||||||
<text x="286" y="177" font-size="11" fill="#d9822b">228</text>
|
|
||||||
<rect x="120" y="182" width="69" height="13" rx="2" fill="#2da44e"/>
|
|
||||||
<text x="195" y="193" font-size="11" fill="#2da44e" font-weight="600">99</text>
|
|
||||||
|
|
||||||
<!-- Task C -->
|
|
||||||
<text x="112" y="237" font-size="12" fill="#8b949e" text-anchor="end">notification dispatcher</text>
|
|
||||||
<rect x="120" y="210" width="566" height="13" rx="2" fill="#8b949e"/>
|
|
||||||
<text x="692" y="221" font-size="11" fill="#8b949e">808</text>
|
|
||||||
<rect x="120" y="226" width="277" height="13" rx="2" fill="#d9822b"/>
|
|
||||||
<text x="403" y="237" font-size="11" fill="#d9822b">396</text>
|
|
||||||
<rect x="120" y="242" width="51" height="13" rx="2" fill="#2da44e"/>
|
|
||||||
<text x="177" y="253" font-size="11" fill="#2da44e" font-weight="600">73</text>
|
|
||||||
|
|
||||||
<!-- Task D -->
|
|
||||||
<text x="112" y="297" font-size="12" fill="#8b949e" text-anchor="end">validation engine</text>
|
|
||||||
<rect x="120" y="270" width="474" height="13" rx="2" fill="#8b949e"/>
|
|
||||||
<text x="600" y="281" font-size="11" fill="#8b949e">677</text>
|
|
||||||
<rect x="120" y="286" width="153" height="13" rx="2" fill="#d9822b"/>
|
|
||||||
<text x="279" y="297" font-size="11" fill="#d9822b">218</text>
|
|
||||||
<rect x="120" y="302" width="49" height="13" rx="2" fill="#2da44e"/>
|
|
||||||
<text x="175" y="313" font-size="11" fill="#2da44e" font-weight="600">70</text>
|
|
||||||
|
|
||||||
<!-- Task E -->
|
|
||||||
<text x="112" y="357" font-size="12" fill="#8b949e" text-anchor="end">auth module</text>
|
|
||||||
<rect x="120" y="330" width="182" height="13" rx="2" fill="#8b949e"/>
|
|
||||||
<text x="308" y="341" font-size="11" fill="#8b949e">260</text>
|
|
||||||
<rect x="120" y="346" width="104" height="13" rx="2" fill="#d9822b"/>
|
|
||||||
<text x="230" y="357" font-size="11" fill="#d9822b">148</text>
|
|
||||||
<rect x="120" y="362" width="34" height="13" rx="2" fill="#2da44e"/>
|
|
||||||
<text x="160" y="373" font-size="11" fill="#2da44e" font-weight="600">49</text>
|
|
||||||
|
|
||||||
<!-- Task F -->
|
|
||||||
<text x="112" y="417" font-size="12" fill="#8b949e" text-anchor="end">money ledger</text>
|
|
||||||
<rect x="120" y="390" width="197" height="13" rx="2" fill="#8b949e"/>
|
|
||||||
<text x="323" y="401" font-size="11" fill="#8b949e">282</text>
|
|
||||||
<rect x="120" y="406" width="117" height="13" rx="2" fill="#d9822b"/>
|
|
||||||
<text x="243" y="417" font-size="11" fill="#d9822b">167</text>
|
|
||||||
<rect x="120" y="422" width="38" height="13" rx="2" fill="#2da44e"/>
|
|
||||||
<text x="164" y="433" font-size="11" fill="#2da44e" font-weight="600">54</text>
|
|
||||||
|
|
||||||
<text x="120" y="458" font-size="11" fill="#8b949e" opacity="0.8">Non-blank LOC, AST-counted. Same model, same specs, one fresh agent per arm. Every arm passes the same security (8/8) and concurrency (6/6) probes. 2026-06-12.</text>
|
|
||||||
</svg>
|
|
||||||
|
Before Width: | Height: | Size: 4.2 KiB |
@@ -0,0 +1,47 @@
|
|||||||
|
# Benchmark
|
||||||
|
|
||||||
|
Three arms (no skill, [caveman](https://github.com/JuliusBrussee/caveman), ponytail), three models, five everyday tasks, **10 runs per cell, median reported**. Code LOC is counted from fenced code blocks; tokens, cost, and latency come straight from the API.
|
||||||
|
|
||||||
|
## Reproduce
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp ../.env.example ../.env # add your ANTHROPIC_API_KEY
|
||||||
|
npx promptfoo@latest eval -c promptfooconfig.yaml --repeat 10
|
||||||
|
npx promptfoo@latest view
|
||||||
|
```
|
||||||
|
|
||||||
|
Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limit (see `promptfooconfig.yaml`). Single-shot completions, default temperature.
|
||||||
|
|
||||||
|
## Median results (10 runs, 2026-06-13)
|
||||||
|
|
||||||
|
**Code (lines)**
|
||||||
|
|
||||||
|
| arm | Haiku | Sonnet | Opus |
|
||||||
|
|---|--:|--:|--:|
|
||||||
|
| baseline (no skill) | 518 | 693 | 256 |
|
||||||
|
| caveman | 116 | 120 | 67 |
|
||||||
|
| **ponytail** | **39** | **44** | **51** |
|
||||||
|
|
||||||
|
**Cost (USD, 5 tasks)**
|
||||||
|
|
||||||
|
| arm | Haiku | Sonnet | Opus |
|
||||||
|
|---|--:|--:|--:|
|
||||||
|
| baseline (no skill) | 0.032 | 0.141 | 0.135 |
|
||||||
|
| caveman | 0.014 | 0.045 | 0.075 |
|
||||||
|
| **ponytail** | **0.010** | **0.032** | **0.071** |
|
||||||
|
|
||||||
|
**Latency (seconds, 5 tasks)**
|
||||||
|
|
||||||
|
| arm | Haiku | Sonnet | Opus |
|
||||||
|
|---|--:|--:|--:|
|
||||||
|
| baseline (no skill) | 37.7 | 124.1 | 58.7 |
|
||||||
|
| caveman | 14.9 | 34.7 | 23.1 |
|
||||||
|
| **ponytail** | **9.9** | **20.1** | **18.0** |
|
||||||
|
|
||||||
|
Versus baseline, ponytail writes **80-94% less code**, costs **47-77% less**, and runs **3-6x faster** — on every model.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- Caveman is a prose-compression skill (it leaves code "normal"), so it lands between baseline and ponytail on code size and wins mainly on prose tokens.
|
||||||
|
- Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so the cost gap widens further in ponytail's favor.
|
||||||
|
- These are everyday tasks. For production-grade specs, where an unconstrained agent bloats much harder, see the writeups in `results/`.
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
// Baseline arm: no skill, just the task.
|
||||||
|
module.exports = ({ vars }) => [{ role: 'user', content: vars.task }];
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
---
|
||||||
|
name: caveman
|
||||||
|
description: >
|
||||||
|
Ultra-compressed communication mode. Cuts token usage ~75% by speaking like caveman
|
||||||
|
while keeping full technical accuracy. Supports intensity levels: lite, full (default), ultra,
|
||||||
|
wenyan-lite, wenyan-full, wenyan-ultra.
|
||||||
|
Use when user says "caveman mode", "talk like caveman", "use caveman", "less tokens",
|
||||||
|
"be brief", or invokes /caveman. Also auto-triggers when token efficiency is requested.
|
||||||
|
---
|
||||||
|
|
||||||
|
Respond terse like smart caveman. All technical substance stay. Only fluff die.
|
||||||
|
|
||||||
|
## Persistence
|
||||||
|
|
||||||
|
ACTIVE EVERY RESPONSE. No revert after many turns. No filler drift. Still active if unsure. Off only: "stop caveman" / "normal mode".
|
||||||
|
|
||||||
|
Default: **full**. Switch: `/caveman lite|full|ultra`.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact.
|
||||||
|
|
||||||
|
Pattern: `[thing] [action] [reason]. [next step].`
|
||||||
|
|
||||||
|
Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..."
|
||||||
|
Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:"
|
||||||
|
|
||||||
|
## Intensity
|
||||||
|
|
||||||
|
| Level | What change |
|
||||||
|
|-------|------------|
|
||||||
|
| **lite** | No filler/hedging. Keep articles + full sentences. Professional but tight |
|
||||||
|
| **full** | Drop articles, fragments OK, short synonyms. Classic caveman |
|
||||||
|
| **ultra** | Abbreviate (DB/auth/config/req/res/fn/impl), strip conjunctions, arrows for causality (X → Y), one word when one word enough |
|
||||||
|
| **wenyan-lite** | Semi-classical. Drop filler/hedging but keep grammar structure, classical register |
|
||||||
|
| **wenyan-full** | Maximum classical terseness. Fully 文言文. 80-90% character reduction. Classical sentence patterns, verbs precede objects, subjects often omitted, classical particles (之/乃/為/其) |
|
||||||
|
| **wenyan-ultra** | Extreme abbreviation while keeping classical Chinese feel. Maximum compression, ultra terse |
|
||||||
|
|
||||||
|
Example — "Why React component re-render?"
|
||||||
|
- lite: "Your component re-renders because you create a new object reference each render. Wrap it in `useMemo`."
|
||||||
|
- full: "New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`."
|
||||||
|
- ultra: "Inline obj prop → new ref → re-render. `useMemo`."
|
||||||
|
- wenyan-lite: "組件頻重繪，以每繪新生對象參照故。以 useMemo 包之。"
|
||||||
|
- wenyan-full: "物出新參照，致重繪。useMemo .Wrap之。"
|
||||||
|
- wenyan-ultra: "新參照→重繪。useMemo Wrap。"
|
||||||
|
|
||||||
|
Example — "Explain database connection pooling."
|
||||||
|
- lite: "Connection pooling reuses open connections instead of creating new ones per request. Avoids repeated handshake overhead."
|
||||||
|
- full: "Pool reuse open DB connections. No new connection per request. Skip handshake overhead."
|
||||||
|
- ultra: "Pool = reuse DB conn. Skip handshake → fast under load."
|
||||||
|
- wenyan-full: "池reuse open connection。不每req新開。skip handshake overhead。"
|
||||||
|
- wenyan-ultra: "池reuse conn。skip handshake → fast。"
|
||||||
|
|
||||||
|
## Auto-Clarity
|
||||||
|
|
||||||
|
Drop caveman for: security warnings, irreversible action confirmations, multi-step sequences where fragment order risks misread, user asks to clarify or repeats question. Resume caveman after clear part done.
|
||||||
|
|
||||||
|
Example — destructive op:
|
||||||
|
> **Warning:** This will permanently delete all rows in the `users` table and cannot be undone.
|
||||||
|
> ```sql
|
||||||
|
> DROP TABLE users;
|
||||||
|
> ```
|
||||||
|
> Caveman resume. Verify backup exist first.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end.
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
// Caveman arm: caveman SKILL.md (full) as the system prompt.
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const system = fs.readFileSync(path.join(__dirname, 'caveman-SKILL.md'), 'utf8');
|
||||||
|
module.exports = ({ vars }) => [
|
||||||
|
{ role: 'system', content: system },
|
||||||
|
{ role: 'user', content: vars.task },
|
||||||
|
];
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
// Ponytail arm: the repo's own SKILL.md (full) as the system prompt. Single source of truth.
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
const system = fs.readFileSync(path.join(__dirname, '..', '..', 'skills', 'ponytail', 'SKILL.md'), 'utf8');
|
||||||
|
module.exports = ({ vars }) => [
|
||||||
|
{ role: 'system', content: system },
|
||||||
|
{ role: 'user', content: vars.task },
|
||||||
|
];
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
// Deterministic code-size metric: non-blank, non-comment lines inside fenced code blocks.
|
||||||
|
// Recorded as the `code_loc` metric per arm (always passes; it is a measurement, not a gate).
|
||||||
|
module.exports = (output) => {
|
||||||
|
const text = String(output || '');
|
||||||
|
const blocks = [...text.matchAll(/```[a-zA-Z0-9_+-]*\n([\s\S]*?)```/g)].map((m) => m[1]);
|
||||||
|
const code = blocks.join('\n');
|
||||||
|
const loc = code
|
||||||
|
.split('\n')
|
||||||
|
.map((l) => l.trim())
|
||||||
|
.filter((l) => l && !l.startsWith('//') && !l.startsWith('#') && l !== '*/' && !l.startsWith('/*') && !l.startsWith('*')).length;
|
||||||
|
return { pass: true, score: loc, reason: loc + ' code LOC' };
|
||||||
|
};
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
# Ponytail benchmark: code size + cost across three arms, same model, same tasks.
|
||||||
|
#
|
||||||
|
# Run: npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml
|
||||||
|
# View: npx promptfoo@latest view
|
||||||
|
# Share: npx promptfoo@latest share (publishes a hosted report URL)
|
||||||
|
#
|
||||||
|
# Needs ANTHROPIC_API_KEY in the environment or a .env file (see benchmarks/README.md).
|
||||||
|
# Caveman arm uses JuliusBrussee/caveman SKILL.md (MIT), vendored at arms/caveman-SKILL.md.
|
||||||
|
description: "Ponytail vs caveman vs no-skill: same model, same tasks. Measures code LOC (deterministic) and tokens/cost (API telemetry)."
|
||||||
|
|
||||||
|
providers:
|
||||||
|
- id: anthropic:messages:claude-haiku-4-5-20251001
|
||||||
|
config: { max_tokens: 8192, temperature: 1 }
|
||||||
|
- id: anthropic:messages:claude-sonnet-4-6
|
||||||
|
config: { max_tokens: 8192, temperature: 1 }
|
||||||
|
- id: anthropic:messages:claude-opus-4-8
|
||||||
|
config: { max_tokens: 8192, temperature: 1 }
|
||||||
|
|
||||||
|
prompts:
|
||||||
|
- id: file://arms/baseline.js
|
||||||
|
label: baseline (no skill)
|
||||||
|
- id: file://arms/caveman.js
|
||||||
|
label: caveman
|
||||||
|
- id: file://arms/ponytail.js
|
||||||
|
label: ponytail
|
||||||
|
|
||||||
|
defaultTest:
|
||||||
|
assert:
|
||||||
|
- type: javascript
|
||||||
|
value: file://loc.js
|
||||||
|
metric: code_loc
|
||||||
|
|
||||||
|
tests:
|
||||||
|
- vars: { task: "Write me a Python function that validates email addresses." }
|
||||||
|
- vars: { task: "Add debounce to a search input in vanilla JavaScript. It currently fires an API call on every keystroke." }
|
||||||
|
- vars: { task: "Write Python code that reads sales.csv and sums the 'amount' column." }
|
||||||
|
- vars: { task: "Build me a countdown timer component in React that counts down from a given number of seconds." }
|
||||||
|
- vars: { task: "Add rate limiting to my FastAPI endpoint so users can't spam it." }
|
||||||
Reference in New Issue
Block a user