feat: reproducible promptfoo benchmark + 3-model results

Commit a promptfoo harness (config + arm prompts + LOC metric + vendored
caveman SKILL) so anyone can re-run the comparison: no-skill vs caveman vs
ponytail, across Haiku / Sonnet / Opus, 10 runs per cell, median reported.

Replace the old unreproducible 6-task chart with assets/benchmark-3model.svg
from this run, and reframe the README to the reproducible numbers: ponytail
writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a
no-skill agent on every model. benchmarks/README.md carries the median tables
and the reproduce command. Drops nothing that is not measured.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Emeriko
2026-06-13 05:08:55 +02:00
co-authored by Claude Opus 4.8
parent 93f3ac1d76
commit 321a59c82f
12 changed files with 221 additions and 74 deletions
+8
View File
@@ -0,0 +1,8 @@
// Ponytail arm: the repo's own SKILL.md (full) as the system prompt. Single source of truth.
const fs = require('fs');
const path = require('path');
const system = fs.readFileSync(path.join(__dirname, '..', '..', 'skills', 'ponytail', 'SKILL.md'), 'utf8');
module.exports = ({ vars }) => [
{ role: 'system', content: system },
{ role: 'user', content: vars.task },
];