DietrichGebertandClaude Opus 4.8
1b4914159e
* docs: correct cost claim to 42-75% from 30-rep re-verification Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus): ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%. The direction holds, both ends came in a few points lower. Updates the README headline and body, the benchmark chart subtitle, and the benchmarks/README cost table, and adds a dated results doc with full method. Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On OpenAI reasoning models ponytail costs more, not less, so the claim stays Claude-scoped. Gemini run pending a fresh-quota day. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scope the body claim to Claude models "on every model" read as cross-provider, but the 30-rep verification shows the cost win reverses on OpenAI reasoning models. Match the caption and benchmarks/README, which already say Claude. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reframe the pitch as the discipline, not token savings The cost/code/latency numbers vary by model and on some (terse reasoning models like GPT-5.5) ponytail costs more, so leading with them as a universal win was misleading. Adds model-variance to the headline caption and a paragraph making the stated point the mental model: write only what the task needs, safety kept, maintainable code. Savings are a model-dependent side effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: name the ladder's reasoning cost The ladder is a deliberation step: on reasoning models the agent spends thinking tokens working through the rungs before it saves any output, which together with the always-on ruleset can outweigh the shorter code. Makes the GPT-5.5 cost increase legible rather than just stating it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: state the single-shot limitation honestly The benchmark is single-shot (one prompt, one completion); it does not measure a real multi-turn agent session, where the ruleset re-injects and the ladder deliberates every turn. Adds that caveat to the README, and corrects the benchmarks/README note that claimed caching widens the gap "in ponytail's favor" (unverified, and a measured agentic A/B in #121 found the opposite can happen). Per-session cost can land either way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix run count in caption (cost is 30 runs, not 10) Cost was re-verified at 30 reps; code and latency are still the original 10. The headline caption said "10 runs" across the board, which undersold the cost verification. Now states the split, matching benchmarks/README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
22 lines
2.7 KiB
XML
22 lines
2.7 KiB
XML
<svg viewBox="0 0 860 336" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
|
<title>Median lines of code per arm across three models</title>
|
|
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Median lines of code. 10 runs per cell. Lower is leaner.</text>
|
|
<text x="20" y="45" font-size="12" fill="#8b949e" opacity="0.85">Ponytail writes 80-94% less code, costs 42-75% less, and runs 3-6x faster than a no-skill agent.</text>
|
|
<rect x="20" y="58" width="12" height="12" rx="2" fill="#8b949e"/><text x="38" y="69" font-size="13" fill="#8b949e">baseline (no skill)</text>
|
|
<rect x="190" y="58" width="12" height="12" rx="2" fill="#d9822b"/><text x="208" y="69" font-size="13" fill="#8b949e">caveman</text>
|
|
<rect x="300" y="58" width="12" height="12" rx="2" fill="#2da44e"/><text x="318" y="69" font-size="13" fill="#8b949e">ponytail</text>
|
|
<text x="112" y="119" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Haiku</text>
|
|
<rect x="120" y="92" width="508" height="14" rx="2" fill="#8b949e"/><text x="634" y="103" font-size="11" fill="#8b949e">518</text>
|
|
<rect x="120" y="110" width="114" height="14" rx="2" fill="#d9822b"/><text x="240" y="121" font-size="11" fill="#d9822b">116</text>
|
|
<rect x="120" y="128" width="38" height="14" rx="2" fill="#2da44e"/><text x="164" y="139" font-size="11" fill="#2da44e" font-weight="600">39</text>
|
|
<text x="112" y="193" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Sonnet</text>
|
|
<rect x="120" y="166" width="680" height="14" rx="2" fill="#8b949e"/><text x="806" y="177" font-size="11" fill="#8b949e">693</text>
|
|
<rect x="120" y="184" width="118" height="14" rx="2" fill="#d9822b"/><text x="244" y="195" font-size="11" fill="#d9822b">120</text>
|
|
<rect x="120" y="202" width="43" height="14" rx="2" fill="#2da44e"/><text x="169" y="213" font-size="11" fill="#2da44e" font-weight="600">44</text>
|
|
<text x="112" y="267" font-size="13" font-weight="600" fill="#8b949e" text-anchor="end">Opus</text>
|
|
<rect x="120" y="240" width="251" height="14" rx="2" fill="#8b949e"/><text x="377" y="251" font-size="11" fill="#8b949e">256</text>
|
|
<rect x="120" y="258" width="66" height="14" rx="2" fill="#d9822b"/><text x="192" y="269" font-size="11" fill="#d9822b">67</text>
|
|
<rect x="120" y="276" width="50" height="14" rx="2" fill="#2da44e"/><text x="176" y="287" font-size="11" fill="#2da44e" font-weight="600">51</text>
|
|
<text x="120" y="324" font-size="11" fill="#8b949e" opacity="0.8">Median of 10 runs/cell, default temperature. 5 tasks (email, debounce, CSV sum, countdown, rate-limit), same model per group. Reproduce: npx promptfoo eval -c benchmarks/promptfooconfig.yaml</text>
|
|
</svg>
|