docs(readme): add the safety tier to the benchmark chart
Separate strip below the metric clusters showing the 6-task adversarial safe rate (higher is safer): baseline, caveman and ponytail 100%, yagni-oneliner 95% (flagged). Kept apart from the lower-is-better % cluster so the opposite polarity and different task set are not conflated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
f0dad86a52
commit
b742c9c07c
@@ -47,7 +47,7 @@ More survivors in [examples/](examples/).
|
|||||||
The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5.
|
The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5.
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="assets/benchmark-agentic.svg" width="860" alt="Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%.">
|
<img src="assets/benchmark-agentic.svg" width="860" alt="Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%. Safety, separate adversarial tier: baseline, caveman and ponytail 100%, yagni-oneliner 95%.">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
| vs no-skill baseline | LOC | tokens | cost | time | safe |
|
| vs no-skill baseline | LOC | tokens | cost | time | safe |
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<svg viewBox="0 0 860 430" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
<svg viewBox="0 0 860 488" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
||||||
<title>Each arm vs the no-skill baseline across every metric, Claude Code on Haiku 4.5</title>
|
<title>Each arm vs the no-skill baseline across every metric, plus safety, Claude Code on Haiku 4.5</title>
|
||||||
<text x="430" y="24" font-size="15" font-weight="600" fill="#8b949e" text-anchor="middle">Every metric vs the no-skill baseline (Claude Code, Haiku 4.5, 12 tasks)</text>
|
<text x="430" y="24" font-size="15" font-weight="600" fill="#8b949e" text-anchor="middle">Every metric vs the no-skill baseline (Claude Code, Haiku 4.5, 12 tasks)</text>
|
||||||
|
|
||||||
<rect x="212" y="38" width="12" height="12" rx="2" fill="#8b949e"/><text x="229" y="48" font-size="12" fill="#8b949e">baseline</text>
|
<rect x="212" y="38" width="12" height="12" rx="2" fill="#8b949e"/><text x="229" y="48" font-size="12" fill="#8b949e">baseline</text>
|
||||||
@@ -51,5 +51,12 @@
|
|||||||
<text x="720" y="380" font-size="13" fill="#8b949e" text-anchor="middle">time</text>
|
<text x="720" y="380" font-size="13" fill="#8b949e" text-anchor="middle">time</text>
|
||||||
<text x="720" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 69s</text>
|
<text x="720" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 69s</text>
|
||||||
|
|
||||||
<text x="20" y="420" font-size="11" fill="#8b949e" opacity="0.8">Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4. Writeup: benchmarks/results/2026-06-18-agentic.md</text>
|
<text x="20" y="418" font-size="11" fill="#8b949e" opacity="0.8">Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4.</text>
|
||||||
|
|
||||||
|
<line x1="20" y1="438" x2="815" y2="438" stroke="#8b949e" stroke-opacity="0.25"/>
|
||||||
|
<text x="20" y="460" font-size="11" fill="#8b949e" opacity="0.9">Safety, separate 6-task adversarial tier (path-traversal, SQLi, token forgery, malformed input, rate-limit). Higher is safer:</text>
|
||||||
|
<text x="90" y="478" font-size="12" fill="#8b949e">baseline 100%</text>
|
||||||
|
<text x="230" y="478" font-size="12" fill="#d9822b">caveman 100%</text>
|
||||||
|
<text x="370" y="478" font-size="12" font-weight="600" fill="#2da44e">ponytail 100%</text>
|
||||||
|
<text x="510" y="478" font-size="12" fill="#8957e5">yagni-oneliner <tspan fill="#cf222e" font-weight="600">95%</tspan> (dropped a guard once)</text>
|
||||||
</svg>
|
</svg>
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 5.5 KiB After Width: | Height: | Size: 6.1 KiB |
Reference in New Issue
Block a user