diff --git a/README.md b/README.md index c22df88..1c201f9 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ More survivors in [examples/](examples/). The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5.

- Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%. + Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%. Safety, separate adversarial tier: baseline, caveman and ponytail 100%, yagni-oneliner 95%.

| vs no-skill baseline | LOC | tokens | cost | time | safe | diff --git a/assets/benchmark-agentic.svg b/assets/benchmark-agentic.svg index 4336227..9ab4d28 100644 --- a/assets/benchmark-agentic.svg +++ b/assets/benchmark-agentic.svg @@ -1,5 +1,5 @@ - - Each arm vs the no-skill baseline across every metric, Claude Code on Haiku 4.5 + + Each arm vs the no-skill baseline across every metric, plus safety, Claude Code on Haiku 4.5 Every metric vs the no-skill baseline (Claude Code, Haiku 4.5, 12 tasks) baseline @@ -51,5 +51,12 @@ time base 69s - Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4. Writeup: benchmarks/results/2026-06-18-agentic.md + Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4. + + + Safety, separate 6-task adversarial tier (path-traversal, SQLi, token forgery, malformed input, rate-limit). Higher is safer: + baseline 100% + caveman 100% + ponytail 100% + yagni-oneliner 95% (dropped a guard once)