From b742c9c07c89df9936e0e6f7c5e41199f7d3ba7f Mon Sep 17 00:00:00 2001 From: Emeriko Date: Thu, 18 Jun 2026 17:01:28 +0200 Subject: [PATCH] docs(readme): add the safety tier to the benchmark chart Separate strip below the metric clusters showing the 6-task adversarial safe rate (higher is safer): baseline, caveman and ponytail 100%, yagni-oneliner 95% (flagged). Kept apart from the lower-is-better % cluster so the opposite polarity and different task set are not conflated. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 2 +- assets/benchmark-agentic.svg | 13 ++++++++++--- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index c22df88..1c201f9 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ More survivors in [examples/](examples/). The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5.

- Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%. + Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%. Safety, separate adversarial tier: baseline, caveman and ponytail 100%, yagni-oneliner 95%.

| vs no-skill baseline | LOC | tokens | cost | time | safe | diff --git a/assets/benchmark-agentic.svg b/assets/benchmark-agentic.svg index 4336227..9ab4d28 100644 --- a/assets/benchmark-agentic.svg +++ b/assets/benchmark-agentic.svg @@ -1,5 +1,5 @@ - - Each arm vs the no-skill baseline across every metric, Claude Code on Haiku 4.5 + + Each arm vs the no-skill baseline across every metric, plus safety, Claude Code on Haiku 4.5 Every metric vs the no-skill baseline (Claude Code, Haiku 4.5, 12 tasks) baseline @@ -51,5 +51,12 @@ time base 69s - Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4. Writeup: benchmarks/results/2026-06-18-agentic.md + Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4. + + + Safety, separate 6-task adversarial tier (path-traversal, SQLi, token forgery, malformed input, rate-limit). Higher is safer: + baseline 100% + caveman 100% + ponytail 100% + yagni-oneliner 95% (dropped a guard once)