Separate strip below the metric clusters showing the 6-task adversarial safe
rate (higher is safer): baseline, caveman and ponytail 100%, yagni-oneliner 95%
(flagged). Kept apart from the lower-is-better % cluster so the opposite
polarity and different task set are not conflated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grouped bars showing LOC, tokens, cost and time, each as a % of the no-skill
baseline (baseline = 100%, lower is leaner/cheaper/faster), instead of LOC
alone. Makes ponytail's win on every metric visible and shows caveman rising
above 100% on tokens/cost/time. Same system-gray palette so it reads on both
GitHub themes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bar chart of mean LOC per task by arm (Haiku 4.5) in the Numbers section, same
system-gray style as the single-shot chart so it reads on both GitHub themes.
The old single-shot chart stays in the collapsed "isolated generation" section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild the benchmark to the standard #126 asked for: real headless Claude Code
sessions (not a bare model) editing a real public repo
(tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline,
caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is
the git diff; the safety tasks execute the produced code against adversarial
input.
Results: ponytail -54% LOC mean (up to -94% on over-build features like the
date/color picker), -22% tokens, -20% cost, -27% time, and never more than
baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal
guard once). caveman writes less code but spends more tokens.
Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook
fired on every arm; now isolated with --setting-sources project,local + per-arm
--plugin-dir) and a Windows subprocess-timeout hang.
Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a
labelled "isolated generation" note; supersede the contaminated 2026-06-17
writeup. Dead react-app fixture left untracked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
