wangjunxiao/ponytail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(benchmarks): agentic LOC + safety benchmark answering #126

Browse Source 
Rebuild the benchmark to the standard #126 asked for: real headless Claude Code
sessions (not a bare model) editing a real public repo
(tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline,
caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is
the git diff; the safety tasks execute the produced code against adversarial
input.

Results: ponytail -54% LOC mean (up to -94% on over-build features like the
date/color picker), -22% tokens, -20% cost, -27% time, and never more than
baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal
guard once). caveman writes less code but spends more tokens.

Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook
fired on every arm; now isolated with --setting-sources project,local + per-arm
--plugin-dir) and a Windows subprocess-timeout hang.

Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a
labelled "isolated generation" note; supersede the contaminated 2026-06-17
writeup. Dead react-app fixture left untracked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Emeriko
2026-06-18 16:19:48 +02:00
co-authored by Claude Opus 4.8
parent 45f7d2f83f
commit 75fea8f743
9 changed files with 1644 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/README.md
+9
View File
@@ -61,6 +61,15 @@ Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limi
Versus baseline, ponytail writes **80-94% less code**, costs **42-75% less**, and runs **3-6x faster**, on every Claude model. Cost re-verified at 30 reps, with OpenAI and Gemini arms, in [results/2026-06-17-cost-verification.md](results/2026-06-17-cost-verification.md).
> **Read this number honestly (updated 2026-06-18).** The gap above is single-shot, against a bare
> model that answers with several options plus commentary, so it counts prose, not just code, and
> overstates the win. [#126](https://github.com/DietrichGebert/ponytail/issues/126) was right about
> that. The [agentic benchmark](agentic/) re-runs the comparison as a *real Claude Code session on a
> real public repo*: ponytail cuts **60-94%** on features with an over-build trap (custom component
> vs native input), is a wash on already-minimal code, never writes more, and stays **100% safe**
> while the bare "one-liner" prompt drops a guard. That is the honest, defensible number. See
> [results/2026-06-18-agentic.md](results/2026-06-18-agentic.md).
## Metrics
| File | Metric | Behavior |