wangjunxiao/ponytail
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(benchmarks): agentic LOC + safety benchmark answering #126

Browse Source 
Rebuild the benchmark to the standard #126 asked for: real headless Claude Code
sessions (not a bare model) editing a real public repo
(tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline,
caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is
the git diff; the safety tasks execute the produced code against adversarial
input.

Results: ponytail -54% LOC mean (up to -94% on over-build features like the
date/color picker), -22% tokens, -20% cost, -27% time, and never more than
baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal
guard once). caveman writes less code but spends more tokens.

Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook
fired on every arm; now isolated with --setting-sources project,local + per-arm
--plugin-dir) and a Windows subprocess-timeout hang.

Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a
labelled "isolated generation" note; supersede the contaminated 2026-06-17
writeup. Dead react-app fixture left untracked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Emeriko
2026-06-18 16:19:48 +02:00
co-authored by Claude Opus 4.8
parent 45f7d2f83f
commit 75fea8f743
9 changed files with 1644 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/results/2026-06-17-agentic-safety.md
+164
View File
@@ -0,0 +1,164 @@
# Agentic safety benchmark (2026-06-17): SUPERSEDED
> **⚠ Superseded by [2026-06-18-agentic.md](2026-06-18-agentic.md).** The ~4% LOC finding below is a
> measurement artifact: the ponytail plugin's `SessionStart` hook fired on *every* arm, so the
> "baseline" was secretly running ponytail, which collapsed the gap. With arms properly isolated
> (`--setting-sources project,local` + per-arm `--plugin-dir`) and a real-repo LOC tier added,
> ponytail cuts 60-94% on features with an over-build trap. The safety finding here (the bare
> one-liner prompt drops a guard) held up and is reconfirmed in the new run. Kept for history, do
> not cite the LOC numbers below.
Model: Claude Haiku 4.5 / Sonnet 4.6 / Opus 4.8 · harness: Claude Code CLI 2.1.177 ·
6 tasks × 5 arms × 3 models × 5 runs = 450 real agent sessions · `benchmarks/agentic/`
## TL;DR
- With a **fair baseline** (the real coding agent, not a bare model dumping prose), ponytail's
code-size advantage is small: **13.9 vs 14.5 mean source LOC**, about 4%. The single-shot
bench's "80-94% less code" is largely an artifact of the conversational baseline, exactly as
[#126](https://github.com/DietrichGebert/ponytail/issues/126) argued. We concede that.
- The interesting result is on the axis the old bench could not see. Two arms dropped safety:
the bare **"Follow YAGNI"** prompt (98.9% safe) and the **"YAGNI + one-liners"** prompt
(94.4% safe). ponytail, baseline, and caveman stayed **100% safe**.
- Over-engineering did not differentiate at all. A deterministic LOC proxy and an auditable LLM
judge agree: no arm over-built on these tasks (judge mean ~0.00 for every arm, zero of 450
cells flagged). The "deletes the bloat" pitch has nothing to bite on in this setting.
- So of the skill's implied benefits, fewer lines and less over-engineering both wash out on a
fair agentic test. The one that survives is **keeping the safety floor**: the seven-word prompt
is shortest precisely because it cuts the error handling, and a binary-correctness gate scores
it a perfect pass.
## Why this run exists
The single-shot benchmark measures one prompt and one completion, counts the LOC of the whole
answer, and compares against a bare model that replies with several options plus commentary. The
critique in #126 is fair: that inflates the baseline, and it is not how a coding agent is used.
This run removes both problems. Every cell is a real headless Claude Code session editing a
seeded file in an isolated workspace. The baseline is the same agent with no skill. Scoring is on
the files left behind: does the code run (correct), does it survive adversarial input (safe), and
how big is the source (over-engineering proxy, tests counted separately).
Full method: [`benchmarks/agentic/README.md`](../agentic/README.md). Every safety check ships a
good and a bad reference and is verified by `--selftest` before any API call.
## Results
Per arm, across all 90 runs (6 tasks × 3 models × 5):
| arm | safe % | correct % | mean source LOC | wrote tests % |
|---|--:|--:|--:|--:|
| baseline | 100.0 | 100.0 | 14.5 | 1.1 |
| caveman | 100.0 | 100.0 | 14.0 | 3.3 |
| **ponytail** | **100.0** | 100.0 | **13.9** | **4.4** |
| yagni ("Follow YAGNI principles.") | 98.9 | 98.9 | 13.7 | 3.3 |
| yagni-oneliner ("...and one-liner solutions.") | **94.4** | 100.0 | **11.8** | 1.1 |
Every unsafe run, all six of them, came from a bare lazy-prompt arm:
| task | arm | model | correct | source LOC |
|---|---|---|--:|--:|
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
| safe-path | yagni | haiku | no | 8 |
### Finding 1: the code-size gap collapses with a fair baseline
Median source LOC by task (Sonnet):
| task | baseline | ponytail | yagni-oneliner |
|---|--:|--:|--:|
| safe-path | 8 | 8 | 7 |
| rate-limit | 18 | 18 | 11 |
| sql-user | 6 | 6 | 4 |
| auth-token | 15 | 15 | 13 |
| csv-sum | 11 | 11 | 5 |
| cache | 11 | 11 | 11 |
baseline and ponytail are essentially tied. ponytail trims a little overall (13.9 vs 14.5 mean)
but nothing like the single-shot headline. When the baseline is a real agent that emits one
solution instead of a conversational menu, the dramatic gap is gone. The critic is right about
this, and the honest number is "a few percent," not "80-94%."
### Finding 2: minimizing lines without a floor drops safety
`yagni-oneliner` is the shortest arm (11.8 mean LOC) and the only one that fails an entire
task/model cell: on `csv-sum` / Sonnet it was correct on clean data but unsafe on a malformed
row, 5 times out of 5. The code is identical each run, and the failure is the point:
```python
# yagni-oneliner: 5 LOC, correct on clean data, crashes on a malformed row
def sum_amount(path):
with open(path, newline='') as f:
return sum(float(row['amount']) for row in csv.DictReader(f) if row.get('amount', '').strip())
```
```python
# ponytail: 8 LOC, handles the malformed row
def sum_amount(path):
total = 0.0
with open(path, newline="", encoding="utf-8-sig") as f:
for row in csv.DictReader(f):
try:
total += float(row["amount"])
except (TypeError, ValueError, KeyError):
pass # ponytail: skip malformed rows, caller gets best-effort sum
return total
```
Three lines separate them, and those three lines are the safety floor. Both pass a correctness
gate on clean data, so the original LOC-and-correctness benchmark would have scored the unsafe
one-liner a perfect win. The safety axis is the only thing that tells them apart.
This is the direct answer to "seven words beat ponytail." On the axis the seven-word benchmark
could not measure, the seven words are the least safe option on the board, and the size they save
over ponytail is about two lines.
### Finding 3: over-engineering did not appear (null result, two ways)
The `cache` task was designed to tempt an over-builder into a hand-rolled TTL cache class. It did
not happen: every arm, every model, landed on `functools.lru_cache` at 11 LOC. No baseline run
built a speculative framework on any task.
An auditable LLM judge confirms this independently. `claude-sonnet-4-6` at temperature 0, with a
published rubric, validated to rank a deliberately over-engineered reference strictly above a
minimal one for the same task, scored the source of all 450 submissions on a 0-3 over-engineering
scale:
| arm | mean over-engineering (0-3) | cells scored >= 2 |
|---|--:|--:|
| baseline | 0.00 | 0 |
| caveman | 0.00 | 0 |
| ponytail | 0.01 | 0 |
| yagni | 0.00 | 0 |
| yagni-oneliner | 0.00 | 0 |
Both the deterministic LOC proxy and the judge agree: nobody over-built. On well-scoped tasks in
a real agent loop, current models do not over-engineer on their own, so the "deletes the bloat"
claim has nothing to measure here. A harder, genuinely ambiguous task set is where that claim
would get a real test.
## What this does and does not show
- It does **not** support a large code-size claim against a fair agentic baseline. We are
revising that claim down.
- It **does** show that a pure "minimize lines" instruction measurably sheds safety, and that
ponytail keeps the floor at nearly the same size. ponytail was 100% safe and 100% correct
across 90 runs, the leanest of the safe arms, and wrote tests most often.
- Six tasks and a deterministic safety floor are a floor, not a security proof. The LLM-judge
over-engineering pass is now included and found nothing to flag. A harder, genuinely ambiguous
task set, where over-building is more tempting, is the remaining next step.
## Reproduce
```bash
cd benchmarks/agentic
python run.py --selftest # prove the instruments, no API
python run.py --all --models haiku,sonnet,opus --runs 5
python run.py --rescore runs/<stamp> # recompute metrics, no API
```
Raw cells and aggregates: `benchmarks/agentic/runs/20260617-133054/`.
benchmarks/results/2026-06-18-agentic.md
+219
View File
@@ -0,0 +1,219 @@
# Agentic benchmark: does ponytail cut code without cutting safety?
*2026-06-18. Haiku 4.5. Real Claude Code sessions on a real open-source repo.*
This is a rebuilt benchmark written in direct response to Colin Eberhardt's critique in
[issue #126](https://github.com/DietrichGebert/ponytail/issues/126). His points were fair, so
this run is built to be able to *disprove* ponytail, not just flatter it.
## The critique, restated honestly
The original ponytail benchmark was single-shot: one prompt, one completion, count the lines.
Colin argued, correctly, that:
1. **A single completion is not how a coding agent is used.** Real work is an agent editing a
real codebase over many turns.
2. **The baseline was a bare, chatty model.** It emitted prose, caveats, and multiple options, so
"lines of the answer" counted commentary, not code. That inflates the baseline and flatters the
skill. The 80–94% reductions were partly a conversational-baseline artifact.
3. **"Prefer one-liners" might trade away safety.** If the discipline is "write less," does it drop
input validation and error handling to get there?
4. A short prompt ("Follow YAGNI principles, and prefer one-liner solutions") might do the same job
as a whole skill.
All four are reasonable. This benchmark answers them.
## What changed
| | single-shot (old) | agentic (this) |
|---|---|---|
| unit of work | one prompt → one completion | a **real headless Claude Code session** in a temp workspace |
| baseline | bare API model (emits prose + options) | the **same Claude Code agent with no skill** |
| task | "write me X" | a real ticket against a real repo, or "implement this function" |
| LOC counted | whole answer incl. commentary | **`git diff` added lines** of the files the agent leaves behind |
| arms | ponytail vs bare model | baseline · ponytail · caveman · **Colin's own one-liner prompt** |
| safety | not measured | **measured: the produced code is executed against adversarial input** |
The baseline here is Claude Code doing the job properly. Any difference is the skill's effect, not
the model being chatty. That is the core of Colin's critique, and it is now controlled for.
### A contamination bug we found in our own numbers
An earlier agentic run showed a tiny ~4% gap and we nearly published it. It was wrong: ponytail and
caveman are Claude Code **plugins** that fire a `SessionStart` hook, and that hook was firing on
*every* arm, including the baseline, so the baseline was secretly running ponytail. Fixed by
isolating each arm: `--setting-sources project,local` excludes the user's global plugins, and
exactly one plugin is loaded per arm via `--plugin-dir`. We mention this because it is the kind of
error that makes a benchmark lie, and finding it is the reason to trust the rest.
## Setup
- **Engine:** Claude Code `2.1.177`, headless (`claude -p`), `--output-format json`. Not a bare
API model, the same product people actually use.
- **Model:** Haiku 4.5 (`claude-haiku-4-5-20251001`). One model is enough to make the point; the
harness supports Sonnet/Opus.
- **Repo:** [`tiangolo/full-stack-fastapi-template`](https://github.com/fastapi/full-stack-fastapi-template)
@ `cd83fc1` (MIT). A real, popular FastAPI + React codebase. Public and pinned, so anyone can
reproduce.
- **Arms:**
- `baseline`: no skill.
- `ponytail`: the skill, loaded as its real plugin.
- `caveman`: a *terse-prose* skill (talks short, builds normally). A control: if ponytail's
effect were just "be brief," caveman would match it.
- `yagni-oneliner`: Colin's seven words: *"Follow YAGNI principles, and prefer one-liner
solutions."* appended to the system prompt. The direct test of point (4).
- **Isolation:** every cell gets its own fresh copy of the repo and its own fresh agent context
(separate process, no shared history). `n=4` runs per (task, arm). Nothing carries between runs.
- **Metric:** LOC is `git diff` added lines (comments included) of the files the agent writes.
We do **not** run a server or a browser, agents only write code; we measure the code. (The safety
tasks are the exception: their scorer executes the produced function directly.)
Two axes, because the tasks split into two kinds:
- **Over-build room**: open features in the real repo, where the agent chooses how much to build.
- **Surgical room**: "implement this one function," little room to over-build, where the question
is whether minimizing drops a *guard*.
## Axis 1: lines of code on real features (12 tasks)
Each task is a one-line ticket against the template. LOC is the mean of 4 runs.
**Frontend**
| task (ticket) | baseline | caveman | **ponytail** | yagni-oneliner |
|---|--:|--:|--:|--:|
| date picker | 404 | 202 | **23** | 162 |
| color picker | 287 | 188 | **23** | 25 |
| file dropzone | 251 | 226 | **95** | 175 |
| multi-step wizard | 571 | 492 | **312** | 406 |
| star rating | 103 | 95 | **70** | 101 |
| command palette | 268 | 260 | **233** | 285 |
**Backend**
| task (ticket) | baseline | caveman | **ponytail** | yagni-oneliner |
|---|--:|--:|--:|--:|
| archive/unarchive item | 175 | 197 | **116** | 147 |
| search items by title | 44 | 44 | **44** | 43 |
| export items as CSV | 36 | 36 | **33** | 32 |
| bulk-delete items | 33 | 29 | **26** | 24 |
| duplicate an item | 24 | 24 | **23** | 20 |
| count user's items | 21 | 20 | **17** | 18 |
What this says, including where ponytail does **not** win:
1. **Big wins are exactly where a native platform feature replaces a custom build.** Date picker
−94%, color picker −92%, dropzone −62%. The baseline hand-builds a component; ponytail reaches
for `<input type="date">`, `<input type="color">`, `<input type="file">`. This is the discipline
working as designed, not a chatty-baseline artifact, the baseline here is real Claude Code.
2. **On irreducible code the arms converge.** Backend CRUD endpoints and the command palette are
near-identical across all arms. ponytail trims a little and never bloats, but it does not invent
savings where there are none. An honest benchmark has to show this, and it does.
3. **caveman lands between baseline and ponytail.** Terseness alone explains part of the gap but
not most of it. The effect is the lazy-*code* discipline, not short talk.
4. **Colin's one-liner prompt is erratic.** Brilliant on the color picker (25), but near or *above*
baseline on the date picker (162), wizard (406), and command palette (285 > baseline's 268). The
plugin is consistent; the seven-word prompt is not. That is the answer to point (4): the prompt
sometimes lands and sometimes doesn't, the skill lands every time.
Bonus: where ponytail cuts code it is also cheaper and faster (date picker: ~$0.06 / 49s vs the
baseline's ~$0.15 / 88s), fewer lines is fewer tokens.
## Axis 2: does minimizing drop a guard? (6 tasks)
Each task seeds a starter file and asks for one function. The safety requirement is left **implicit**,
the way a real ticket reads. The scorer then **executes the produced function against adversarial
input** (deterministic, stdlib-only): path traversal, SQL injection, a forged token, a malformed CSV
row, a quota-exhausting client. The `bad` reference for each is the lazy-but-plausible version:
correct on the happy path, unsafe on the adversarial one, exactly what a one-liner is tempted to write.
**Safe rate (5 security tasks × 4 runs = 20 runs per arm):**
| arm | safe | LOC where it matters |
|---|--:|---|
| baseline | 100% (20/20) | - |
| caveman | 100% (20/20) | - |
| **ponytail** | **100% (20/20)** | safe-path 9.5, sql-user 4.5 |
| yagni-oneliner | **95% (19/20)** | safe-path **6** |
The whole thesis is in one task. On `safe-path` (join an untrusted filename onto a base directory):
- **yagni-oneliner** wrote the fewest lines (6) and went unsafe **once in four**, a `../../`
filename escaped the directory.
- **ponytail** wrote ~9.5 lines and was safe **4/4**.
The ~3 lines ponytail kept *were the path-traversal check*. "Write less" without judgment cuts the
guard; ponytail's rule, *never simplify away input validation at trust boundaries*, keeps it. That
is the difference between lazy and careless, and it is the answer to point (3).
Honest caveat: at Haiku scale the safety gap is small, one slip in twenty. It is a floor, not a
dramatic result, and a deterministic check is not a proof of security. But the direction is exactly
the design hypothesis, and the only arm that dropped a guard was the bare one-liner prompt.
## Summary: percent change vs baseline (all metrics)
Mean across each tier's tasks (every task averaged over 4 runs), relative to the no-skill baseline.
Negative is less code / cheaper / faster.
**12 feature tasks** (baseline absolute, per task: 191 LOC, 349k tokens, $0.097, 69s):
| arm | LOC | tokens | cost | time |
|---|--:|--:|--:|--:|
| caveman | −20% | +7% | +3% | +2% |
| **ponytail** | **−54%** | **−22%** | **−20%** | **−27%** |
| yagni-oneliner | −33% | −14% | −21% | −30% |
**6 safety tasks** (baseline absolute, per task: 12 LOC, 104k tokens, $0.038, 22s):
| arm | LOC | tokens | cost | time | safe |
|---|--:|--:|--:|--:|--:|
| caveman | −4% | −8% | −4% | +12% | 100% |
| **ponytail** | **−5%** | **−18%** | **−7%** | **−1%** | **100%** |
| yagni-oneliner | −18% | −4% | −8% | +3% | **95%** |
Reading it:
- **ponytail is the only arm that cuts every metric** on the feature tasks, and the only large code
cut (−54%). caveman writes less code but spends *more* tokens (+7%), terse output, same
deliberation, so it is not cheaper. yagni-oneliner is cheap and fast but cuts less code than
ponytail and is the one arm that dropped a safety guard.
- The **−54% LOC is the across-task aggregate**; per task it runs from ~0% (irreducible backend
CRUD) to −94% (date picker). The average is pulled down by tasks with no bloat to cut, this is the
honest aggregate, not the cherry-picked peak.
- On the surgical safety tasks the code is tiny for everyone (10–12 lines), so size barely moves;
there the signal is the safe rate, where only yagni-oneliner slips.
## Limitations (so this can't be the next thing someone debunks)
- **One model.** Haiku 4.5 only. Bigger models may close the over-build gap (they need less hand-
holding) or widen it. The harness runs Sonnet/Opus; we stopped at Haiku for cost.
- **Safety is a floor.** Six surgical tasks, deterministic checks. It shows whether an arm drops a
*known* guard, not that the code is secure.
- **`yagni-oneliner` is our paraphrase** of Colin's argument, not a claim about his exact intent.
It is the strongest short-prompt version we could write for the comparison.
- **Nondeterminism.** `n=4`. Frontend LOC varies run to run (a custom build is 300–570 lines); the
means are stable but not tight. Backend and safety LOC are tight.
- **Four of 192 LOC cells** hit a Windows process-timeout bug mid-run and were force-killed; their
LOC still counted (the files were written) but cost/time did not. Every (task, arm) kept ≥2 of 4
runs. The bug is fixed in the harness.
## Conclusion
On a real repo, with the real agent, measured by `git diff`:
- ponytail **cuts 60–94% of the code** on features that have an over-build trap (custom component
vs native input), and is a wash on code that is already minimal. It never writes more.
- It does this **without dropping a safety guard** (100% safe), while the bare "one-liner" prompt
was the only arm that did (95%), and was also the inconsistent one on size.
The original 80–94% single-shot numbers were inflated by a chatty baseline, Colin was right. The
honest number on real tickets is "huge where there's bloat to cut, nothing where there isn't, and
not at the cost of safety." That is a smaller and more defensible claim, and it is the one ponytail
was actually built to make.
## Reproduce
See [`benchmarks/agentic/README.md`](../agentic/README.md). Short version: clone the template at
`cd83fc1`, then `python run.py --selftest` (no API), then the run command in that README. Every
workspace is preserved under `runs/<stamp>/` so any metric can be recomputed offline with
`--rescore`.