ponytail

wangjunxiao/ponytail

Fork 0

Author	SHA1	Message	Date
EmerikoandClaude Opus 4.8	eed2a1c63e	docs(readme): add the agentic benchmark chart Grouped bars of LOC, tokens, cost and time as a % of the no-skill baseline (lower is leaner/cheaper/faster), plus a separate safety strip (baseline, caveman and ponytail 100%; yagni-oneliner 95%). System-gray palette so it reads on both GitHub themes. The chart commits landed after #158 had already squash-merged, so this brings the chart onto main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 17:08:36 +02:00
DietrichGebertandClaude Opus 4.8	b8d6aa7e9f	feat(benchmarks): agentic LOC + safety benchmark answering #126 (#158 ) Rebuild the benchmark to the standard #126 asked for: real headless Claude Code sessions (not a bare model) editing a real public repo (tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline, caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is the git diff; the safety tasks execute the produced code against adversarial input. Results: ponytail -54% LOC mean (up to -94% on over-build features like the date/color picker), -22% tokens, -20% cost, -27% time, and never more than baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal guard once). caveman writes less code but spends more tokens. Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook fired on every arm; now isolated with --setting-sources project,local + per-arm --plugin-dir) and a Windows subprocess-timeout hang. Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a labelled "isolated generation" note; supersede the contaminated 2026-06-17 writeup. Dead react-app fixture left untracked. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-18 16:42:38 +02:00

Author

SHA1

Message

Date

EmerikoandClaude Opus 4.8

eed2a1c63e

docs(readme): add the agentic benchmark chart

Grouped bars of LOC, tokens, cost and time as a % of the no-skill baseline
(lower is leaner/cheaper/faster), plus a separate safety strip (baseline,
caveman and ponytail 100%; yagni-oneliner 95%). System-gray palette so it reads
on both GitHub themes. The chart commits landed after #158 had already
squash-merged, so this brings the chart onto main.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-18 17:08:36 +02:00

DietrichGebertandClaude Opus 4.8

b8d6aa7e9f

feat(benchmarks): agentic LOC + safety benchmark answering #126 (#158 )

Rebuild the benchmark to the standard #126 asked for: real headless Claude Code
sessions (not a bare model) editing a real public repo
(tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline,
caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is
the git diff; the safety tasks execute the produced code against adversarial
input.

Results: ponytail -54% LOC mean (up to -94% on over-build features like the
date/color picker), -22% tokens, -20% cost, -27% time, and never more than
baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal
guard once). caveman writes less code but spends more tokens.

Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook
fired on every arm; now isolated with --setting-sources project,local + per-arm
--plugin-dir) and a Windows subprocess-timeout hang.

Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a
labelled "isolated generation" note; supersede the contaminated 2026-06-17
writeup. Dead react-app fixture left untracked.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-18 16:42:38 +02:00

Compare commits

Diff Content Not Available