DietrichGebertandClaude Opus 4.8
b8d6aa7e9f
Rebuild the benchmark to the standard #126 asked for: real headless Claude Code sessions (not a bare model) editing a real public repo (tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline, caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is the git diff; the safety tasks execute the produced code against adversarial input. Results: ponytail -54% LOC mean (up to -94% on over-build features like the date/color picker), -22% tokens, -20% cost, -27% time, and never more than baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal guard once). caveman writes less code but spends more tokens. Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook fired on every arm; now isolated with --setting-sources project,local + per-arm --plugin-dir) and a Windows subprocess-timeout hang. Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a labelled "isolated generation" note; supersede the contaminated 2026-06-17 writeup. Dead react-app fixture left untracked. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
27 lines
506 B
Plaintext
27 lines
506 B
Plaintext
# Secrets, never commit API keys
|
|
.env
|
|
.env.*
|
|
!.env.example
|
|
|
|
# Dependencies
|
|
node_modules/
|
|
|
|
# promptfoo eval artifacts
|
|
.promptfoo/
|
|
benchmarks/output*
|
|
benchmarks/benchmark-local-results.json
|
|
|
|
# Python
|
|
__pycache__/
|
|
|
|
# one-off social/announcement art, not repo content
|
|
announce-*.png
|
|
changelog-*.png
|
|
ponytail-*.gif
|
|
|
|
# Claude Code local settings (machine-specific permission grants)
|
|
.claude/settings.local.json
|
|
|
|
# agentic benchmark workspaces (agent output, kept locally for inspection)
|
|
benchmarks/agentic/runs/
|