The LOC tier scores the open feature tasks (vibe-*, tmpl-fe-*, open-*) on
git diff alone -- score_vibe only checks "it compiles", score_fixture only
checks "a new file exists". So an arm can win the LOC metric by shipping a
stub: fewer lines because it does less, not because it is less bloated.
That is the most credible attack left on the headline number raised in #126.
complete.py is a second LLM judge (same auditable footing as judge.py: fixed
model, temperature 0, published rubric) that rates how FULLY each submission
implements its task, 0..3. Read alongside the LOC table, a low-LOC arm whose
completeness also drops is caught, not rewarded.
- judge_call gains a `system=` param so the HTTP/key/source plumbing is reused
instead of duplicated (one rubric is the only delta between the two passes).
- --selftest: the judge must rank a complete reference strictly above a stub.
- --selftest-offline: validates the gate logic with no API call / no key.
- README documents the pass and updates the can/cannot-show limitations.
Fixes#126
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The ponytail/caveman arms hardcoded one machine's Windows plugin-cache
paths (C:\Users\Dietr\...), so only baseline/yagni/yagni-oneliner were
reproducible off the maintainer's box — undercutting the "fully
reproducible" claim the rebuilt benchmark (#126) was meant to establish.
Resolve per-arm at use-site: env override (PONYTAIL_PLUGIN_DIR /
CAVEMAN_PLUGIN_DIR) -> latest version dir under ~/.claude/plugins/cache
-> clear sys.exit. No pinned version/hash. Selftest extended to cover
env-override and missing-install paths.
Fixes#169
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rebuild the benchmark to the standard #126 asked for: real headless Claude Code
sessions (not a bare model) editing a real public repo
(tiangolo/full-stack-fastapi-template @ cd83fc1, MIT), fair arms (baseline,
caveman, ponytail, and the "YAGNI + one-liners" prompt), n=4, Haiku 4.5. LOC is
the git diff; the safety tasks execute the produced code against adversarial
input.
Results: ponytail -54% LOC mean (up to -94% on over-build features like the
date/color picker), -22% tokens, -20% cost, -27% time, and never more than
baseline; 100% safe vs the one-liner prompt's 95% (it dropped a path-traversal
guard once). caveman writes less code but spends more tokens.
Also fixes a baseline-contamination bug (the ponytail plugin's SessionStart hook
fired on every arm; now isolated with --setting-sources project,local + per-arm
--plugin-dir) and a Windows subprocess-timeout hang.
Lead both READMEs with the agentic numbers; demote the single-shot 80-94% to a
labelled "isolated generation" note; supersede the contaminated 2026-06-17
writeup. Dead react-app fixture left untracked.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
