feat(benchmarks): add completeness judge so LOC wins can't hide under-delivery (#171)

The LOC tier scores the open feature tasks (vibe-*, tmpl-fe-*, open-*) on
git diff alone -- score_vibe only checks "it compiles", score_fixture only
checks "a new file exists". So an arm can win the LOC metric by shipping a
stub: fewer lines because it does less, not because it is less bloated.
That is the most credible attack left on the headline number raised in #126.

complete.py is a second LLM judge (same auditable footing as judge.py: fixed
model, temperature 0, published rubric) that rates how FULLY each submission
implements its task, 0..3. Read alongside the LOC table, a low-LOC arm whose
completeness also drops is caught, not rewarded.

- judge_call gains a `system=` param so the HTTP/key/source plumbing is reused
  instead of duplicated (one rubric is the only delta between the two passes).
- --selftest: the judge must rank a complete reference strictly above a stub.
- --selftest-offline: validates the gate logic with no API call / no key.
- README documents the pass and updates the can/cannot-show limitations.

Fixes #126

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ben Younes
2026-06-18 22:40:09 +02:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 44babb22ef
commit 955fff537c
3 changed files with 182 additions and 5 deletions
+26 -3
View File
@@ -87,6 +87,27 @@ python judge.py --selftest # validate the judge (small spend)
python judge.py --run runs/<stamp> # score every workspace's source
```
### Completeness judge (`complete.py`)
Fewer lines only counts as a win if the code still does the job. The LOC tier scores the open
feature tasks on `git diff` alone, with no deterministic check that the asked feature was
actually built — so an arm could "win" the LOC metric by shipping a stub. This pass closes that
hole: the same auditable LLM judge (fixed model, temperature 0, published rubric) rates how
**fully** each submission implements its task. Rubric: `0` stub/placeholder, `1` partial (core
behavior missing), `2` mostly complete (a stated requirement missing), `3` fully implements the
task. Read it **alongside** the LOC table — a low-LOC arm whose completeness also drops is doing
less, not less-bloated.
Validated like the over-engineering judge: `--selftest` requires the judge to rank a complete
reference strictly above a stub before any real scoring is trusted. `--selftest-offline` checks
the gate logic with no API call (no key needed).
```bash
python complete.py --selftest-offline # validate the gate logic, no API
python complete.py --selftest # validate the judge (small spend)
python complete.py --run runs/<stamp> # completeness-score every workspace
```
## Reproduce
Needs the `claude` CLI (this is the harness, no SDK), Python 3, an authenticated Claude Code, and a
@@ -117,11 +138,13 @@ re-applied offline with `--rescore`, you never pay the API twice for a measureme
## What this can and cannot show
- It **can** show whether a skill keeps code minimal *without* dropping safety, on real
multi-file edits, across model sizes, with variance.
- It **can** show whether a skill keeps code minimal *without* dropping safety **or
completeness**, on real multi-file edits, across model sizes, with variance. Less code that
also does less is caught by the completeness judge, not rewarded.
- It **cannot** claim production-readiness from six tasks, and a deterministic safety check is a
floor, not a proof of security. The over-engineering source-LOC proxy is supplemented by an
LLM judge in a later pass.
LLM judge (`judge.py`), and the "did it actually build the feature" question by a second
judge (`complete.py`).
- If the arms converge (everyone safe, similar size), the benchmark says so. It is built to be
able to disprove the skill's value, not only to confirm it.