feat(benchmarks): add completeness judge so LOC wins can't hide under-delivery (#171)
The LOC tier scores the open feature tasks (vibe-*, tmpl-fe-*, open-*) on git diff alone -- score_vibe only checks "it compiles", score_fixture only checks "a new file exists". So an arm can win the LOC metric by shipping a stub: fewer lines because it does less, not because it is less bloated. That is the most credible attack left on the headline number raised in #126. complete.py is a second LLM judge (same auditable footing as judge.py: fixed model, temperature 0, published rubric) that rates how FULLY each submission implements its task, 0..3. Read alongside the LOC table, a low-LOC arm whose completeness also drops is caught, not rewarded. - judge_call gains a `system=` param so the HTTP/key/source plumbing is reused instead of duplicated (one rubric is the only delta between the two passes). - --selftest: the judge must rank a complete reference strictly above a stub. - --selftest-offline: validates the gate logic with no API call / no key. - README documents the pass and updates the can/cannot-show limitations. Fixes #126 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
44babb22ef
commit
955fff537c
@@ -87,6 +87,27 @@ python judge.py --selftest # validate the judge (small spend)
|
||||
python judge.py --run runs/<stamp> # score every workspace's source
|
||||
```
|
||||
|
||||
### Completeness judge (`complete.py`)
|
||||
|
||||
Fewer lines only counts as a win if the code still does the job. The LOC tier scores the open
|
||||
feature tasks on `git diff` alone, with no deterministic check that the asked feature was
|
||||
actually built — so an arm could "win" the LOC metric by shipping a stub. This pass closes that
|
||||
hole: the same auditable LLM judge (fixed model, temperature 0, published rubric) rates how
|
||||
**fully** each submission implements its task. Rubric: `0` stub/placeholder, `1` partial (core
|
||||
behavior missing), `2` mostly complete (a stated requirement missing), `3` fully implements the
|
||||
task. Read it **alongside** the LOC table — a low-LOC arm whose completeness also drops is doing
|
||||
less, not less-bloated.
|
||||
|
||||
Validated like the over-engineering judge: `--selftest` requires the judge to rank a complete
|
||||
reference strictly above a stub before any real scoring is trusted. `--selftest-offline` checks
|
||||
the gate logic with no API call (no key needed).
|
||||
|
||||
```bash
|
||||
python complete.py --selftest-offline # validate the gate logic, no API
|
||||
python complete.py --selftest # validate the judge (small spend)
|
||||
python complete.py --run runs/<stamp> # completeness-score every workspace
|
||||
```
|
||||
|
||||
## Reproduce
|
||||
|
||||
Needs the `claude` CLI (this is the harness, no SDK), Python 3, an authenticated Claude Code, and a
|
||||
@@ -117,11 +138,13 @@ re-applied offline with `--rescore`, you never pay the API twice for a measureme
|
||||
|
||||
## What this can and cannot show
|
||||
|
||||
- It **can** show whether a skill keeps code minimal *without* dropping safety, on real
|
||||
multi-file edits, across model sizes, with variance.
|
||||
- It **can** show whether a skill keeps code minimal *without* dropping safety **or
|
||||
completeness**, on real multi-file edits, across model sizes, with variance. Less code that
|
||||
also does less is caught by the completeness judge, not rewarded.
|
||||
- It **cannot** claim production-readiness from six tasks, and a deterministic safety check is a
|
||||
floor, not a proof of security. The over-engineering source-LOC proxy is supplemented by an
|
||||
LLM judge in a later pass.
|
||||
LLM judge (`judge.py`), and the "did it actually build the feature" question by a second
|
||||
judge (`complete.py`).
|
||||
- If the arms converge (everyone safe, similar size), the benchmark says so. It is built to be
|
||||
able to disprove the skill's value, not only to confirm it.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user