feat(benchmarks): add completeness judge so LOC wins can't hide under-delivery (#171)

The LOC tier scores the open feature tasks (vibe-*, tmpl-fe-*, open-*) on
git diff alone -- score_vibe only checks "it compiles", score_fixture only
checks "a new file exists". So an arm can win the LOC metric by shipping a
stub: fewer lines because it does less, not because it is less bloated.
That is the most credible attack left on the headline number raised in #126.

complete.py is a second LLM judge (same auditable footing as judge.py: fixed
model, temperature 0, published rubric) that rates how FULLY each submission
implements its task, 0..3. Read alongside the LOC table, a low-LOC arm whose
completeness also drops is caught, not rewarded.

- judge_call gains a `system=` param so the HTTP/key/source plumbing is reused
  instead of duplicated (one rubric is the only delta between the two passes).
- --selftest: the judge must rank a complete reference strictly above a stub.
- --selftest-offline: validates the gate logic with no API call / no key.
- README documents the pass and updates the can/cannot-show limitations.

Fixes #126

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ben Younes
2026-06-18 22:40:09 +02:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 44babb22ef
commit 955fff537c
3 changed files with 182 additions and 5 deletions
+2 -2
View File
@@ -61,10 +61,10 @@ def source_text(workdir: Path):
except Exception: continue
return "\n\n".join(out)
def judge_call(task_prompt, files, key, retries=3):
def judge_call(task_prompt, files, key, retries=3, system=RUBRIC):
user = f"TASK GIVEN TO THE AUTHOR:\n{task_prompt}\n\nFILES THEY WROTE:\n{files}"
body = json.dumps({"model": JUDGE_MODEL, "max_tokens": 300, "temperature": 0,
"system": RUBRIC, "messages": [{"role": "user", "content": user}]}).encode()
"system": system, "messages": [{"role": "user", "content": user}]}).encode()
for attempt in range(retries):
try:
req = urllib.request.Request("https://api.anthropic.com/v1/messages", data=body,