feat(benchmarks): add completeness judge so LOC wins can't hide under-delivery (#171)
The LOC tier scores the open feature tasks (vibe-*, tmpl-fe-*, open-*) on git diff alone -- score_vibe only checks "it compiles", score_fixture only checks "a new file exists". So an arm can win the LOC metric by shipping a stub: fewer lines because it does less, not because it is less bloated. That is the most credible attack left on the headline number raised in #126. complete.py is a second LLM judge (same auditable footing as judge.py: fixed model, temperature 0, published rubric) that rates how FULLY each submission implements its task, 0..3. Read alongside the LOC table, a low-LOC arm whose completeness also drops is caught, not rewarded. - judge_call gains a `system=` param so the HTTP/key/source plumbing is reused instead of duplicated (one rubric is the only delta between the two passes). - --selftest: the judge must rank a complete reference strictly above a stub. - --selftest-offline: validates the gate logic with no API call / no key. - README documents the pass and updates the can/cannot-show limitations. Fixes #126 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
44babb22ef
commit
955fff537c
@@ -61,10 +61,10 @@ def source_text(workdir: Path):
|
||||
except Exception: continue
|
||||
return "\n\n".join(out)
|
||||
|
||||
def judge_call(task_prompt, files, key, retries=3):
|
||||
def judge_call(task_prompt, files, key, retries=3, system=RUBRIC):
|
||||
user = f"TASK GIVEN TO THE AUTHOR:\n{task_prompt}\n\nFILES THEY WROTE:\n{files}"
|
||||
body = json.dumps({"model": JUDGE_MODEL, "max_tokens": 300, "temperature": 0,
|
||||
"system": RUBRIC, "messages": [{"role": "user", "content": user}]}).encode()
|
||||
"system": system, "messages": [{"role": "user", "content": user}]}).encode()
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
req = urllib.request.Request("https://api.anthropic.com/v1/messages", data=body,
|
||||
|
||||
Reference in New Issue
Block a user