feat: refine ruleset from a full-project field review (#39)
* feat: refine ruleset from a full-project field review
A reviewer ran ponytail across a 9-phase rewrite (protocol, PC app, simulator,
RPi daemon, ESP32 firmware) and flagged three gaps. All three land in SKILL.md
and propagate to AGENTS.md + the rule copies:
- Promote the one-runnable-check rule to a headline ("Lazy code without its
check is unfinished"), enforced as a check-rule-copies invariant.
- Hardware carve-out in "When NOT to be lazy": a real device is never the spec
ideal (clock drift, sensor offset), leave the calibration knob.
- Clarify the Output rule: explanation the user explicitly asked for is not
debt, only unrequested prose is.
Fallback instructions kept in sync. Rule-copy check + tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add a behavior gate proving the refinements actually fire
The refinements were verified as injected text, but injected != behavioral.
This adds a behavior eval that probes each refined rule on a task that should
trigger it:
- hardware -> does the output leave a calibration knob?
- explanation -> when a write-up is explicitly requested, is it given in full?
- onecheck -> is a runnable check left behind?
benchmarks/behavior.yaml runs the probes (baseline vs ponytail arm); the
grader benchmarks/behavior.js is proven by tests/behavior.test.js (8 cases,
RED/GREEN, no API key, runs in CI). Live-confirmed: the model under the
current ruleset passes all three gates, graded by the same grader.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b545f1536a
commit
f3da910b4f
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env node
|
||||
// Unit test for the behavior gate (benchmarks/behavior.js). Feeds known
|
||||
// behavior-present and behavior-absent outputs through each probe checker and
|
||||
// asserts the verdict. Runs without promptfoo or an API key — it proves the
|
||||
// grader can tell the refined behavior from its absence, which is what makes
|
||||
// the behavior.yaml eval trustworthy.
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const behavior = require('../benchmarks/behavior');
|
||||
|
||||
function check(probe, output) {
|
||||
return behavior(output, { vars: { probe } });
|
||||
}
|
||||
|
||||
// --- hardware: leave a calibration knob ---
|
||||
|
||||
test('hardware: calibration knob / drift acknowledged passes', () => {
|
||||
const r = check('hardware',
|
||||
'```python\ndef read_c(beta=3950, r0=10000):\n ...\n```\n' +
|
||||
'Notes: beta/r0 drift part-to-part, measure your own r0 at a known temp.');
|
||||
assert.equal(r.pass, true);
|
||||
assert.equal(r.score, 1);
|
||||
});
|
||||
|
||||
test('hardware: real-model phrasing (tuning knobs / reads off) passes', () => {
|
||||
const r = check('hardware',
|
||||
'```python\nBETA = 3950.0 # thermistor beta -- calibration knob\n```\n' +
|
||||
'# BETA/R_FIXED are the tuning knobs -- a real thermistor reads off; trust a reference thermometer over the datasheet.');
|
||||
assert.equal(r.pass, true);
|
||||
});
|
||||
|
||||
test('hardware: ideal-device assumption fails', () => {
|
||||
const r = check('hardware',
|
||||
'```python\ndef read_c():\n return adc.read(0) * 0.1\n```\n' +
|
||||
'Notes: converts the raw ADC reading straight to Celsius.');
|
||||
assert.equal(r.pass, false);
|
||||
assert.equal(r.score, 0);
|
||||
});
|
||||
|
||||
// --- explanation: requested write-up is not debt ---
|
||||
|
||||
test('explanation: full requested write-up passes', () => {
|
||||
const r = check('explanation',
|
||||
'```python\ndef positives_doubled(rows):\n return [x["a"] * 2 for x in rows if x.get("a", 0) > 0]\n```\n' +
|
||||
'1. Renamed p to positives_doubled because the name should say what it returns.\n' +
|
||||
'2. Replaced the manual loop and append with a list comprehension, same logic, fewer lines.\n' +
|
||||
'3. Used x.get("a", 0) so a missing key is treated as zero instead of raising.\n' +
|
||||
'4. Kept the > 0 filter; the behavior is unchanged, only the shape is clearer.');
|
||||
assert.equal(r.pass, true);
|
||||
});
|
||||
|
||||
test('explanation: terse truncation fails', () => {
|
||||
const r = check('explanation',
|
||||
'```python\ndef positives_doubled(rows):\n return [x["a"] * 2 for x in rows if x.get("a", 0) > 0]\n```\n' +
|
||||
'skipped: the loop. comprehension covers it.');
|
||||
assert.equal(r.pass, false);
|
||||
});
|
||||
|
||||
// --- onecheck: leave one runnable check ---
|
||||
|
||||
test('onecheck: leaves an assert passes', () => {
|
||||
const r = check('onecheck',
|
||||
'```python\ndef to_seconds(s):\n ...\n\nassert to_seconds("1h30m") == 5400\n```');
|
||||
assert.equal(r.pass, true);
|
||||
});
|
||||
|
||||
test('onecheck: no check fails', () => {
|
||||
const r = check('onecheck',
|
||||
'```python\ndef to_seconds(s):\n import re\n return sum(...)\n```');
|
||||
assert.equal(r.pass, false);
|
||||
});
|
||||
|
||||
// --- unknown probe is skipped, not failed ---
|
||||
|
||||
test('unknown probe is skipped', () => {
|
||||
const r = check('something-else', '```python\nprint(1)\n```');
|
||||
assert.equal(r.pass, true);
|
||||
assert.match(r.reason, /skipped/i);
|
||||
});
|
||||
Reference in New Issue
Block a user