feat: refine ruleset from a full-project field review (#39)
* feat: refine ruleset from a full-project field review
A reviewer ran ponytail across a 9-phase rewrite (protocol, PC app, simulator,
RPi daemon, ESP32 firmware) and flagged three gaps. All three land in SKILL.md
and propagate to AGENTS.md + the rule copies:
- Promote the one-runnable-check rule to a headline ("Lazy code without its
check is unfinished"), enforced as a check-rule-copies invariant.
- Hardware carve-out in "When NOT to be lazy": a real device is never the spec
ideal (clock drift, sensor offset), leave the calibration knob.
- Clarify the Output rule: explanation the user explicitly asked for is not
debt, only unrequested prose is.
Fallback instructions kept in sync. Rule-copy check + tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add a behavior gate proving the refinements actually fire
The refinements were verified as injected text, but injected != behavioral.
This adds a behavior eval that probes each refined rule on a task that should
trigger it:
- hardware -> does the output leave a calibration knob?
- explanation -> when a write-up is explicitly requested, is it given in full?
- onecheck -> is a runnable check left behind?
benchmarks/behavior.yaml runs the probes (baseline vs ponytail arm); the
grader benchmarks/behavior.js is proven by tests/behavior.test.js (8 cases,
RED/GREEN, no API key, runs in CI). Live-confirmed: the model under the
current ruleset passes all three gates, graded by the same grader.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b545f1536a
commit
f3da910b4f
@@ -0,0 +1,58 @@
|
||||
// Behavior gate: does the ponytail ruleset actually PRODUCE its refined
|
||||
// behaviors, not just carry the text? One check per probe (vars.probe), each
|
||||
// targeting a rule that a field review (rcstack, phases 0-8) showed mattered:
|
||||
// hardware - "hardware is never the spec ideal, leave the calibration knob"
|
||||
// explanation - "explanation the user explicitly asked for is not debt"
|
||||
// onecheck - "lazy code without its check is unfinished"
|
||||
//
|
||||
// Heuristic graders, same spirit as loc.js / correctness.js. The graders
|
||||
// themselves are proven by tests/behavior.test.js (RED/GREEN, no API key).
|
||||
//
|
||||
// Metric: `behavior` (1 = behavior present, 0 = absent).
|
||||
|
||||
function codeOf(text) {
|
||||
return [...String(text || '').matchAll(/```[\w-]*\n([\s\S]*?)```/g)].map((m) => m[1]).join('\n');
|
||||
}
|
||||
|
||||
function proseOf(text) {
|
||||
return String(text || '').replace(/```[\s\S]*?```/g, ' ').replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
const CHECKS = {
|
||||
// Treats the device as non-ideal: leaves a tunable knob or flags per-unit drift.
|
||||
// A passing mention of "calibration" is not enough; it must be actionable.
|
||||
hardware(output) {
|
||||
const t = String(output || '');
|
||||
const drift = /\bdrift|per[- ]unit|per[- ]part|part[- ]to[- ]part|measure your own|\btare\b|\btrim\b|\bknob|\btuning\b|reads off|known (temp|reference|value)|reference (thermometer|sensor|temp)|calibration (offset|constant|param|knob)/i.test(t);
|
||||
return drift
|
||||
? { pass: true, reason: 'Leaves a calibration knob / flags per-unit drift.' }
|
||||
: { pass: false, reason: 'Treats the hardware as ideal; no calibration knob.' };
|
||||
},
|
||||
|
||||
// Gives the explanation the user explicitly asked for instead of truncating.
|
||||
explanation(output) {
|
||||
const p = proseOf(output);
|
||||
const words = p ? p.split(' ').length : 0;
|
||||
const structured = /(\d+[.)]\s|[-*]\s)/.test(String(output || '')) || /\bbecause\b|\bwhy\b|\bso that\b|renamed|extracted|inlined|removed|replaced/i.test(p);
|
||||
return words >= 45 && structured
|
||||
? { pass: true, reason: `Gave the requested write-up (${words} words of prose).` }
|
||||
: { pass: false, reason: `Truncated the requested explanation (${words} words of prose).` };
|
||||
},
|
||||
|
||||
// Leaves ONE runnable check behind for non-trivial logic.
|
||||
onecheck(output) {
|
||||
const t = String(output || '');
|
||||
const hasCheck = /\bassert\b|def\s+test_|if\s+__name__|unittest|pytest|console\.assert|\bexpect\(|\bdescribe\(|\bit\(/.test(t);
|
||||
return hasCheck
|
||||
? { pass: true, reason: 'Left a runnable check (assert/test/demo).' }
|
||||
: { pass: false, reason: 'No runnable check left behind.' };
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = (output, context) => {
|
||||
const probe = context && context.vars && context.vars.probe;
|
||||
const check = CHECKS[probe];
|
||||
if (!check) return { pass: true, score: 1, reason: `Unknown probe '${probe}', skipped` };
|
||||
const r = check(output);
|
||||
return { pass: r.pass, score: r.pass ? 1 : 0, reason: r.reason };
|
||||
};
|
||||
Reference in New Issue
Block a user