DietrichGebertandClaude Opus 4.8
f3da910b4f
* feat: refine ruleset from a full-project field review
A reviewer ran ponytail across a 9-phase rewrite (protocol, PC app, simulator,
RPi daemon, ESP32 firmware) and flagged three gaps. All three land in SKILL.md
and propagate to AGENTS.md + the rule copies:
- Promote the one-runnable-check rule to a headline ("Lazy code without its
check is unfinished"), enforced as a check-rule-copies invariant.
- Hardware carve-out in "When NOT to be lazy": a real device is never the spec
ideal (clock drift, sensor offset), leave the calibration knob.
- Clarify the Output rule: explanation the user explicitly asked for is not
debt, only unrequested prose is.
Fallback instructions kept in sync. Rule-copy check + tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: add a behavior gate proving the refinements actually fire
The refinements were verified as injected text, but injected != behavioral.
This adds a behavior eval that probes each refined rule on a task that should
trigger it:
- hardware -> does the output leave a calibration knob?
- explanation -> when a write-up is explicitly requested, is it given in full?
- onecheck -> is a runnable check left behind?
benchmarks/behavior.yaml runs the probes (baseline vs ponytail arm); the
grader benchmarks/behavior.js is proven by tests/behavior.test.js (8 cases,
RED/GREEN, no API key, runs in CI). Live-confirmed: the model under the
current ruleset passes all three gates, graded by the same grader.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
41 lines
1.6 KiB
YAML
41 lines
1.6 KiB
YAML
# Ponytail behavior gates: does the ruleset actually produce its refined
|
|
# behaviors (not just carry the text)? Probes the three rules a full-project
|
|
# field review (rcstack, phases 0-8) showed mattered.
|
|
#
|
|
# Run: npx promptfoo@latest eval -c benchmarks/behavior.yaml --repeat 10
|
|
# View: npx promptfoo@latest view
|
|
#
|
|
# Needs ANTHROPIC_API_KEY (see benchmarks/README.md). The grader (behavior.js)
|
|
# is proven separately by tests/behavior.test.js, which needs no API key.
|
|
#
|
|
# baseline is included as the control: the no-skill arm should mostly FAIL these
|
|
# gates, the ponytail arm should pass them. That delta is the point.
|
|
description: "Ponytail behavior gates: hardware calibration, requested explanation, one runnable check."
|
|
|
|
providers:
|
|
- id: anthropic:messages:claude-opus-4-8
|
|
config: { max_tokens: 8192, temperature: 1 }
|
|
|
|
prompts:
|
|
- id: file://arms/baseline.js
|
|
label: baseline (no skill)
|
|
- id: file://arms/ponytail.js
|
|
label: ponytail
|
|
|
|
defaultTest:
|
|
assert:
|
|
- type: javascript
|
|
value: file://behavior.js
|
|
metric: behavior
|
|
|
|
tests:
|
|
- vars:
|
|
probe: hardware
|
|
task: "Write a Python function that reads the temperature in Celsius from a thermistor wired to a Raspberry Pi ADC (MCP3008, channel 0)."
|
|
- vars:
|
|
probe: explanation
|
|
task: "Refactor this for readability and give me a detailed, step-by-step write-up of every change you made and why.\n\ndef p(d):\n r = []\n for x in d:\n if x.get('a') and x['a'] > 0:\n r.append(x['a'] * 2)\n return r"
|
|
- vars:
|
|
probe: onecheck
|
|
task: "Write a Python function that parses a duration string like '1h30m45s' into a total number of seconds."
|