benchmarks: add local model support and Node version note (#63)

Adds benchmarks/benchmark-local.py (Ollama-based local runner), a results writeup, and a Node version note. Thanks @mandavillivijay.
2026-06-15 15:27:11 +02:00
parent 60a75f8159
commit 386f95734a
3 changed files with 241 additions and 0 deletions
@@ -4,12 +4,30 @@ Three arms (no skill, [caveman](https://github.com/JuliusBrussee/caveman), ponyt

 ## Reproduce

+### Claude (Haiku / Sonnet / Opus)
+
+Requires an Anthropic API key and **Node.js ≥ 22.22.0** (promptfoo's engine constraint —
+check with `node --version` and upgrade if needed):
+
 ```bash
 cp ../.env.example ../.env      # add your ANTHROPIC_API_KEY
 npx promptfoo@latest eval -c promptfooconfig.yaml --repeat 10
 npx promptfoo@latest view
 ```

+### Local models via Ollama
+
+No API key or promptfoo required. Runs against any model served by Ollama:
+
+```bash
+ollama pull llama3.2          # or any other model
+python benchmarks/benchmark-local.py --model llama3.2 --repeat 3
+```
+
+See `benchmarks/results/2026-06-15-llama3.2-local.md` for what to expect: the skill works
+well on instruction-following models (Claude-class) but transfers poorly to small local
+models where the multi-step decision ladder isn't reliably followed.
+
 Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limit (see `promptfooconfig.yaml`). Single-shot completions, default temperature.

 ## Median results (10 runs, 2026-06-13)