DietrichGebertandClaude Opus 4.8
2e6a93765a
Fixes the local benchmark LOC counter (counted only fenced code, scored bare output 0), makes summary output ASCII-safe (a Unicode arrow crashed the script on Windows cp1252), gitignores generated artifacts, and refreshes the llama3.2 writeup with n=5 data showing the LOC effect is within the noise floor. Follow-up to #63. Verified live. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
77 lines
3.1 KiB
Markdown
77 lines
3.1 KiB
Markdown
# Local model benchmark: llama3.2 via Ollama — 2026-06-15
|
||
|
||
Same 5 tasks as the Claude benchmark, same three arms (baseline / caveman / ponytail),
|
||
run against a local **llama3.2:latest** (3.2B, Q4_K_M) via Ollama on a Windows 11 machine.
|
||
Tooling: `benchmarks/benchmark-local.py` (no promptfoo needed).
|
||
|
||
> **Updated 2026-06-15:** the LOC counter now counts bare, unfenced code. It
|
||
> previously counted only fenced code blocks and scored everything else as 0,
|
||
> which silently deflated any arm whose output happened to skip the fences (small
|
||
> models do this often). Numbers below use the corrected counter at n=5 median.
|
||
> Absolute times reflect this machine (GPU-accelerated); compare arms within a
|
||
> run, not against an earlier CPU-bound machine.
|
||
|
||
## Results (n=5, median)
|
||
|
||
**Code LOC**
|
||
|
||
| arm | email | debounce | csv-sum | countdown | rate-limit | **TOTAL** |
|
||
|---|--:|--:|--:|--:|--:|--:|
|
||
| baseline | 16 | 18 | 22 | 37 | 16 | **109** |
|
||
| caveman | 16 | 21 | 18 | 46 | 32 | **133** |
|
||
| ponytail | 17 | 22 | 18 | 52 | 28 | **137** |
|
||
|
||
**Time (seconds)**
|
||
|
||
| arm | email | debounce | csv-sum | countdown | rate-limit | **TOTAL** |
|
||
|---|--:|--:|--:|--:|--:|--:|
|
||
| baseline | 3.1 | 3.7 | 3.6 | 4.2 | 4.8 | **19.4** |
|
||
| caveman | 4.1 | 4.2 | 3.6 | 4.4 | 4.8 | **21.1** |
|
||
| ponytail | 4.1 | 4.2 | 3.8 | 4.8 | 4.9 | **21.8** |
|
||
|
||
## Key findings
|
||
|
||
**On llama3.2 the LOC effect is inside the noise floor.** At temperature 0.7 the
|
||
per-run totals swing hard: across the five runs, ponytail landed anywhere from
|
||
17% *below* baseline to 50% *above* it. The n=5 median came out +26%; a separate
|
||
n=3 median came out −17%. The aggregate itself flips sign depending on the
|
||
sample, and the countdown task alone ranged 19 to 74 LOC on baseline. There is no
|
||
stable LOC reduction to report.
|
||
|
||
**Ponytail does not transfer to llama3.2.** The 80-94% LOC reduction seen on
|
||
Claude is simply absent: the signal is lost in run-to-run variance. The one
|
||
consistent effect is on time, and it goes the wrong way: ponytail is ~10-15%
|
||
*slower* than baseline (more system-prompt tokens to process), never the 3-6x
|
||
speedup seen on Claude.
|
||
|
||
**Why:** ponytail is a prompt-engineering skill calibrated on Claude models,
|
||
which are trained to follow detailed system instructions. A 3.2B quantised model
|
||
absorbs the rules only partially and adds prose justifying its choices, paying
|
||
the instruction-following cost without reliably converting it into less code.
|
||
|
||
## Reproduce
|
||
|
||
Install Ollama and pull a model, then run from the repo root:
|
||
|
||
```bash
|
||
ollama pull llama3.2
|
||
python benchmarks/benchmark-local.py --model llama3.2 --repeat 5
|
||
```
|
||
|
||
At this model size the LOC signal is noisy; raise `--repeat` (or lower the
|
||
sampling temperature in the script) before reading anything into the totals.
|
||
|
||
Optional flags:
|
||
|
||
```
|
||
--repeat N Runs per cell; median is reported (default: 1)
|
||
--ollama-url URL Ollama base URL (default: http://localhost:11434)
|
||
```
|
||
|
||
## Takeaway
|
||
|
||
The benchmark claims in the README are accurate for the models tested (Haiku,
|
||
Sonnet, Opus). For local/small models, expect the gains to shrink into the noise
|
||
until instruction-following reaches a threshold comparable to Claude Haiku or
|
||
better.
|