Mandavilli Vijay
386f95734a
Adds benchmarks/benchmark-local.py (Ollama-based local runner), a results writeup, and a Node version note. Thanks @mandavillivijay.
69 lines
2.5 KiB
Markdown
69 lines
2.5 KiB
Markdown
# Local model benchmark: llama3.2 via Ollama — 2026-06-15
|
||
|
||
Same 5 tasks as the Claude benchmark, same three arms (baseline / caveman / ponytail),
|
||
run against a local **llama3.2:latest** (3.2B, Q4_K_M) via Ollama on a Windows 11 machine.
|
||
n=1 per cell. Tooling: `benchmarks/benchmark-local.py` (no promptfoo needed).
|
||
|
||
## Results
|
||
|
||
**Code LOC**
|
||
|
||
| arm | email | debounce | csv-sum | countdown | rate-limit | **TOTAL** |
|
||
|---|--:|--:|--:|--:|--:|--:|
|
||
| baseline | 13 | 13 | 38 | 44 | 32 | **140** |
|
||
| caveman | 16 | 12 | 5 | 45 | 28 | **106** |
|
||
| ponytail | 18 | 21 | 7 | 38 | 49 | **133** |
|
||
|
||
**Time (seconds)**
|
||
|
||
| arm | email | debounce | csv-sum | countdown | rate-limit | **TOTAL** |
|
||
|---|--:|--:|--:|--:|--:|--:|
|
||
| baseline | 52.6 | 37.2 | 63.4 | 65.2 | 71.0 | **289.4** |
|
||
| caveman | 79.6 | 54.4 | 28.3 | 71.4 | 61.0 | **294.7** |
|
||
| ponytail | 99.7 | 71.0 | 25.4 | 74.8 | 97.3 | **368.2** |
|
||
|
||
**LOC vs baseline**
|
||
|
||
| arm | total LOC | vs baseline |
|
||
|---|--:|---|
|
||
| caveman | 106 | −24% |
|
||
| ponytail | 133 | −5% |
|
||
|
||
## Key findings
|
||
|
||
**Ponytail does not transfer to llama3.2.** On 3 of 5 tasks (email: 13→18, debounce: 13→21,
|
||
rate-limit: 32→49) ponytail produced *more* code than the no-skill baseline. Total LOC
|
||
reduction was −5% vs the 80–94% seen on Claude. Response time increased by +27% (368s vs
|
||
289s) rather than the 3–6× speedup seen on Claude.
|
||
|
||
**Caveman outperformed ponytail on this model** (−24% LOC, similar time to baseline).
|
||
Caveman's rules are simpler prose instructions that a small model can follow more reliably;
|
||
ponytail's multi-step decision ladder requires a stronger instruction-follower.
|
||
|
||
**Why this happens:** Ponytail is a prompt-engineering skill calibrated on Claude models,
|
||
which are specifically trained to follow detailed system instructions. A 3.2B quantised model
|
||
partially absorbs the ponytail rules and then adds extra prose *justifying* its choices —
|
||
paying the complexity cost without getting the minimalism benefit.
|
||
|
||
## Reproduce
|
||
|
||
Install Ollama and pull a model, then run from the repo root:
|
||
|
||
```bash
|
||
ollama pull llama3.2
|
||
python benchmarks/benchmark-local.py --model llama3.2
|
||
```
|
||
|
||
Optional flags:
|
||
|
||
```
|
||
--repeat N Runs per cell; median is reported (default: 1)
|
||
--ollama-url URL Ollama base URL (default: http://localhost:11434)
|
||
```
|
||
|
||
## Takeaway
|
||
|
||
The benchmark claims in the README are accurate for the models tested (Haiku, Sonnet, Opus).
|
||
For local/small models, expect significantly smaller — or even negative — gains until
|
||
instruction-following capability reaches a threshold comparable to Claude Haiku or better.
|