ponytail/benchmarks/results/2026-06-15-llama3.2-local.md

Local model benchmark: llama3.2 via Ollama — 2026-06-15

Same 5 tasks as the Claude benchmark, same three arms (baseline / caveman / ponytail), run against a local llama3.2:latest (3.2B, Q4_K_M) via Ollama on a Windows 11 machine. n=1 per cell. Tooling: benchmarks/benchmark-local.py (no promptfoo needed).

Results

Code LOC

arm	email	debounce	csv-sum	countdown	rate-limit	TOTAL
baseline	13	13	38	44	32	140
caveman	16	12	5	45	28	106
ponytail	18	21	7	38	49	133

Time (seconds)

arm	email	debounce	csv-sum	countdown	rate-limit	TOTAL
baseline	52.6	37.2	63.4	65.2	71.0	289.4
caveman	79.6	54.4	28.3	71.4	61.0	294.7
ponytail	99.7	71.0	25.4	74.8	97.3	368.2

LOC vs baseline

arm	total LOC	vs baseline
caveman	106	−24%
ponytail	133	−5%

Key findings

Ponytail does not transfer to llama3.2. On 3 of 5 tasks (email: 13→18, debounce: 13→21, rate-limit: 32→49) ponytail produced more code than the no-skill baseline. Total LOC reduction was −5% vs the 80–94% seen on Claude. Response time increased by +27% (368s vs 289s) rather than the 3–6× speedup seen on Claude.

Caveman outperformed ponytail on this model (−24% LOC, similar time to baseline). Caveman's rules are simpler prose instructions that a small model can follow more reliably; ponytail's multi-step decision ladder requires a stronger instruction-follower.

Why this happens: Ponytail is a prompt-engineering skill calibrated on Claude models, which are specifically trained to follow detailed system instructions. A 3.2B quantised model partially absorbs the ponytail rules and then adds extra prose justifying its choices — paying the complexity cost without getting the minimalism benefit.

Reproduce

Install Ollama and pull a model, then run from the repo root:

ollama pull llama3.2
python benchmarks/benchmark-local.py --model llama3.2

Optional flags:

--repeat N       Runs per cell; median is reported (default: 1)
--ollama-url URL Ollama base URL (default: http://localhost:11434)

Takeaway

The benchmark claims in the README are accurate for the models tested (Haiku, Sonnet, Opus). For local/small models, expect significantly smaller — or even negative — gains until instruction-following capability reaches a threshold comparable to Claude Haiku or better.