ponytail/benchmarks/results/2026-06-15-llama3.2-local.md

Local model benchmark: llama3.2 via Ollama — 2026-06-15

Same 5 tasks as the Claude benchmark, same three arms (baseline / caveman / ponytail), run against a local llama3.2:latest (3.2B, Q4_K_M) via Ollama on a Windows 11 machine. Tooling: benchmarks/benchmark-local.py (no promptfoo needed).

Updated 2026-06-15: the LOC counter now counts bare, unfenced code. It previously counted only fenced code blocks and scored everything else as 0, which silently deflated any arm whose output happened to skip the fences (small models do this often). Numbers below use the corrected counter at n=5 median. Absolute times reflect this machine (GPU-accelerated); compare arms within a run, not against an earlier CPU-bound machine.

Results (n=5, median)

Code LOC

arm	email	debounce	csv-sum	countdown	rate-limit	TOTAL
baseline	16	18	22	37	16	109
caveman	16	21	18	46	32	133
ponytail	17	22	18	52	28	137

Time (seconds)

arm	email	debounce	csv-sum	countdown	rate-limit	TOTAL
baseline	3.1	3.7	3.6	4.2	4.8	19.4
caveman	4.1	4.2	3.6	4.4	4.8	21.1
ponytail	4.1	4.2	3.8	4.8	4.9	21.8

Key findings

On llama3.2 the LOC effect is inside the noise floor. At temperature 0.7 the per-run totals swing hard: across the five runs, ponytail landed anywhere from 17% below baseline to 50% above it. The n=5 median came out +26%; a separate n=3 median came out −17%. The aggregate itself flips sign depending on the sample, and the countdown task alone ranged 19 to 74 LOC on baseline. There is no stable LOC reduction to report.

Ponytail does not transfer to llama3.2. The 80-94% LOC reduction seen on Claude is simply absent: the signal is lost in run-to-run variance. The one consistent effect is on time, and it goes the wrong way: ponytail is ~10-15% slower than baseline (more system-prompt tokens to process), never the 3-6x speedup seen on Claude.

Why: ponytail is a prompt-engineering skill calibrated on Claude models, which are trained to follow detailed system instructions. A 3.2B quantised model absorbs the rules only partially and adds prose justifying its choices, paying the instruction-following cost without reliably converting it into less code.

Reproduce

Install Ollama and pull a model, then run from the repo root:

ollama pull llama3.2
python benchmarks/benchmark-local.py --model llama3.2 --repeat 5

At this model size the LOC signal is noisy; raise --repeat (or lower the sampling temperature in the script) before reading anything into the totals.

Optional flags:

--repeat N       Runs per cell; median is reported (default: 1)
--ollama-url URL Ollama base URL (default: http://localhost:11434)

Takeaway

The benchmark claims in the README are accurate for the models tested (Haiku, Sonnet, Opus). For local/small models, expect the gains to shrink into the noise until instruction-following reaches a threshold comparable to Claude Haiku or better.