ponytail/benchmarks/results/2026-06-16-correctness-gate-fix.md

Correctness under Ponytail: gate fixes + GPT-mini reproduction (2026-06-16)

Context: issue #65 asked whether Ponytail degrades model performance. A community run (Pyseph) reported a large correctness drop on gpt-4.1-mini (10/15 with Ponytail vs 15/15 without) and a small one on gpt-5.4-mini (14/15 vs 15/15).

Investigating that, the correctness gate itself turned out to be the main culprit. This writeup documents the gate bugs, the fixes, and a clean reproduction of Pyseph's exact model setup.

TL;DR

• The correct gate had two bugs that under-reported correctness for terse models — it could not read unfenced code, and the debounce task tested for a deliverable the prompt never asked for.
• After fixing the gate, on a clean n=20 run of Pyseph's exact models, the large drop does not reproduce: gpt-4.1-mini is 100% with and without Ponytail.
• Ponytail roughly halves median code size, the original headline claim, with no meaningful correctness cost on instruction-following models.
• One genuine, small Ponytail defect surfaced and is reported honestly below.

The gate bugs

1. Unfenced code was scored as "no code blocks." extractBlocks() only matched ```fenced``` blocks. Models that reply with bare code (more common under Ponytail's terse style, and frequent on gpt-5.4-mini) scored an automatic fail even when the code was correct. This alone accounted for 41 of 74 failures in the first GPT run.
2. The debounce task tested the wrong deliverable. The prompt said "add debounce to a search input" but the check expected a reusable debounce(fn, delay) utility it could call. A correct inline answer (input.addEventListener(... clearTimeout ...)) failed with searchInput is not defined. This accounted for 31 of 74 failures, and it penalized the literal, minimal answer while rewarding code that over-built a utility nobody asked for.

Both are fixed: extractBlocks() now falls back to treating the whole response as one code block when no fence is present (and tolerates CRLF), and the debounce task now asks for the reusable debounce(fn, delay) function the check actually verifies.

Method

Two arms (baseline = no skill, ponytail), Pyseph's two models, the five repo tasks, n=20 per cell, run serially (--max-concurrency 1) so transient quota 429s never reduced the denominators. Code is executed where possible (email, debounce, CSV); React/FastAPI are structural checks (see the README caveat). Claude numbers are a free re-score of the committed output-10x.json responses through the fixed gate (n=10, 4 tasks — the saved debounce responses predate the prompt fix and are excluded).

Results

GPT-mini (clean n=20, 0 errors, full denominators)

model	baseline	ponytail	median LOC (base → pony)
gpt-4.1-mini	100/100	100/100	15 → 7
gpt-5.4-mini	100/100	98/100	16 → 7

Pyseph's reported gpt-4.1-mini drop (10/15 ≈ 67%) does not reproduce — it scores 100% here. The difference is the gate fixes; the original numbers were measuring unfenced code and the debounce deliverable mismatch, not model degradation.

Claude (fixed gate, re-score of committed responses, n=10, 4 tasks)

model	baseline	ponytail
claude-haiku-4-5	38/40 (95%)	40/40 (100%)
claude-opus-4-8	40/40 (100%)	40/40 (100%)
claude-sonnet-4-6	28/40 (70%)	40/40 (100%)

On instruction-following models Ponytail ties or slightly beats baseline. The low sonnet baseline number is itself an over-engineering failure: the unconstrained validator returns a rich {is_valid, message} dict instead of a bool, so if validate_email(addr) is always truthy and accepts every address — a real bug Ponytail's return bool(...) avoids.

The one real Ponytail defect

On gpt-5.4-mini, 2 of 20 Ponytail email runs failed because the model reached for the laziest stdlib option:

from email.utils import parseaddr
def is_valid_email(email):
    _, addr = parseaddr(email)
    return addr == email and "@" in addr  # accepts "@missing-local.com"

parseaddr does not require a local part, so "@missing-local.com" is accepted. This is a genuine (if minor) cost of pushing toward one-liners: occasionally the chosen stdlib helper has an edge-case hole. The other 18 runs used a regex and passed.

Reproduce

# GPT arms (needs OPENAI_API_KEY in ../.env)
cd benchmarks
npx promptfoo@latest eval -c promptfooconfig.gpt.yaml --env-file ../.env --repeat 20 --max-concurrency 1

# Claude re-score of committed responses through the fixed gate
node -e 'const c=require("./correctness.js"),d=require("./output-10x.json");/* score d.results.results through c */'

Takeaway

The "Ponytail hurts correctness" reports trace to a benchmark that could not read terse output, not to the skill. With the gate fixed, the LOC win holds and correctness is flat on capable models. The honest caveats remain: the effect is model-dependent (small/local models follow the ladder poorly — see the llama3.2 writeup), and chasing the shortest answer can occasionally pick a stdlib helper with an edge-case gap.