Files

T

EmerikoandClaude Opus 4.8 08440027f2 fix(examples): replace hand-written examples with real benchmark output

The examples/ before/after blocks were authored by hand, not produced by a
model. Issue #127 correctly noted that nobody hand-rolls quicksort for "sort
this array" - every model just calls .sort(). Regenerate all examples verbatim
from a real benchmark run (Claude Haiku 4.5, no-skill arm vs ponytail arm,
benchmarks/output.json) so the before/after is reproducible, not authored:

  email 75->3, debounce 116->10, csv 20->3, countdown 267->9, rate-limit 128->10 LOC

- Delete sorting.md (pure strawman) plus the other hand-written caricatures
  (api-endpoint, caching, date-picker)
- Add benchmarks/generate-examples.mjs to regenerate examples from any run
- examples/README.md indexes the set and documents how to reproduce

Closes #127

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-17 04:31:52 +02:00

1.7 KiB

Raw Blame History

CSV Sum

Task: "Write Python code that reads sales.csv and sums the 'amount' column."

Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source benchmarks/output.json. Reproduce: npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml.

Without Ponytail — 20 lines of code

Python code to read sales.csv and sum the 'amount' column

import pandas as pd

# Read the CSV file
df = pd.read_csv('sales.csv')

# Sum the 'amount' column
total_amount = df['amount'].sum()

# Print the result
print(f"Total amount: ${total_amount:,.2f}")

Alternative methods:

Using CSV module (without pandas):

import csv

total_amount = 0

with open('sales.csv', 'r') as file:
    reader = csv.DictReader(file)
    for row in reader:
        total_amount += float(row['amount'])

print(f"Total amount: ${total_amount:,.2f}")

Using pandas with error handling:

import pandas as pd

try:
    df = pd.read_csv('sales.csv')
    total_amount = df['amount'].sum()
    print(f"Total amount: ${total_amount:,.2f}")
except FileNotFoundError:
    print("Error: sales.csv not found")
except KeyError:
    print("Error: 'amount' column not found")

The pandas method is recommended because it's:

More concise and readable
Handles data types automatically
Provides built-in error handling
Efficient for large files

With Ponytail — 3 lines of code

import csv

total = sum(float(row['amount']) for row in csv.DictReader(open('sales.csv')))
print(total)

Skipped: pandas, error handling, file closing — add when the CSV is large, malformed, or you need more analysis.

20 → 3 lines of code — same model, same prompt.

1.7 KiB Raw Blame History