EmerikoandClaude Opus 4.8
0bf152a987
Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus): ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%. The direction holds, both ends came in a few points lower. Updates the README headline and body, the benchmark chart subtitle, and the benchmarks/README cost table, and adds a dated results doc with full method. Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On OpenAI reasoning models ponytail costs more, not less, so the claim stays Claude-scoped. Gemini run pending a fresh-quota day. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
33 lines
1.4 KiB
YAML
33 lines
1.4 KiB
YAML
# Ponytail vs baseline, latest Gemini: gemini-3.5-flash (mini) + gemini-3.1-pro-preview (top).
|
|
# npx promptfoo@latest eval -c benchmarks/promptfooconfig.gemini.yaml --env-file .env --repeat 30
|
|
# Needs GOOGLE_API_KEY in .env (AI Studio).
|
|
description: "Ponytail vs baseline, latest Gemini (3.5-flash, 3.1-pro). LOC + correctness, cost telemetry."
|
|
|
|
providers:
|
|
- id: google:gemini-3.5-flash
|
|
config: { temperature: 1, maxOutputTokens: 8192 }
|
|
- id: google:gemini-3.1-pro-preview
|
|
config: { maxOutputTokens: 8192 }
|
|
|
|
prompts:
|
|
- id: file://arms/baseline.js
|
|
label: baseline (no skill)
|
|
- id: file://arms/ponytail.js
|
|
label: ponytail
|
|
|
|
defaultTest:
|
|
assert:
|
|
- type: javascript
|
|
value: file://loc.js
|
|
metric: code_loc
|
|
- type: javascript
|
|
value: file://correctness.js
|
|
metric: correct
|
|
|
|
tests:
|
|
- vars: { task: "Write me a Python function that validates email addresses." }
|
|
- vars: { task: "Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call." }
|
|
- vars: { task: "Write Python code that reads sales.csv and sums the 'amount' column." }
|
|
- vars: { task: "Build me a countdown timer component in React that counts down from a given number of seconds." }
|
|
- vars: { task: "Add rate limiting to my FastAPI endpoint so users can't spam it." }
|