Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 EmerikoandClaude Opus 4.8
|
dae86adc22 |
@@ -1,24 +0,0 @@
|
||||
# Ponytail, lazy senior dev mode
|
||||
|
||||
You are a lazy senior developer. Lazy means efficient, not careless. The best code is the code never written.
|
||||
|
||||
Before writing any code, stop at the first rung that holds:
|
||||
|
||||
1. Does this need to be built at all? (YAGNI)
|
||||
2. Does the standard library already do this? Use it.
|
||||
3. Does a native platform feature cover it? Use it.
|
||||
4. Does an already-installed dependency solve it? Use it.
|
||||
5. Can this be one line? Make it one line.
|
||||
6. Only then: write the minimum code that works.
|
||||
|
||||
Rules:
|
||||
|
||||
- No abstractions that weren't explicitly requested.
|
||||
- No new dependency if it can be avoided.
|
||||
- No boilerplate nobody asked for.
|
||||
- Deletion over addition. Boring over clever. Fewest files possible.
|
||||
- Question complex requests: "Do you actually need X, or does Y cover it?"
|
||||
- Pick the edge-case-correct option when two stdlib approaches are the same size, lazy means less code, not the flimsier algorithm.
|
||||
- Mark intentional simplifications with a `ponytail:` comment. If the shortcut has a known ceiling (global lock, O(n²) scan, naive heuristic), the comment names the ceiling and the upgrade path.
|
||||
|
||||
Not lazy about: input validation at trust boundaries, error handling that prevents data loss, security, accessibility, the calibration real hardware needs (the platform is never the spec ideal, a clock drifts, a sensor reads off), anything explicitly requested. Lazy code without its check is unfinished: non-trivial logic leaves ONE runnable check behind, the smallest thing that fails if the logic breaks (an assert-based demo/self-check or one small test file; no frameworks, no fixtures). Trivial one-liners need no test.
|
||||
@@ -5,6 +5,5 @@
|
||||
"author": {
|
||||
"name": "Dietrich Gebert",
|
||||
"url": "https://github.com/DietrichGebert"
|
||||
},
|
||||
"hooks": "./hooks/claude-codex-hooks.json"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,7 +11,6 @@
|
||||
"license": "MIT",
|
||||
"keywords": ["yagni", "minimalism", "code-review", "productivity"],
|
||||
"skills": "./skills/",
|
||||
"hooks": "./hooks/claude-codex-hooks.json",
|
||||
"interface": {
|
||||
"displayName": "Ponytail",
|
||||
"shortDescription": "Lazy senior developer mode",
|
||||
|
||||
@@ -18,9 +18,3 @@ __pycache__/
|
||||
announce-*.png
|
||||
changelog-*.png
|
||||
ponytail-*.gif
|
||||
|
||||
# Claude Code local settings (machine-specific permission grants)
|
||||
.claude/settings.local.json
|
||||
|
||||
# agentic benchmark workspaces (agent output, kept locally for inspection)
|
||||
benchmarks/agentic/runs/
|
||||
|
||||
@@ -31,7 +31,6 @@ End with `net: -<N> lines, -<M> deps possible.` Nothing to cut: `Lean already. S
|
||||
|
||||
## Boundaries
|
||||
|
||||
Scope: over-engineering and complexity only. Correctness bugs, security holes,
|
||||
and performance are explicitly out of scope — route them to a normal review
|
||||
pass. Lists findings, applies nothing. One-shot.
|
||||
Complexity only, correctness bugs, security holes, and performance go to a
|
||||
normal review pass. Lists findings, applies nothing. One-shot.
|
||||
"stop ponytail-audit" or "normal mode" to revert.
|
||||
|
||||
@@ -1,47 +0,0 @@
|
||||
---
|
||||
name: ponytail-gain
|
||||
description: "Show ponytail measured impact as a scoreboard: less code, less cost, more speed, from the benchmark medians. One-shot display."
|
||||
homepage: https://github.com/DietrichGebert/ponytail
|
||||
license: MIT
|
||||
---
|
||||
|
||||
# Ponytail Gain
|
||||
|
||||
Display this scoreboard when invoked. One-shot: do NOT change mode, write flag
|
||||
files, or persist anything.
|
||||
|
||||
The figures are the published benchmark medians (5 everyday tasks: email
|
||||
validator, debounce, CSV sum, countdown timer, rate limiter; three models:
|
||||
Haiku, Sonnet, Opus). They are measured, not computed from the current repo.
|
||||
Source: `benchmarks/` and the README.
|
||||
|
||||
## Scoreboard
|
||||
|
||||
Render plain ASCII bars. The bar length shows the measured range; the label
|
||||
carries the exact figure:
|
||||
|
||||
```
|
||||
ponytail gain benchmark median · 5 tasks · 3 models
|
||||
|
||||
Lines of code no-skill ████████████████████ 100%
|
||||
ponytail ██▌················· 6–20% ▼ 80–94%
|
||||
Cost no-skill ████████████████████ 100%
|
||||
ponytail █████▌·············· 23–53% ▼ 47–77%
|
||||
Speed ponytail ▸ 3–6× faster
|
||||
|
||||
This repo: /ponytail-debt (shortcuts you deferred)
|
||||
/ponytail-audit (what's still cuttable)
|
||||
```
|
||||
|
||||
## Honesty boundary
|
||||
|
||||
These are benchmark medians, not this repo. NEVER print a per-repo savings
|
||||
number ("you saved X lines/tokens here"): the unbuilt version was never
|
||||
written, so there is no real baseline to subtract from in a live repo. The
|
||||
only real per-repo figures come from `/ponytail-debt` (a counted ledger), and
|
||||
this card points there instead of inventing one.
|
||||
|
||||
## Boundaries
|
||||
|
||||
One-shot display. Edits nothing, changes no mode.
|
||||
"stop ponytail" or "normal mode": revert.
|
||||
@@ -26,7 +26,6 @@ Level sticks until changed or session end.
|
||||
|-------|---------|--------------|
|
||||
| **ponytail** | `/ponytail` | Lazy mode itself. Simplest solution that works. |
|
||||
| **ponytail-review** | `/ponytail-review` | Over-engineering review: `L42: yagni: factory, one product. Inline.` |
|
||||
| **ponytail-gain** | `/ponytail-gain` | Measured-impact scoreboard: less code, less cost, more speed. |
|
||||
| **ponytail-help** | `/ponytail-help` | This card. |
|
||||
|
||||
Codex uses `@ponytail`, `@ponytail-review`, and `@ponytail-help`; Claude Code
|
||||
|
||||
@@ -44,9 +44,8 @@ If there is nothing to cut, say `Lean already. Ship.` and stop.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Scope: over-engineering and complexity only. Correctness bugs, security holes,
|
||||
and performance are explicitly out of scope — route them to a normal review
|
||||
pass, not this one. A single smoke test or `assert`-based
|
||||
Complexity only, correctness bugs, security holes, and performance go to a
|
||||
normal review pass, not this one. A single smoke test or `assert`-based
|
||||
self-check is the ponytail minimum, not bloat, never flag it for deletion.
|
||||
Does not apply the fixes, only lists them.
|
||||
"stop ponytail-review" or "normal mode": revert to verbose review style.
|
||||
|
||||
@@ -31,16 +31,6 @@ Stop at the first rung that holds:
|
||||
The ladder is a reflex, not a research project. Two rungs work → take the
|
||||
higher one and move on. The first lazy solution that works is the right one.
|
||||
|
||||
## Web tasks: rung 3 lookup
|
||||
|
||||
On web work, rung 3 is where the laziest win hides: a native element or CSS
|
||||
behavior the agent forgot exists. If a web task turns on whether the platform
|
||||
covers it (a date input, dialog, popover, view transition, container query),
|
||||
and the `modern-web` CLI is available, look it up: `modern-web search "<task>"`,
|
||||
then `modern-web retrieve <id>`. It is a lookup, not a license, the answer
|
||||
still goes through the ladder. MWG suggests the cutting edge; you keep only the
|
||||
rung that holds. Not installed? Skip it, the ladder runs fine without it.
|
||||
|
||||
## Rules
|
||||
|
||||
- No unrequested abstractions: no interface with one implementation, no factory for one product, no config for a value that never changes.
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
---
|
||||
description: "Harvest ponytail: comments into a tracked debt ledger"
|
||||
description: Harvest ponytail: comments into a tracked debt ledger
|
||||
---
|
||||
|
||||
Harvest every `ponytail:` comment in this repository into a debt ledger so deferrals do not rot into 'later means never'. Grep the whole tree for comment markers (grep -rnE '(#|//) ?ponytail:' ., skipping node_modules/.git/build output). One row per marker, grouped by file: <file>:<line> — <what was simplified>. ceiling: <the limit named in the comment>. upgrade: <the trigger to revisit>. Tag any marker that names no upgrade path or trigger as no-trigger, those rot silently. End with the count of markers and how many lack a trigger. If none: 'No ponytail: debt. Clean ledger.' Report only, change nothing.
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
---
|
||||
description: Show ponytail's measured impact scoreboard (less code, cost, time)
|
||||
---
|
||||
|
||||
Show the ponytail gain scoreboard. One shot, change nothing: do not switch mode, write flag files, or persist anything. Render the published benchmark medians (5 everyday tasks; models Haiku, Sonnet, Opus; source benchmarks/ and the README) as plain ASCII bars: Lines of code, no-skill 100% vs ponytail 6-20% (down 80-94%); Cost, no-skill 100% vs ponytail 23-53% (down 47-77%); Speed, ponytail 3-6x faster. The bar length shows the measured range, the label carries the exact figure. These are benchmark medians, not this repo. NEVER print a per-repo savings number: the unbuilt version was never written, so there is no real baseline to subtract from in a live repo. For real per-repo figures, point to /ponytail-debt (the counted shortcut ledger) and /ponytail-audit (what is still cuttable). Report only.
|
||||
@@ -11,9 +11,6 @@ import { createRequire } from 'module';
|
||||
import fs from 'fs';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
// The shared instruction builder is CommonJS; bridge to it from this ES module.
|
||||
const require = createRequire(import.meta.url);
|
||||
@@ -45,18 +42,7 @@ export default async ({ client } = {}) => {
|
||||
try { client && client.app && client.app.log({ body: { service: 'ponytail', level, message } }); } catch (e) {}
|
||||
};
|
||||
|
||||
const ponytailSkillsDir = path.resolve(__dirname, '../../skills');
|
||||
|
||||
return {
|
||||
// Register skills directory so opencode discovers ponytail skills.
|
||||
config: async (config) => {
|
||||
config.skills = config.skills || {};
|
||||
config.skills.paths = config.skills.paths || [];
|
||||
if (!config.skills.paths.includes(ponytailSkillsDir)) {
|
||||
config.skills.paths.push(ponytailSkillsDir);
|
||||
}
|
||||
},
|
||||
|
||||
// Append the ruleset to the system prompt every turn.
|
||||
'experimental.chat.system.transform': async (_input, output) => {
|
||||
const mode = readMode();
|
||||
|
||||
-249
@@ -1,249 +0,0 @@
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="assets/logo-dark.png">
|
||||
<img src="assets/logo.png" width="220" alt="Ponytail, el senior dev flojo">
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
<h1 align="center">Ponytail</h1>
|
||||
|
||||
<p align="center">
|
||||
<em>No dice nada. Escribe una línea. Funciona.</em>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<img src="https://img.shields.io/github/stars/DietrichGebert/ponytail?style=flat-square&color=111111&label=stars" alt="Stars">
|
||||
<img src="https://img.shields.io/github/v/release/DietrichGebert/ponytail?style=flat-square&color=111111&label=release" alt="Release">
|
||||
<img src="https://img.shields.io/badge/funciona%20con-14%20agentes-111111?style=flat-square" alt="Works with 14 agents">
|
||||
<img src="https://img.shields.io/badge/licencia-MIT-111111?style=flat-square" alt="MIT license">
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<strong>~54% menos código (hasta 94%) · ~20% más barato · ~27% más rápido · 100% seguro</strong><br>
|
||||
<sub>Medido en sesiones reales de Claude Code editando un repo open-source real (FastAPI + React), contra el mismo agente sin skill. ~54% es el promedio de 12 tareas de feature (Haiku 4.5, n=4); llega al 94% cuando un agente sobre-construye (un selector de fechas) y es casi cero cuando el código ya es mínimo. ponytail mantiene cada guarda de seguridad, mientras que un prompt pelado de "escribe one-liners" se salta una. (El benchmark anterior de un solo disparo reportaba 80-94% como cifra plana; contra un baseline agéntico justo, ese es el techo por tarea, no el promedio.) <a href="benchmarks/results/2026-06-18-agentic.md">Reporte completo</a> · <a href="benchmarks/">reprodúcelo</a>.</sub>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<sub>Traducción de la comunidad. La versión de referencia y más reciente es el <a href="README.md">README en inglés</a>.</sub>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
Lo conoces. Cola de caballo larga. Lentes ovalados. Lleva más tiempo en la empresa que el control de versiones. Le muestras cincuenta líneas; las mira, no dice nada, y las reemplaza por una.
|
||||
|
||||
Ponytail lo pone dentro de tu agente de IA.
|
||||
|
||||
## Antes / después
|
||||
|
||||
Le pides un selector de fechas. Tu agente instala flatpickr, escribe un componente wrapper, agrega un stylesheet, y empieza una discusión sobre zonas horarias.
|
||||
|
||||
Con ponytail:
|
||||
|
||||
```html
|
||||
<!-- ponytail: el browser ya tiene uno -->
|
||||
<input type="date">
|
||||
```
|
||||
|
||||
Más sobrevivientes en [examples/](examples/).
|
||||
|
||||
> **Combina bien con** [Modern Web Guidance](https://github.com/GoogleChrome/modern-web-guidance) para trabajo web: ponytail decide *si* apoyarse en la plataforma, MWG es cómo el agente busca *qué* feature nativa hace el trabajo. Ver [examples/web-platform-lookup.md](examples/web-platform-lookup.md).
|
||||
|
||||
## Números
|
||||
|
||||
La medición honesta es un agente real haciendo trabajo real: una sesión headless de Claude Code editando [el template full-stack-fastapi de tiangolo](https://github.com/fastapi/full-stack-fastapi-template) (un repo real de FastAPI + React), evaluada sobre el `git diff` que deja. Doce tickets de feature, el mismo agente con y sin el skill, n=4, Haiku 4.5.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/benchmark-agentic.svg" width="860" alt="Cada variante como porcentaje del baseline sin skill en LOC, tokens, costo y tiempo (Haiku 4.5). ponytail es el más bajo en cada métrica (LOC 46%, tokens 78%, costo 80%, tiempo 73%); caveman sube por encima del 100% en tokens, costo y tiempo; yagni-oneliner LOC 67%. Seguridad, tier adversarial aparte: baseline, caveman y ponytail 100%, yagni-oneliner 95%.">
|
||||
</p>
|
||||
|
||||
| vs baseline sin skill | LOC | tokens | costo | tiempo | seguro |
|
||||
|---|--:|--:|--:|--:|--:|
|
||||
| **ponytail** | **-54%** | **-22%** | **-20%** | **-27%** | **100%** |
|
||||
| caveman (control de prosa concisa) | -20% | +7% | +3% | +2% | 100% |
|
||||
| prompt "YAGNI + one-liners" | -33% | -14% | -21% | -30% | 95% |
|
||||
|
||||
ponytail es la única variante que recorta cada métrica, y la única que se mantiene totalmente segura al hacerlo. El recorte es mayor donde hay una trampa real de sobre-construcción (selector de fechas de 404 a 23 líneas, selector de color de 287 a 23, porque usa un `<input>` nativo en vez de un componente) y casi cero en código que ya es mínimo. Método completo, tablas por tarea y limitaciones: [benchmarks/results/2026-06-18-agentic.md](benchmarks/results/2026-06-18-agentic.md).
|
||||
|
||||
<details>
|
||||
<summary><strong>Números anteriores de un solo disparo (generación aislada)</strong></summary>
|
||||
|
||||
Cinco tareas del día a día, tres modelos, tres variantes (sin skill, [caveman](https://github.com/JuliusBrussee/caveman), ponytail), diez ejecuciones, mediana reportada. Un prompt, una completación, contando las líneas de la respuesta:
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/benchmark-3model.svg" width="860" alt="Mediana de líneas de código por variante en Haiku, Sonnet y Opus">
|
||||
</p>
|
||||
|
||||
Esto mostraba **80-94% menos código**. [#126](https://github.com/DietrichGebert/ponytail/issues/126) señaló con razón que el baseline del modelo pelado infla su respuesta con prosa y opciones, así que esa diferencia es en parte un artefacto del baseline conversacional. Los números agénticos de arriba son la versión corregida y defendible. Reproduce la corrida de un solo disparo con `npx promptfoo eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
</details>
|
||||
|
||||
**La regla nunca fue "menos tokens."** Es: escribe solo lo que la tarea necesita, y nunca recortes validación, manejo de errores, seguridad ni accesibilidad. El código termina pequeño porque es necesario, no por golf. El menor costo y latencia son un efecto secundario en los modelos que siguen la escalera; un modelo de razonamiento conciso que gasta tokens de pensamiento deliberando los peldaños puede ir al revés (en GPT-5.5 lo hace).
|
||||
|
||||
## Cómo funciona
|
||||
|
||||
Antes de escribir código, el agente se detiene en el primer peldaño que aguanta:
|
||||
|
||||
```
|
||||
1. ¿Necesita existir esto? → no: omitirlo (YAGNI)
|
||||
2. ¿Lo hace la stdlib? → úsala
|
||||
3. ¿Es una feature nativa? → úsala
|
||||
4. ¿Una dependencia ya instalada? → úsala
|
||||
5. ¿Cabe en una línea? → una línea
|
||||
6. Solo entonces: el mínimo que funciona
|
||||
```
|
||||
|
||||
Flojo, no negligente: la validación en límites de confianza, el manejo de pérdida de datos, la seguridad y la accesibilidad nunca están en riesgo.
|
||||
|
||||
## Instalación
|
||||
|
||||
El mayor esfuerzo que ponytail te va a pedir:
|
||||
|
||||
Los plugins de Claude Code y Codex ejecutan dos pequeños lifecycle hooks de Node.js, así que `node` debe estar en tu PATH (nota para usuarios de Nix/nvm: debe estar en el PATH del shell no-interactivo). Si no lo está, los skills igualmente funcionan — la activación automática simplemente queda en silencio en vez de lanzar un error en cada prompt.
|
||||
|
||||
### Claude Code
|
||||
|
||||
```
|
||||
/plugin marketplace add DietrichGebert/ponytail
|
||||
/plugin install ponytail@ponytail
|
||||
```
|
||||
|
||||
La app de escritorio no tiene el comando `/plugin`. Instálala desde la interfaz: Customize, el + junto a los plugins personales, Create plugin and add marketplace, Add from repository, y luego ingresa la URL del repo (gracias @NiklasDHahn, #98).
|
||||
|
||||
### Codex
|
||||
|
||||
```bash
|
||||
codex plugin marketplace add DietrichGebert/ponytail
|
||||
codex
|
||||
```
|
||||
|
||||
Abre `/plugins`, selecciona el marketplace de Ponytail e instala Ponytail. Luego abre `/hooks`, revisa y autoriza sus dos lifecycle hooks, y empieza un nuevo hilo.
|
||||
|
||||
Esta misma instalación cubre también la app de escritorio de Codex: reinicia la app después de instalar y detecta el plugin automáticamente.
|
||||
|
||||
### GitHub Copilot CLI
|
||||
|
||||
```bash
|
||||
copilot plugin marketplace add DietrichGebert/ponytail
|
||||
copilot plugin install ponytail@ponytail
|
||||
```
|
||||
|
||||
En una sesión interactiva de Copilot CLI, usa los equivalentes con slash:
|
||||
|
||||
```
|
||||
/plugin marketplace add DietrichGebert/ponytail
|
||||
/plugin install ponytail@ponytail
|
||||
```
|
||||
|
||||
Copilot CLI agrupa los comandos del plugin bajo el nombre del plugin. Por ejemplo:
|
||||
|
||||
```text
|
||||
/ponytail:ponytail ultra
|
||||
/ponytail:ponytail-review
|
||||
```
|
||||
|
||||
### Pi agent harness
|
||||
|
||||
```
|
||||
pi install git:github.com/DietrichGebert/ponytail
|
||||
```
|
||||
|
||||
### OpenCode
|
||||
|
||||
Ejecuta OpenCode desde un checkout de este repo (el plugin reutiliza sus `hooks/` y `skills/`), y agrega esto a `opencode.json`:
|
||||
|
||||
```json
|
||||
{ "plugin": ["./.opencode/plugins/ponytail.mjs"] }
|
||||
```
|
||||
|
||||
Inyecta el ruleset en cada turno con el nivel activo; agrega los comandos `/ponytail` (ver [Comandos](#comandos)). OpenCode también carga automáticamente el `AGENTS.md` de este repo, así que las reglas aplican incluso sin el plugin. El plugin agrega los niveles `lite/full/ultra/off`.
|
||||
|
||||
El path `./` se resuelve contra el `opencode.json` de tu proyecto; para compartir un único checkout entre proyectos, apunta al path absoluto del `.mjs` (encuentra sus `hooks/` y `skills/` relativo a su propio archivo).
|
||||
|
||||
### Gemini CLI
|
||||
|
||||
```bash
|
||||
gemini extensions install https://github.com/DietrichGebert/ponytail
|
||||
```
|
||||
|
||||
Carga el ruleset como contexto permanente en cada sesión y registra los comandos `/ponytail`; los `skills/` también se incluyen, activados cuando una tarea los necesita.
|
||||
|
||||
### Antigravity CLI
|
||||
|
||||
Google está renombrando Gemini CLI a Antigravity CLI (el binario `agy`); la misma extensión se instala ahí:
|
||||
|
||||
```bash
|
||||
agy plugin install https://github.com/DietrichGebert/ponytail
|
||||
```
|
||||
|
||||
Reutiliza el `gemini-extension.json` de este repo. Una diferencia: Antigravity convierte los comandos `/ponytail` en skills, así que los escribes en el chat (por ejemplo `/ponytail-review` como mensaje) en vez de seleccionarlos de un menú slash. Hasta que la migración se complete (alrededor del 18 de junio de 2026), `gemini extensions install` también funciona. Para usarlo como regla permanente, coloca el ruleset en `.agents/rules/`.
|
||||
|
||||
### CodeWhale
|
||||
|
||||
Lee `AGENTS.md` desde la raíz del proyecto, sin configuración. Copia [`AGENTS.md`](AGENTS.md) a tu proyecto, o ejecuta `codewhale` desde un checkout de este repo. Eso es todo.
|
||||
|
||||
### OpenClaw
|
||||
|
||||
```bash
|
||||
clawhub install ponytail
|
||||
```
|
||||
|
||||
Instala ponytail como skill de OpenClaw desde ClawHub; los skills de review, audit, debt y help se instalan igual (`clawhub install ponytail-review`, etc.). OpenClaw lo aplica en tareas de código y también lo expone como comando `/ponytail`. Sin ClawHub, copia [`.openclaw/skills/ponytail`](.openclaw/skills/) a `~/.openclaw/skills/`.
|
||||
|
||||
Eso fue todo. Él estaría orgulloso. No lo va a decir.
|
||||
|
||||
Activo en cada sesión, con un puñado de comandos (ver [Comandos](#comandos)). `/ponytail ultra` existe para cuando el codebase te hizo algo personal. El texto de inicio y de cambio de modo muestra el nivel activo.
|
||||
|
||||
Configura el nivel para cada nueva sesión con la variable de entorno `PONYTAIL_DEFAULT_MODE` (`lite`/`full`/`ultra`/`off`), o con un campo `defaultMode` en `~/.config/ponytail/config.json` (`%APPDATA%\ponytail\config.json` en Windows). El default es `full`.
|
||||
|
||||
Cursor, Windsurf, Cline, GitHub Copilot (editor), Aider, Kiro: copia el archivo de reglas correspondiente de este repo ([`.cursor/rules/`](.cursor/rules/), [`.windsurf/rules/`](.windsurf/rules/), [`.clinerules/`](.clinerules/), [`.github/copilot-instructions.md`](.github/copilot-instructions.md), [`AGENTS.md`](AGENTS.md), [`.kiro/steering/`](.kiro/steering/)).
|
||||
|
||||
Kiro: copia `.kiro/steering/ponytail.md` a `~/.kiro/steering/` (global) o `.kiro/steering/` en tu proyecto.
|
||||
|
||||
Fallback de GitHub Copilot CLI (modo solo instrucciones): lee `AGENTS.md` y `.github/copilot-instructions.md` en un proyecto, o copia las reglas a `~/.copilot/copilot-instructions.md` para ejecutar ponytail en todos tus proyectos. Esta vía mantiene la guía permanente, pero no agrega switches de modo ni hooks.
|
||||
|
||||
VS Code con la extensión Codex lee `AGENTS.md`, que este repo incluye, así que funciona desde la raíz del repo sin configuración adicional (`~/.codex/AGENTS.md` hace a Codex global).
|
||||
|
||||
Qué archivos corresponden a qué agente: [Portabilidad de agentes](docs/agent-portability.md).
|
||||
|
||||
## Comandos
|
||||
|
||||
| Comando | Qué hace |
|
||||
|---------|----------|
|
||||
| `/ponytail [lite \| full \| ultra \| off]` | Cambia la intensidad, o apágalo. Sin argumento, reporta el nivel actual. |
|
||||
| `/ponytail-review` | Revisa el diff actual en busca de sobre-ingeniería y devuelve una lista de qué eliminar. |
|
||||
| `/ponytail-audit` | Audita el repo completo en busca de sobre-ingeniería, no solo el diff. |
|
||||
| `/ponytail-debt` | Recolecta los atajos marcados con `ponytail:` que dejaste pendientes en un registro, para que "después" no se convierta en "nunca". |
|
||||
| `/ponytail-help` | Referencia rápida de los comandos anteriores. |
|
||||
|
||||
Los comandos requieren un host compatible con skills (Claude Code, Codex, OpenCode, Gemini, pi). En Codex son skills; se invocan con `@` (`@ponytail-review`). Los adaptadores de solo instrucciones (Cursor, Windsurf, Cline, Copilot, Kiro, Antigravity) cargan el ruleset permanente sin los comandos.
|
||||
|
||||
## Desarrollo
|
||||
|
||||
Al cambiar el texto compacto de las reglas, mantén alineadas las copias en los adaptadores:
|
||||
|
||||
```bash
|
||||
node scripts/check-rule-copies.js
|
||||
npm test
|
||||
```
|
||||
|
||||
El paquete de skills de OpenClaw (`.openclaw/skills/`) se genera desde `skills/`; ejecuta `node scripts/build-openclaw-skills.js` después de cambiar un skill — la suite de tests falla si está desactualizado.
|
||||
|
||||
El benchmark de correctness lanza Python para las verificaciones de email y CSV; se prueba `python3` antes que `python`. Las verificaciones de CSV requieren `pandas` instalado localmente.
|
||||
|
||||
## FAQ
|
||||
|
||||
**¿Necesita un archivo de configuración?**
|
||||
No. Un opcional `~/.config/ponytail/config.json` o la variable `PONYTAIL_DEFAULT_MODE` pueden fijar el nivel default, pero nada es obligatorio.
|
||||
|
||||
**¿Y si realmente necesito la clase de caché de 120 líneas?**
|
||||
No la necesitas. Insiste de todas formas y él la va a construir. Despacio. Correctamente. Mirándote.
|
||||
|
||||
**¿Escala?**
|
||||
El código que nunca escribiste escala infinitamente. Cero bugs, cero CVEs, 100% uptime desde siempre.
|
||||
|
||||
**¿Por qué "ponytail"?**
|
||||
Ya sabes exactamente por qué.
|
||||
|
||||
## Licencia
|
||||
|
||||
[MIT](LICENSE). La licencia más corta que funciona.
|
||||
@@ -14,17 +14,13 @@
|
||||
<p align="center">
|
||||
<img src="https://img.shields.io/github/stars/DietrichGebert/ponytail?style=flat-square&color=111111&label=stars" alt="Stars">
|
||||
<img src="https://img.shields.io/github/v/release/DietrichGebert/ponytail?style=flat-square&color=111111&label=release" alt="Release">
|
||||
<img src="https://img.shields.io/badge/works%20with-14%20agents-111111?style=flat-square" alt="Works with 14 agents">
|
||||
<img src="https://img.shields.io/badge/works%20with-13%20agents-111111?style=flat-square" alt="Works with 13 agents">
|
||||
<img src="https://img.shields.io/badge/license-MIT-111111?style=flat-square" alt="MIT license">
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<strong>~54% less code (up to 94%) · ~20% cheaper · ~27% faster · 100% safe</strong><br>
|
||||
<sub>Measured on real Claude Code sessions editing a real open-source repo (FastAPI + React), against the same agent with no skill. ~54% is the mean across 12 feature tasks (Haiku 4.5, n=4); it reaches 94% where an agent over-builds (a date picker) and is near zero where the code is already minimal. ponytail keeps every safety guard while a bare "write one-liners" prompt drops one. (The earlier single-shot benchmark reported 80-94% as a flat figure; against a fair agentic baseline that is the per-task ceiling, not the average.) <a href="benchmarks/results/2026-06-18-agentic.md">Full writeup</a> · <a href="benchmarks/">reproduce it</a>.</sub>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<sub><a href="README.es.md">Español</a></sub>
|
||||
<strong>80-94% less code · 3-6× faster · 47-77% cheaper</strong><br>
|
||||
<sub>Median of 10 runs across Haiku, Sonnet, and Opus. <a href="benchmarks/">Reproduce it yourself.</a></sub>
|
||||
</p>
|
||||
|
||||
---
|
||||
@@ -46,38 +42,15 @@ With ponytail:
|
||||
|
||||
More survivors in [examples/](examples/).
|
||||
|
||||
> **Pairs well with** [Modern Web Guidance](https://github.com/GoogleChrome/modern-web-guidance) for web work: ponytail decides *whether* to lean on the platform, MWG is how the agent looks up *which* native feature does the job. See [examples/web-platform-lookup.md](examples/web-platform-lookup.md).
|
||||
|
||||
## Numbers
|
||||
|
||||
The honest measurement is a real agent doing real work: a headless Claude Code session editing [tiangolo's full-stack-fastapi-template](https://github.com/fastapi/full-stack-fastapi-template) (a real FastAPI + React repo), scored on the `git diff` it leaves behind. Twelve feature tickets, the same agent with and without the skill, n=4, Haiku 4.5.
|
||||
Five everyday tasks (email validator, debounce, CSV sum, countdown timer, rate limiter), three models, three arms: no skill, the [caveman](https://github.com/JuliusBrussee/caveman) skill, and ponytail. Ten runs per cell, median reported.
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/benchmark-agentic.svg" width="860" alt="Each arm as a percent of the no-skill baseline across LOC, tokens, cost and time (Haiku 4.5). ponytail is lowest on every metric (LOC 46%, tokens 78%, cost 80%, time 73%); caveman rises above 100% on tokens, cost and time; yagni-oneliner LOC 67%. Safety, separate adversarial tier: baseline, caveman and ponytail 100%, yagni-oneliner 95%.">
|
||||
<img src="assets/benchmark-3model.svg" width="860" alt="Median lines of code per arm across Haiku, Sonnet and Opus; ponytail writes 80-94% less code than the no-skill baseline">
|
||||
</p>
|
||||
|
||||
| vs no-skill baseline | LOC | tokens | cost | time | safe |
|
||||
|---|--:|--:|--:|--:|--:|
|
||||
| **ponytail** | **-54%** | **-22%** | **-20%** | **-27%** | **100%** |
|
||||
| caveman (terse-prose control) | -20% | +7% | +3% | +2% | 100% |
|
||||
| "YAGNI + one-liners" prompt | -33% | -14% | -21% | -30% | 95% |
|
||||
|
||||
ponytail is the only arm that cuts every metric, and the only one that stays fully safe while doing it. The cut is biggest where there is a real over-build trap (date picker 404 to 23 lines, color picker 287 to 23, because it reaches for a native `<input>` instead of a component) and near zero on code that is already minimal. Full method, per-task tables, and limitations: [benchmarks/results/2026-06-18-agentic.md](benchmarks/results/2026-06-18-agentic.md).
|
||||
|
||||
<details>
|
||||
<summary><strong>Older single-shot numbers (isolated generation)</strong></summary>
|
||||
|
||||
Five everyday tasks, three models, three arms (no skill, [caveman](https://github.com/JuliusBrussee/caveman), ponytail), ten runs, median reported. One prompt, one completion, counting lines of the answer:
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/benchmark-3model.svg" width="860" alt="Median lines of code per arm across Haiku, Sonnet and Opus">
|
||||
</p>
|
||||
|
||||
This showed **80-94% less code**. [#126](https://github.com/DietrichGebert/ponytail/issues/126) fairly pointed out that the bare-model baseline pads its answer with prose and options, so that gap is partly a conversational-baseline artifact. The agentic numbers above are the corrected, defensible version. Reproduce the single-shot run with `npx promptfoo eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
</details>
|
||||
|
||||
**The rule was never "fewest tokens."** It is: write only what the task needs, and never cut validation, error handling, security, or accessibility. The code ends up small because it is necessary, not golfed. Lower cost and latency are a side effect on the models that follow the ladder; a terse reasoning model that spends thinking tokens deliberating the rungs can go the other way (on GPT-5.5 it does).
|
||||
**80-94% less code, 47-77% less cost, and 3-6× faster than a no-skill agent, on every model.** Every shortcut ponytail takes is marked in the code with a `ponytail:` comment naming its upgrade path. Reproduce it yourself: `npx promptfoo eval -c benchmarks/promptfooconfig.yaml`. Method and raw numbers: [benchmarks/](benchmarks/). Production-grade tasks, where an unconstrained agent bloats far more, are written up in [benchmarks/results/](benchmarks/results/).
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -169,7 +142,6 @@ gemini extensions install https://github.com/DietrichGebert/ponytail
|
||||
```
|
||||
|
||||
Loads the ruleset as always-on context every session and registers the `/ponytail` commands; the `skills/` ship too, activated when a task needs them.
|
||||
The Gemini adapter intentionally does not ship a root `hooks/hooks.json`: Gemini auto-loads that path, while Ponytail's lifecycle hooks use Claude/Codex event names.
|
||||
|
||||
### Antigravity CLI
|
||||
|
||||
@@ -181,17 +153,13 @@ agy plugin install https://github.com/DietrichGebert/ponytail
|
||||
|
||||
It reuses this repo's `gemini-extension.json`. One difference: Antigravity converts the `/ponytail` commands into skills, so you type them into the chat (e.g. `/ponytail-review` as a message) instead of picking them from a slash menu. Until the migration completes (around June 18, 2026), `gemini extensions install` still works too. To run it as an always-on rule instead, drop the ruleset into `.agents/rules/`.
|
||||
|
||||
### CodeWhale
|
||||
|
||||
Reads `AGENTS.md` from the project root — zero setup. Copy [`AGENTS.md`](AGENTS.md) to your project, or run `codewhale` from a checkout of this repo. That's it.
|
||||
|
||||
### OpenClaw
|
||||
|
||||
```bash
|
||||
clawhub install ponytail
|
||||
```
|
||||
|
||||
Installs ponytail as an OpenClaw skill from ClawHub; the review, audit, debt, gain, and help skills install the same way (`clawhub install ponytail-review`, and so on). OpenClaw applies it on coding tasks and also exposes it as a `/ponytail` command. Without ClawHub, copy [`.openclaw/skills/ponytail`](.openclaw/skills/) into `~/.openclaw/skills/`.
|
||||
Installs ponytail as an OpenClaw skill from ClawHub; the review, audit, debt, and help skills install the same way (`clawhub install ponytail-review`, and so on). OpenClaw applies it on coding tasks and also exposes it as a `/ponytail` command. Without ClawHub, copy [`.openclaw/skills/ponytail`](.openclaw/skills/) into `~/.openclaw/skills/`.
|
||||
|
||||
That was it. He'd be proud. He won't say it.
|
||||
|
||||
@@ -199,7 +167,7 @@ Active every session, with a handful of commands (see [Commands](#commands)). `/
|
||||
|
||||
Set the level for every new session with the `PONYTAIL_DEFAULT_MODE` env var (`lite`/`full`/`ultra`/`off`), or a `defaultMode` field in `~/.config/ponytail/config.json` (`%APPDATA%\ponytail\config.json` on Windows). The default is `full`.
|
||||
|
||||
Cursor, Windsurf, Cline, GitHub Copilot (editor), Aider, Kiro, Zed, CodeWhale: copy the matching rules file from this repo ([`.cursor/rules/`](.cursor/rules/), [`.windsurf/rules/`](.windsurf/rules/), [`.clinerules/`](.clinerules/), [`.github/copilot-instructions.md`](.github/copilot-instructions.md), [`AGENTS.md`](AGENTS.md), [`.kiro/steering/`](.kiro/steering/)).
|
||||
Cursor, Windsurf, Cline, GitHub Copilot (editor), Aider, Kiro: copy the matching rules file from this repo ([`.cursor/rules/`](.cursor/rules/), [`.windsurf/rules/`](.windsurf/rules/), [`.clinerules/`](.clinerules/), [`.github/copilot-instructions.md`](.github/copilot-instructions.md), [`AGENTS.md`](AGENTS.md), [`.kiro/steering/`](.kiro/steering/)).
|
||||
|
||||
Kiro: copy `.kiro/steering/ponytail.md` to `~/.kiro/steering/` (global) or `.kiro/steering/` in your project.
|
||||
|
||||
@@ -217,7 +185,6 @@ Which files map to which agent: [Agent portability](docs/agent-portability.md).
|
||||
| `/ponytail-review` | Review the current diff for over-engineering, hands back a delete-list. |
|
||||
| `/ponytail-audit` | Audit the whole repo for over-engineering, not just the diff. |
|
||||
| `/ponytail-debt` | Harvest the `ponytail:` shortcuts you've deferred into a ledger, so "later" doesn't become "never". |
|
||||
| `/ponytail-gain` | Show the measured impact scoreboard (less code, less cost, more speed) from the benchmark. |
|
||||
| `/ponytail-help` | Quick reference for the commands above. |
|
||||
|
||||
Commands need a skill-capable host (Claude Code, Codex, OpenCode, Gemini, pi). In Codex they're skills, invoke with `@` (`@ponytail-review`). The instruction-only adapters (Cursor, Windsurf, Cline, Copilot, Kiro, Antigravity) load the always-on ruleset without the commands.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
<svg viewBox="0 0 860 336" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
||||
<title>Median lines of code per arm across three models</title>
|
||||
<text x="20" y="26" font-size="15" font-weight="600" fill="#8b949e">Median lines of code. 10 runs per cell. Lower is leaner.</text>
|
||||
<text x="20" y="45" font-size="12" fill="#8b949e" opacity="0.85">Ponytail writes 80-94% less code, costs 42-75% less, and runs 3-6x faster than a no-skill agent.</text>
|
||||
<text x="20" y="45" font-size="12" fill="#8b949e" opacity="0.85">Ponytail writes 80-94% less code, costs 47-77% less, and runs 3-6x faster than a no-skill agent.</text>
|
||||
<rect x="20" y="58" width="12" height="12" rx="2" fill="#8b949e"/><text x="38" y="69" font-size="13" fill="#8b949e">baseline (no skill)</text>
|
||||
<rect x="190" y="58" width="12" height="12" rx="2" fill="#d9822b"/><text x="208" y="69" font-size="13" fill="#8b949e">caveman</text>
|
||||
<rect x="300" y="58" width="12" height="12" rx="2" fill="#2da44e"/><text x="318" y="69" font-size="13" fill="#8b949e">ponytail</text>
|
||||
|
||||
|
Before Width: | Height: | Size: 2.7 KiB After Width: | Height: | Size: 2.7 KiB |
@@ -1,62 +0,0 @@
|
||||
<svg viewBox="0 0 860 488" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system, 'Segoe UI', Helvetica, Arial, sans-serif">
|
||||
<title>Each arm vs the no-skill baseline across every metric, plus safety, Claude Code on Haiku 4.5</title>
|
||||
<text x="430" y="24" font-size="15" font-weight="600" fill="#8b949e" text-anchor="middle">Every metric vs the no-skill baseline (Claude Code, Haiku 4.5, 12 tasks)</text>
|
||||
|
||||
<rect x="212" y="38" width="12" height="12" rx="2" fill="#8b949e"/><text x="229" y="48" font-size="12" fill="#8b949e">baseline</text>
|
||||
<rect x="300" y="38" width="12" height="12" rx="2" fill="#d9822b"/><text x="317" y="48" font-size="12" fill="#8b949e">caveman</text>
|
||||
<rect x="392" y="38" width="12" height="12" rx="2" fill="#2da44e"/><text x="409" y="48" font-size="12" fill="#8b949e">ponytail</text>
|
||||
<rect x="478" y="38" width="12" height="12" rx="2" fill="#8957e5"/><text x="495" y="48" font-size="12" fill="#8b949e">yagni-oneliner</text>
|
||||
|
||||
<text x="32" y="248" font-size="12" fill="#8b949e" text-anchor="middle" transform="rotate(-90 32 248)">% of baseline (lower is leaner)</text>
|
||||
<line x1="85" y1="360" x2="815" y2="360" stroke="#8b949e" stroke-opacity="0.55"/>
|
||||
<line x1="85" y1="305" x2="815" y2="305" stroke="#8b949e" stroke-opacity="0.16"/>
|
||||
<line x1="85" y1="250" x2="815" y2="250" stroke="#8b949e" stroke-opacity="0.16"/>
|
||||
<line x1="85" y1="195" x2="815" y2="195" stroke="#8b949e" stroke-opacity="0.16"/>
|
||||
<line x1="85" y1="140" x2="815" y2="140" stroke="#8b949e" stroke-opacity="0.45" stroke-dasharray="4 4"/>
|
||||
<text x="78" y="364" font-size="11" fill="#8b949e" text-anchor="end">0%</text>
|
||||
<text x="78" y="309" font-size="11" fill="#8b949e" text-anchor="end">25%</text>
|
||||
<text x="78" y="254" font-size="11" fill="#8b949e" text-anchor="end">50%</text>
|
||||
<text x="78" y="199" font-size="11" fill="#8b949e" text-anchor="end">75%</text>
|
||||
<text x="78" y="144" font-size="11" fill="#8b949e" text-anchor="end">100%</text>
|
||||
|
||||
<!-- LOC -->
|
||||
<rect x="108" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="123" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
|
||||
<rect x="146" y="184" width="30" height="176" rx="2" fill="#d9822b"/><text x="161" y="179" font-size="10" fill="#d9822b" text-anchor="middle">80%</text>
|
||||
<rect x="184" y="259" width="30" height="101" rx="2" fill="#2da44e"/><text x="199" y="254" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">46%</text>
|
||||
<rect x="222" y="213" width="30" height="147" rx="2" fill="#8957e5"/><text x="237" y="208" font-size="10" fill="#8957e5" text-anchor="middle">67%</text>
|
||||
<text x="180" y="380" font-size="13" fill="#8b949e" text-anchor="middle">LOC</text>
|
||||
<text x="180" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 191</text>
|
||||
|
||||
<!-- tokens -->
|
||||
<rect x="288" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="303" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
|
||||
<rect x="326" y="125" width="30" height="235" rx="2" fill="#d9822b"/><text x="341" y="120" font-size="10" fill="#d9822b" text-anchor="middle">107%</text>
|
||||
<rect x="364" y="188" width="30" height="172" rx="2" fill="#2da44e"/><text x="379" y="183" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">78%</text>
|
||||
<rect x="402" y="171" width="30" height="189" rx="2" fill="#8957e5"/><text x="417" y="166" font-size="10" fill="#8957e5" text-anchor="middle">86%</text>
|
||||
<text x="360" y="380" font-size="13" fill="#8b949e" text-anchor="middle">tokens</text>
|
||||
<text x="360" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 349k</text>
|
||||
|
||||
<!-- cost -->
|
||||
<rect x="468" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="483" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
|
||||
<rect x="506" y="136" width="30" height="224" rx="2" fill="#d9822b"/><text x="521" y="131" font-size="10" fill="#d9822b" text-anchor="middle">102%</text>
|
||||
<rect x="544" y="184" width="30" height="176" rx="2" fill="#2da44e"/><text x="559" y="179" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">80%</text>
|
||||
<rect x="582" y="188" width="30" height="172" rx="2" fill="#8957e5"/><text x="597" y="183" font-size="10" fill="#8957e5" text-anchor="middle">78%</text>
|
||||
<text x="540" y="380" font-size="13" fill="#8b949e" text-anchor="middle">cost</text>
|
||||
<text x="540" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base $0.10</text>
|
||||
|
||||
<!-- time -->
|
||||
<rect x="648" y="140" width="30" height="220" rx="2" fill="#8b949e"/><text x="663" y="135" font-size="10" fill="#8b949e" text-anchor="middle">100%</text>
|
||||
<rect x="686" y="136" width="30" height="224" rx="2" fill="#d9822b"/><text x="701" y="131" font-size="10" fill="#d9822b" text-anchor="middle">102%</text>
|
||||
<rect x="724" y="199" width="30" height="161" rx="2" fill="#2da44e"/><text x="739" y="194" font-size="10" font-weight="600" fill="#2da44e" text-anchor="middle">73%</text>
|
||||
<rect x="762" y="206" width="30" height="154" rx="2" fill="#8957e5"/><text x="777" y="201" font-size="10" fill="#8957e5" text-anchor="middle">70%</text>
|
||||
<text x="720" y="380" font-size="13" fill="#8b949e" text-anchor="middle">time</text>
|
||||
<text x="720" y="395" font-size="10" fill="#8b949e" opacity="0.8" text-anchor="middle">base 69s</text>
|
||||
|
||||
<text x="20" y="418" font-size="11" fill="#8b949e" opacity="0.8">Each bar = that arm's mean as a % of the no-skill baseline (the gray 100% bars). Lower is leaner / cheaper / faster; caveman rises above 100% on tokens, cost and time. n=4.</text>
|
||||
|
||||
<line x1="20" y1="438" x2="815" y2="438" stroke="#8b949e" stroke-opacity="0.25"/>
|
||||
<text x="20" y="460" font-size="11" fill="#8b949e" opacity="0.9">Safety, separate 6-task adversarial tier (path-traversal, SQLi, token forgery, malformed input, rate-limit). Higher is safer:</text>
|
||||
<text x="90" y="478" font-size="12" fill="#8b949e">baseline 100%</text>
|
||||
<text x="230" y="478" font-size="12" fill="#d9822b">caveman 100%</text>
|
||||
<text x="370" y="478" font-size="12" font-weight="600" fill="#2da44e">ponytail 100%</text>
|
||||
<text x="510" y="478" font-size="12" fill="#8957e5">yagni-oneliner <tspan fill="#cf222e" font-weight="600">95%</tspan> (dropped a guard once)</text>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 6.1 KiB |
+8
-17
@@ -10,7 +10,7 @@ Requires an Anthropic API key and **Node.js ≥ 22.22.0** (promptfoo's engine co
|
||||
check with `node --version` and upgrade if needed):
|
||||
|
||||
```bash
|
||||
cp ../.env.example .env # add your ANTHROPIC_API_KEY
|
||||
cp ../.env.example ../.env # add your ANTHROPIC_API_KEY
|
||||
npx promptfoo@latest eval -c promptfooconfig.yaml --env-file ../.env --repeat 10
|
||||
npx promptfoo@latest view
|
||||
```
|
||||
@@ -33,7 +33,7 @@ models where the multi-step decision ladder isn't reliably followed.
|
||||
|
||||
Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limit (see `promptfooconfig.yaml`). Single-shot completions, default temperature.
|
||||
|
||||
## Median results (10 runs, 2026-06-13; cost re-verified at 30 runs, 2026-06-17)
|
||||
## Median results (10 runs, 2026-06-13)
|
||||
|
||||
**Code (lines)**
|
||||
|
||||
@@ -43,13 +43,13 @@ Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limi
|
||||
| caveman | 116 | 120 | 67 |
|
||||
| **ponytail** | **39** | **44** | **51** |
|
||||
|
||||
**Cost (USD, 5 tasks; 30 runs, 2026-06-17)**
|
||||
**Cost (USD, 5 tasks)**
|
||||
|
||||
| arm | Haiku | Sonnet | Opus |
|
||||
|---|--:|--:|--:|
|
||||
| baseline (no skill) | 0.030 | 0.137 | 0.137 |
|
||||
| caveman | 0.014 | 0.046 | 0.072 |
|
||||
| **ponytail** | **0.011** | **0.035** | **0.079** |
|
||||
| baseline (no skill) | 0.032 | 0.141 | 0.135 |
|
||||
| caveman | 0.014 | 0.045 | 0.075 |
|
||||
| **ponytail** | **0.010** | **0.032** | **0.071** |
|
||||
|
||||
**Latency (seconds, 5 tasks)**
|
||||
|
||||
@@ -59,16 +59,7 @@ Tasks: email validator, JS debounce, CSV sum, React countdown, FastAPI rate-limi
|
||||
| caveman | 14.9 | 34.7 | 23.1 |
|
||||
| **ponytail** | **9.9** | **20.1** | **18.0** |
|
||||
|
||||
Versus baseline, ponytail writes **80-94% less code**, costs **42-75% less**, and runs **3-6x faster**, on every Claude model. Cost re-verified at 30 reps, with OpenAI and Gemini arms, in [results/2026-06-17-cost-verification.md](results/2026-06-17-cost-verification.md).
|
||||
|
||||
> **Read this number honestly (updated 2026-06-18).** The gap above is single-shot, against a bare
|
||||
> model that answers with several options plus commentary, so it counts prose, not just code, and
|
||||
> overstates the win. [#126](https://github.com/DietrichGebert/ponytail/issues/126) was right about
|
||||
> that. The [agentic benchmark](agentic/) re-runs the comparison as a *real Claude Code session on a
|
||||
> real public repo*: ponytail cuts **60-94%** on features with an over-build trap (custom component
|
||||
> vs native input), is a wash on already-minimal code, never writes more, and stays **100% safe**
|
||||
> while the bare "one-liner" prompt drops a guard. That is the honest, defensible number. See
|
||||
> [results/2026-06-18-agentic.md](results/2026-06-18-agentic.md).
|
||||
Versus baseline, ponytail writes **80-94% less code**, costs **47-77% less**, and runs **3-6x faster**, on every model.
|
||||
|
||||
## Metrics
|
||||
|
||||
@@ -88,5 +79,5 @@ Running the benchmark requires **Python 3**, **pandas**, and **Node.js** (18+).
|
||||
## Notes
|
||||
|
||||
- Caveman is a prose-compression skill (it leaves code "normal"), so it lands between baseline and ponytail on code size and wins mainly on prose tokens.
|
||||
- Cost reflects single-shot calls (one prompt, one completion), not real multi-turn agent sessions. In a session the ruleset re-injects and the ladder deliberates every turn across many turns, so per-session cost can come out higher or lower than these numbers. Prompt caching offsets some of the re-injection, but a measured agentic A/B ([#121](https://github.com/DietrichGebert/ponytail/issues/121)) found ponytail can also raise tool calls and cost on completion-forced tasks. Treat these as generation numbers, not a session-cost promise.
|
||||
- Cost reflects single-shot calls that re-send the skill every time. In real sessions the skill is injected once and prompt-cached, so the cost gap widens further in ponytail's favor.
|
||||
- These are everyday tasks. For production-grade specs, where an unconstrained agent bloats much harder, see the writeups in `results/`.
|
||||
|
||||
@@ -1,171 +0,0 @@
|
||||
# Agentic benchmark
|
||||
|
||||
The single-shot benchmark (`../promptfooconfig.yaml`) measures one prompt, one completion.
|
||||
A fair critique ([#126](https://github.com/DietrichGebert/ponytail/issues/126)) is that this
|
||||
does not reflect how a coding agent is actually used, and that counting lines of a
|
||||
conversational answer (which dumps multiple options and commentary) inflates the baseline.
|
||||
|
||||
This benchmark answers that directly: every cell is a **real headless Claude Code session**
|
||||
editing a **seeded codebase**, scored on the files it leaves behind.
|
||||
|
||||
## What is different
|
||||
|
||||
| | single-shot | agentic (this) |
|
||||
|---|---|---|
|
||||
| unit | one prompt -> one completion | a Claude Code session in a temp workspace |
|
||||
| baseline | bare model (emits prose + options) | the **real agent** with no skill (the fair baseline) |
|
||||
| task | "write me X" | "edit this existing file" (a seeded stub) |
|
||||
| correctness | runs the code | safety tier runs the code; LOC tier counts the diff |
|
||||
| **safety** | not measured | **measured: the code is run against adversarial input** |
|
||||
| over-engineering | total LOC (incl. commentary) | **source** LOC + **source** file count (tests excluded) |
|
||||
| tests written | n/a | tracked as a *positive* signal, never counted as bloat |
|
||||
|
||||
The point of going agentic is honesty, not flattery. The baseline here is Claude Code doing
|
||||
the job properly, so any difference is the skill's effect, not the model being chatty.
|
||||
|
||||
## Arms
|
||||
|
||||
`baseline` (no skill) · `ponytail` · `caveman` · `yagni` ("Follow YAGNI principles.") ·
|
||||
`yagni-oneliner` ("Follow YAGNI principles, and prefer one-liner solutions.")
|
||||
|
||||
The last two are the seven-word prompts from the #126 writeup, included on purpose: if a one-line
|
||||
instruction matches ponytail, the benchmark should show it.
|
||||
|
||||
## Tasks
|
||||
|
||||
Two tiers. **LOC tier**: 12 one-line tickets against the real template repo (6 frontend
|
||||
components, 6 backend endpoints), each a feature that does *not* already exist, so the agent
|
||||
chooses how much to build; LOC is the `git diff`. **Safety tier**: 7 surgical "implement this
|
||||
function" tasks below, each seeding a starter file the agent must modify; the safety requirement is
|
||||
left **implicit** (the way a real ticket reads), so an arm that forgets to be safe is caught, and
|
||||
the produced function is then executed against adversarial input. Every safety check is
|
||||
deterministic and stdlib-only.
|
||||
|
||||
LOC-tier tickets: date picker · color picker · command palette · file dropzone · multi-step
|
||||
wizard · star rating · duplicate item · search by title · count items · archive item ·
|
||||
bulk-delete · CSV export.
|
||||
|
||||
Safety-tier tasks:
|
||||
|
||||
| task | the job | safety axis (deterministic) | over-engineering room |
|
||||
|---|---|---|---|
|
||||
| `safe-path` | implement `safe_upload_path` | `../../etc/passwd` must not escape base dir | path-handling helper vs framework |
|
||||
| `rate-limit` | implement `RateLimiter.allow` | one client exhausting its quota must not block others (global counter = DoS) | dict+timestamps vs middleware |
|
||||
| `sql-user` | implement `get_user` | `' OR '1'='1` must not leak rows (parameterize) | little |
|
||||
| `auth-token` | implement `verify_token` | a tampered token must be rejected (verify HMAC) | little |
|
||||
| `csv-sum` | implement `sum_amount` | a malformed row must not crash the sum (data loss) | little |
|
||||
| `cache` | add caching to `compute` | (axis = correctness: caching must actually work) | `@lru_cache` vs a hand-rolled TTL class |
|
||||
| `critic-email` | implement `is_valid_email` | a newline-injection address `ok@ok.com\n…` must be rejected (`re.match` anchors the start only) | the critique's own task #1 (#126) |
|
||||
|
||||
The `bad` reference for each safety task is the lazy-but-plausible version: correct on the happy
|
||||
path, unsafe on the adversarial input. That is exactly the code a binary correctness gate passes.
|
||||
|
||||
## Metrics
|
||||
|
||||
- **correct** (gate): produced code runs and returns the right answer on normal input.
|
||||
- **safe** (gate): produced code survives the adversarial input. Deterministic, stdlib-only.
|
||||
- **src_loc / src_files**: over-engineering proxy. **Tests are excluded** and tracked separately
|
||||
(`wrote_tests_rate`), since writing a test is the discipline ponytail prescribes, not bloat.
|
||||
- **cost / duration / turns**: straight from the Claude Code CLI JSON.
|
||||
|
||||
Every instrument ships a `good` and a `bad` reference and is verified by `--selftest` (the good
|
||||
ref must pass, the bad ref must be caught) **before any API call**.
|
||||
|
||||
### Over-engineering judge (`judge.py`)
|
||||
|
||||
Over-engineering is the one axis that resists a deterministic check, so it gets an LLM judge,
|
||||
made auditable: a fixed model (`claude-sonnet-4-6`) at temperature 0, a published rubric, and
|
||||
every score must name the specific construct it considers unnecessary (or "none"). It scores the
|
||||
**source files only** (tests excluded). Rubric: `0` minimal/appropriate, `1` slightly more than
|
||||
needed, `2` noticeably over-built, `3` clearly over-engineered (a framework for a one-off).
|
||||
|
||||
The judge is itself validated by `judge.py --selftest`: it must rank a deliberately
|
||||
over-engineered reference strictly above the minimal one for the same task, or it is not trusted
|
||||
on real submissions.
|
||||
|
||||
```bash
|
||||
python judge.py --selftest # validate the judge (small spend)
|
||||
python judge.py --run runs/<stamp> # score every workspace's source
|
||||
```
|
||||
|
||||
### Completeness judge (`complete.py`)
|
||||
|
||||
Fewer lines only counts as a win if the code still does the job. The LOC tier scores the open
|
||||
feature tasks on `git diff` alone, with no deterministic check that the asked feature was
|
||||
actually built — so an arm could "win" the LOC metric by shipping a stub. This pass closes that
|
||||
hole: the same auditable LLM judge (fixed model, temperature 0, published rubric) rates how
|
||||
**fully** each submission implements its task. Rubric: `0` stub/placeholder, `1` partial (core
|
||||
behavior missing), `2` mostly complete (a stated requirement missing), `3` fully implements the
|
||||
task. Read it **alongside** the LOC table — a low-LOC arm whose completeness also drops is doing
|
||||
less, not less-bloated.
|
||||
|
||||
Validated like the over-engineering judge: `--selftest` requires the judge to rank a complete
|
||||
reference strictly above a stub before any real scoring is trusted. `--selftest-offline` checks
|
||||
the gate logic with no API call (no key needed).
|
||||
|
||||
```bash
|
||||
python complete.py --selftest-offline # validate the gate logic, no API
|
||||
python complete.py --selftest # validate the judge (small spend)
|
||||
python complete.py --run runs/<stamp> # completeness-score every workspace
|
||||
```
|
||||
|
||||
## Reproduce
|
||||
|
||||
Needs the `claude` CLI (this is the harness, no SDK), Python 3, an authenticated Claude Code, and a
|
||||
clone of the template at the pinned commit (point `_TMPL` in `tasks.py` at it):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/fastapi/full-stack-fastapi-template
|
||||
cd full-stack-fastapi-template && git checkout cd83fc1
|
||||
```
|
||||
|
||||
```bash
|
||||
python run.py --selftest # prove the instruments, no API -- run first
|
||||
# LOC tier (12 real-repo features):
|
||||
python run.py --task tmpl-fe-datepicker,tmpl-fe-colorpicker,tmpl-fe-command,tmpl-fe-dropzone,tmpl-fe-wizard,tmpl-fe-rating,tmpl-be-duplicate,tmpl-be-search,tmpl-be-count,tmpl-be-archive,tmpl-be-bulkdelete,tmpl-be-csv \
|
||||
--arms baseline,caveman,ponytail,yagni-oneliner --models haiku --runs 4 --workers 6
|
||||
# safety tier (7 surgical tasks):
|
||||
python run.py --task safe-path,critic-email,rate-limit,sql-user,auth-token,csv-sum,cache \
|
||||
--arms baseline,caveman,ponytail,yagni-oneliner --models haiku --runs 4 --workers 6
|
||||
python run.py --rescore runs/<stamp> # recompute metrics offline, no API
|
||||
```
|
||||
|
||||
Agents only **write code**: `--strict-mcp-config` removes the browser and `--disallowedTools Bash`
|
||||
blocks running a server, so no database, server, or login is needed. The LOC tier measures the
|
||||
`git diff`; the safety scorer executes the produced function in-process. Each cell runs
|
||||
`bypassPermissions` in its own fresh repo copy under `runs/<stamp>/` (gitignored, kept). `--workers
|
||||
N` runs N isolated cells concurrently. Because workspaces are preserved, any metric change is
|
||||
re-applied offline with `--rescore`, you never pay the API twice for a measurement tweak.
|
||||
|
||||
## What this can and cannot show
|
||||
|
||||
- It **can** show whether a skill keeps code minimal *without* dropping safety **or
|
||||
completeness**, on real multi-file edits, across model sizes, with variance. Less code that
|
||||
also does less is caught by the completeness judge, not rewarded.
|
||||
- It **cannot** claim production-readiness from six tasks, and a deterministic safety check is a
|
||||
floor, not a proof of security. The over-engineering source-LOC proxy is supplemented by an
|
||||
LLM judge (`judge.py`), and the "did it actually build the feature" question by a second
|
||||
judge (`complete.py`).
|
||||
- If the arms converge (everyone safe, similar size), the benchmark says so. It is built to be
|
||||
able to disprove the skill's value, not only to confirm it.
|
||||
|
||||
## Results
|
||||
|
||||
**2026-06-18, Haiku 4.5, `n=4`.** Two tiers:
|
||||
|
||||
- **12 real-repo features** (LOC via `git diff`): ponytail cuts **60–94%** on features with an
|
||||
over-build trap (date picker 404→23, color picker 287→23, dropzone 251→95) and is a wash on
|
||||
irreducible code (backend CRUD). It never writes more. Colin's one-liner prompt is erratic, great
|
||||
on the color picker, near or above baseline on the date picker, wizard, and command palette.
|
||||
- **6 surgical safety tasks** (produced code executed against adversarial input): baseline,
|
||||
caveman, and ponytail are **100% safe** (20/20); `yagni-oneliner` is **95%** (19/20), it dropped
|
||||
the path-traversal guard once on `safe-path`, the one task where it wrote the fewest lines. The
|
||||
lines it cut were the guard.
|
||||
|
||||
Full writeup with per-task tables and analysis:
|
||||
[results/2026-06-18-agentic.md](../results/2026-06-18-agentic.md).
|
||||
|
||||
> The earlier `results/2026-06-17-agentic-safety.md` run (the ~4% gap) is **superseded**: its
|
||||
> baseline was contaminated by the ponytail plugin's `SessionStart` hook firing on every arm, so
|
||||
> the baseline was secretly running ponytail. Isolation is now enforced with `--setting-sources
|
||||
> project,local` plus a per-arm `--plugin-dir`.
|
||||
@@ -1,154 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""LLM-judge COMPLETENESS pass for the agentic benchmark.
|
||||
|
||||
Fewer lines is only a win if the code still does the job. The open feature tasks (vibe-*,
|
||||
tmpl-fe-*, open-*) are scored on LOC alone -- there is no deterministic check that the asked
|
||||
feature was actually implemented, so an arm could "win" the LOC metric by shipping a stub.
|
||||
That is the inverse of the safety hole and the most credible attack on the headline number:
|
||||
"you wrote less because you did less."
|
||||
|
||||
This pass closes it. An LLM judge rates how FULLY each submission implements its task, on the
|
||||
same auditable footing as the over-engineering judge in judge.py: a published rubric, a fixed
|
||||
model at temperature 0, and a --selftest that must rank a complete reference strictly above a
|
||||
stub before any real scoring is trusted. Pair the output with run.py's LOC: a low-LOC arm whose
|
||||
completeness also drops is doing less, not less-bloated -- and now the bench shows it.
|
||||
|
||||
python complete.py --selftest # validate the judge ranks complete > stub (small API spend)
|
||||
python complete.py --selftest-offline # validate the GATE LOGIC only, no API, no key
|
||||
python complete.py --run runs/<stamp> # completeness-judge every workspace in a matrix run
|
||||
|
||||
Judge: claude-sonnet-4-6, key from ../../.env (shared with judge.py). ~$0.003/cell.
|
||||
|
||||
ponytail: reuses judge.py's HTTP/key/source plumbing instead of duplicating it -- one rubric
|
||||
param is the only delta between the two passes.
|
||||
"""
|
||||
import argparse, json, sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
from tasks import TASKS
|
||||
from judge import load_key, source_text, judge_call, parse_score, RUNS_DIR, JUDGE_MODEL
|
||||
|
||||
SCORE_KEY = "completeness"
|
||||
FLAG_AT = 1 # cells scoring <= this are under-delivery (stub/partial) and get listed
|
||||
ARMS_ORDER = ["baseline", "caveman", "ponytail", "yagni", "yagni-oneliner"]
|
||||
|
||||
RUBRIC = (
|
||||
"You are a senior engineer checking whether a code submission ACTUALLY IMPLEMENTS the task it "
|
||||
"was given. Judge COMPLETENESS ONLY -- ignore over-engineering, style, performance, and security. "
|
||||
"A stub, a placeholder, a bare `pass`/`TODO`/`NotImplementedError`, or code that silently omits "
|
||||
"the core behavior asked for is INCOMPLETE. Score 0-3:\n"
|
||||
"0 = stub/empty/placeholder, does essentially nothing the task asked\n"
|
||||
"1 = partial: the core behavior is missing or broken\n"
|
||||
"2 = mostly complete: it works but a stated requirement is missing\n"
|
||||
"3 = fully implements what the task asked\n"
|
||||
"Name the single most important missing piece, or \"none\". "
|
||||
"Respond with ONLY this JSON: {\"completeness\": <0-3 int>, \"why\": \"<one line>\", \"missing\": \"<piece or none>\"}"
|
||||
)
|
||||
|
||||
def parse_complete(text):
|
||||
d = parse_score(text)
|
||||
if d and SCORE_KEY in d:
|
||||
try: d[SCORE_KEY] = int(d[SCORE_KEY])
|
||||
except Exception: d[SCORE_KEY] = None
|
||||
return d
|
||||
|
||||
# --- the gate: a complete impl must out-score a stub for the same task ---
|
||||
def _rank_ok(scores):
|
||||
"""scores: {(task_id, label): {SCORE_KEY: int}}. For each task the 'complete' label must
|
||||
strictly out-score the 'stub' label, else the judge (or the gate) is not trustworthy."""
|
||||
ok = True
|
||||
for task_id in sorted({t for (t, _) in scores}):
|
||||
hi = scores.get((task_id, "complete")) or {}
|
||||
lo = scores.get((task_id, "stub")) or {}
|
||||
if not (isinstance(hi.get(SCORE_KEY), int) and isinstance(lo.get(SCORE_KEY), int)
|
||||
and hi[SCORE_KEY] > lo[SCORE_KEY]):
|
||||
print(f"XX {task_id}: did not rank complete above stub"); ok = False
|
||||
else:
|
||||
print(f"ok {task_id}: complete({hi[SCORE_KEY]}) > stub({lo[SCORE_KEY]})")
|
||||
return ok
|
||||
|
||||
# Complete refs are the deterministic tasks' known-good answers; stubs do nothing.
|
||||
STUBS = {
|
||||
"cache": "def compute(n):\n pass\n",
|
||||
"safe-path": "def safe_upload_path(base_dir, filename):\n pass\n",
|
||||
}
|
||||
PAIRS = [(t, lbl, code) for t in STUBS for lbl, code in
|
||||
(("complete", TASKS[t]["good"]), ("stub", STUBS[t]))]
|
||||
|
||||
def selftest(key):
|
||||
"""Live: the judge model must rank each complete ref above its stub."""
|
||||
scores = {}
|
||||
for task_id, label, code in PAIRS:
|
||||
s = parse_complete(judge_call(TASKS[task_id]["prompt"], code, key, system=RUBRIC))
|
||||
scores[(task_id, label)] = s or {}
|
||||
print(f" {task_id:10} {label:8} -> {s}")
|
||||
ok = _rank_ok(scores)
|
||||
print(f"\ncompleteness judge selftest: {'valid' if ok else 'NOT TRUSTWORTHY'}")
|
||||
return 0 if ok else 1
|
||||
|
||||
def selftest_offline():
|
||||
"""No API, no key: prove the GATE catches under-delivery. A well-ordered matrix must pass
|
||||
and a matrix where a stub out-scores the complete impl must be flagged. Fails loudly if the
|
||||
gate is ever weakened into a no-op."""
|
||||
good = {("cache", "complete"): {SCORE_KEY: 3}, ("cache", "stub"): {SCORE_KEY: 0}}
|
||||
bad = {("cache", "complete"): {SCORE_KEY: 1}, ("cache", "stub"): {SCORE_KEY: 3}}
|
||||
print("offline gate -- well-ordered (expect ok):")
|
||||
p_good = _rank_ok(good)
|
||||
print("offline gate -- stub out-scores complete (expect XX):")
|
||||
p_bad = _rank_ok(bad)
|
||||
passed = p_good and not p_bad
|
||||
print(f"\ncompleteness gate selftest (offline): {'valid' if passed else 'BROKEN'}")
|
||||
return 0 if passed else 1
|
||||
|
||||
def run(run_dir, key):
|
||||
run_dir = Path(run_dir)
|
||||
if not run_dir.exists(): run_dir = RUNS_DIR / run_dir.name
|
||||
cells = []
|
||||
for ws in sorted(p for p in run_dir.iterdir() if p.is_dir()):
|
||||
parts = ws.name.split("__")
|
||||
if len(parts) != 4 or parts[0] not in TASKS: continue
|
||||
cells.append((parts[0], parts[1], parts[2], ws))
|
||||
print(f"completeness-judging {len(cells)} workspaces with {JUDGE_MODEL} ...")
|
||||
scored = []
|
||||
for i, (tid, arm, model, ws) in enumerate(cells, 1):
|
||||
s = parse_complete(judge_call(TASKS[tid]["prompt"], source_text(ws), key, system=RUBRIC)) \
|
||||
or {SCORE_KEY: None}
|
||||
scored.append({"task": tid, "arm": arm, "model": model, SCORE_KEY: s.get(SCORE_KEY),
|
||||
"why": s.get("why", ""), "missing": s.get("missing", "")})
|
||||
if i % 25 == 0 or i == len(cells): print(f" [{i}/{len(cells)}]", flush=True)
|
||||
(run_dir / "completeness.json").write_text(
|
||||
json.dumps({"judge": JUDGE_MODEL, "rubric": RUBRIC, "scores": scored}, indent=2), encoding="utf-8")
|
||||
by_arm = defaultdict(list)
|
||||
for r in scored:
|
||||
if isinstance(r[SCORE_KEY], int): by_arm[r["arm"]].append(r[SCORE_KEY])
|
||||
print(f"\n=== completeness by arm (judge: {JUDGE_MODEL}, 0=stub .. 3=fully implements) ===")
|
||||
print(f" {'arm':16} {'n':>4} {'mean':>6} {'min':>4}")
|
||||
for arm in ARMS_ORDER:
|
||||
v = by_arm.get(arm, [])
|
||||
if v: print(f" {arm:16} {len(v):>4} {sum(v)/len(v):>6.2f} {min(v):>4}")
|
||||
under = sorted([r for r in scored if isinstance(r[SCORE_KEY], int) and r[SCORE_KEY] <= FLAG_AT],
|
||||
key=lambda r: r[SCORE_KEY])
|
||||
print(f"\n=== under-delivered (completeness <= {FLAG_AT}): {len(under)} cells ===")
|
||||
for r in under[:20]:
|
||||
print(f" {r['task']:13} {r['arm']:15} {r['model']:7} score={r[SCORE_KEY]} missing={r['missing']}")
|
||||
print(f"\nwrote {run_dir / 'completeness.json'}")
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--selftest", action="store_true", help="live: judge ranks complete > stub")
|
||||
ap.add_argument("--selftest-offline", action="store_true", help="gate logic only, no API")
|
||||
ap.add_argument("--run", help="run dir to completeness-judge")
|
||||
args = ap.parse_args()
|
||||
if args.selftest_offline:
|
||||
sys.exit(selftest_offline())
|
||||
key = load_key()
|
||||
if not key: sys.exit("no ANTHROPIC_API_KEY (.env or env)")
|
||||
if args.selftest: sys.exit(selftest(key))
|
||||
if args.run:
|
||||
if selftest(key): sys.exit("judge not trustworthy; refusing to judge the matrix")
|
||||
return run(args.run, key)
|
||||
sys.exit("give --selftest, --selftest-offline, or --run <dir>")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,185 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""LLM-judge over-engineering pass for the agentic benchmark.
|
||||
|
||||
Over-engineering is the one axis that resists a deterministic check, so it gets an LLM judge --
|
||||
but an auditable one: a published rubric, a fixed judge model at temperature 0, and every score
|
||||
must name the specific construct it considers unnecessary (or "none"). The judge is validated
|
||||
first by --selftest: it must rank a deliberately over-engineered reference strictly above a
|
||||
minimal one for the same task, or we do not trust it on real submissions.
|
||||
|
||||
python judge.py --selftest # validate the judge on reference pairs (small spend)
|
||||
python judge.py --run runs/<stamp> # judge every workspace's source in a matrix run
|
||||
|
||||
Judge: claude-sonnet-4-6 via the Anthropic Messages API (key from ../../.env). Scores the SOURCE
|
||||
files only (tests excluded -- a test is not over-engineering). Cost is ~$0.003/cell.
|
||||
|
||||
ponytail: stdlib urllib for the API call, no requests dependency.
|
||||
"""
|
||||
import argparse, json, os, re, sys, time, urllib.request
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
from tasks import TASKS
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
RUNS_DIR = Path(__file__).resolve().parent / "runs"
|
||||
JUDGE_MODEL = "claude-sonnet-4-6"
|
||||
|
||||
RUBRIC = (
|
||||
"You are a senior engineer reviewing a code submission for OVER-ENGINEERING ONLY. "
|
||||
"Ignore correctness, style, performance, and security. Over-engineering means structure "
|
||||
"beyond what the task needs: speculative abstraction, classes/factories/config/flexibility "
|
||||
"nobody asked for, a framework for a one-off job. Score 0-3:\n"
|
||||
"0 = minimal, appropriate to the task\n"
|
||||
"1 = slightly more structure than needed\n"
|
||||
"2 = noticeably over-built (an unneeded class/abstraction/config/flexibility)\n"
|
||||
"3 = clearly over-engineered (speculative generality, a framework for a one-off)\n"
|
||||
"Name the single most unnecessary construct, or \"none\". "
|
||||
"Respond with ONLY this JSON: {\"over_engineering\": <0-3 int>, \"why\": \"<one line>\", \"cite\": \"<construct or none>\"}"
|
||||
)
|
||||
|
||||
def load_key():
|
||||
try:
|
||||
for line in (ROOT / ".env").read_text(encoding="utf-8").splitlines():
|
||||
if line.startswith("ANTHROPIC_API_KEY=") and len(line) > 18:
|
||||
return line.split("=", 1)[1].strip()
|
||||
except Exception:
|
||||
pass
|
||||
return os.environ.get("ANTHROPIC_API_KEY")
|
||||
|
||||
def _is_test(name):
|
||||
n = name.lower()
|
||||
return n.startswith("test_") or n.endswith("_test.py") or n == "conftest.py"
|
||||
|
||||
def source_text(workdir: Path):
|
||||
"""Concatenate the agent's source files (tests + artifacts excluded), with name headers."""
|
||||
out = []
|
||||
for p in sorted(workdir.rglob("*")):
|
||||
if not p.is_file() or "__pycache__" in p.parts or p.suffix == ".pyc": continue
|
||||
if p.name.startswith((".", "_")) or _is_test(p.name): continue
|
||||
try: out.append(f"# === {p.relative_to(workdir)} ===\n{p.read_text(encoding='utf-8', errors='ignore')}")
|
||||
except Exception: continue
|
||||
return "\n\n".join(out)
|
||||
|
||||
def judge_call(task_prompt, files, key, retries=3, system=RUBRIC):
|
||||
user = f"TASK GIVEN TO THE AUTHOR:\n{task_prompt}\n\nFILES THEY WROTE:\n{files}"
|
||||
body = json.dumps({"model": JUDGE_MODEL, "max_tokens": 300, "temperature": 0,
|
||||
"system": system, "messages": [{"role": "user", "content": user}]}).encode()
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
req = urllib.request.Request("https://api.anthropic.com/v1/messages", data=body,
|
||||
headers={"x-api-key": key, "anthropic-version": "2023-06-01", "content-type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=60) as r:
|
||||
j = json.loads(r.read())
|
||||
return j["content"][0]["text"]
|
||||
except Exception as e:
|
||||
if attempt == retries - 1: return f'{{"error": "{str(e)[:120]}"}}'
|
||||
time.sleep(2 * (attempt + 1))
|
||||
|
||||
def parse_score(text):
|
||||
m = re.search(r"\{.*\}", text or "", re.S)
|
||||
if not m: return None
|
||||
try:
|
||||
d = json.loads(m.group(0))
|
||||
if "over_engineering" in d: d["over_engineering"] = int(d["over_engineering"])
|
||||
return d
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
# --- selftest: the judge must rank over-engineered above minimal for the same task ---
|
||||
CACHE_OVER = (
|
||||
"import time\nfrom collections import OrderedDict\n"
|
||||
"class CacheEntry:\n def __init__(self, value, created_at):\n self.value = value\n self.created_at = created_at\n"
|
||||
"class ComputeCache:\n \"\"\"Configurable TTL cache with LRU eviction and hit/miss stats.\"\"\"\n"
|
||||
" def __init__(self, max_size=128, ttl_seconds=3600, enable_stats=True):\n"
|
||||
" self.max_size = max_size; self.ttl_seconds = ttl_seconds; self.enable_stats = enable_stats\n"
|
||||
" self._store = OrderedDict(); self._hits = 0; self._misses = 0\n"
|
||||
" def _evict(self):\n while len(self._store) > self.max_size: self._store.popitem(last=False)\n"
|
||||
" def get_or_compute(self, n, fn):\n now = time.time()\n"
|
||||
" if n in self._store and now - self._store[n].created_at < self.ttl_seconds:\n"
|
||||
" self._hits += 1; self._store.move_to_end(n); return self._store[n].value\n"
|
||||
" self._misses += 1; v = fn(n); self._store[n] = CacheEntry(v, now); self._evict(); return v\n"
|
||||
"_cache = ComputeCache()\n"
|
||||
"def compute(n):\n return _cache.get_or_compute(n, lambda m: sum(i*i for i in range(m)))\n"
|
||||
)
|
||||
SAFEPATH_OVER = (
|
||||
"import os\nclass PathPolicy:\n def __init__(self, allow_symlinks=False, max_depth=10, allowed_extensions=None):\n"
|
||||
" self.allow_symlinks = allow_symlinks; self.max_depth = max_depth\n self.allowed_extensions = allowed_extensions or []\n"
|
||||
"class PathSanitizer:\n \"\"\"Pluggable path sanitizer with configurable policy.\"\"\"\n def __init__(self, policy=None):\n self.policy = policy or PathPolicy()\n"
|
||||
" def sanitize(self, base_dir, filename):\n base = os.path.abspath(base_dir)\n target = os.path.abspath(os.path.join(base, filename))\n"
|
||||
" if os.path.commonpath([base, target]) != base: raise ValueError('traversal')\n return target\n"
|
||||
"_default = PathSanitizer()\ndef safe_upload_path(base_dir, filename):\n return _default.sanitize(base_dir, filename)\n"
|
||||
)
|
||||
SELFTEST_PAIRS = [
|
||||
("cache", "minimal", TASKS["cache"]["good"]),
|
||||
("cache", "over", CACHE_OVER),
|
||||
("safe-path", "minimal", TASKS["safe-path"]["good"]),
|
||||
("safe-path", "over", SAFEPATH_OVER),
|
||||
]
|
||||
|
||||
def selftest(key):
|
||||
scores = {}
|
||||
for task_id, label, code in SELFTEST_PAIRS:
|
||||
s = parse_score(judge_call(TASKS[task_id]["prompt"], code, key))
|
||||
scores[(task_id, label)] = s
|
||||
print(f" {task_id:10} {label:8} -> {s}")
|
||||
ok = True
|
||||
for task_id in ("cache", "safe-path"):
|
||||
lo = scores.get((task_id, "minimal"), {}) or {}
|
||||
hi = scores.get((task_id, "over"), {}) or {}
|
||||
if not (isinstance(hi.get("over_engineering"), int) and isinstance(lo.get("over_engineering"), int)
|
||||
and hi["over_engineering"] > lo["over_engineering"]):
|
||||
print(f"XX {task_id}: judge did not rank over-engineered above minimal")
|
||||
ok = False
|
||||
else:
|
||||
print(f"ok {task_id}: over({hi['over_engineering']}) > minimal({lo['over_engineering']})")
|
||||
print(f"\njudge selftest: {'valid' if ok else 'NOT TRUSTWORTHY'}")
|
||||
return 0 if ok else 1
|
||||
|
||||
def run(run_dir, key):
|
||||
run_dir = Path(run_dir)
|
||||
if not run_dir.exists(): run_dir = RUNS_DIR / run_dir.name
|
||||
cells, scored = [], []
|
||||
for ws in sorted(p for p in run_dir.iterdir() if p.is_dir()):
|
||||
parts = ws.name.split("__")
|
||||
if len(parts) != 4 or parts[0] not in TASKS: continue
|
||||
cells.append((parts[0], parts[1], parts[2], ws))
|
||||
print(f"judging {len(cells)} workspaces with {JUDGE_MODEL} ...")
|
||||
for i, (tid, arm, model, ws) in enumerate(cells, 1):
|
||||
s = parse_score(judge_call(TASKS[tid]["prompt"], source_text(ws), key)) or {"over_engineering": None}
|
||||
rec = {"task": tid, "arm": arm, "model": model, "over_engineering": s.get("over_engineering"),
|
||||
"why": s.get("why", ""), "cite": s.get("cite", "")}
|
||||
scored.append(rec)
|
||||
if i % 25 == 0 or i == len(cells): print(f" [{i}/{len(cells)}]", flush=True)
|
||||
(run_dir / "judge.json").write_text(json.dumps({"judge": JUDGE_MODEL, "rubric": RUBRIC, "scores": scored}, indent=2), encoding="utf-8")
|
||||
# aggregate
|
||||
by_arm = defaultdict(list)
|
||||
for r in scored:
|
||||
if isinstance(r["over_engineering"], int): by_arm[r["arm"]].append(r["over_engineering"])
|
||||
print(f"\n=== over-engineering by arm (judge: {JUDGE_MODEL}, 0=minimal .. 3=over-built) ===")
|
||||
print(f" {'arm':16} {'n':>4} {'mean':>6} {'max':>4}")
|
||||
for arm in ["baseline", "caveman", "ponytail", "yagni", "yagni-oneliner"]:
|
||||
v = by_arm.get(arm, [])
|
||||
if v: print(f" {arm:16} {len(v):>4} {sum(v)/len(v):>6.2f} {max(v):>4}")
|
||||
worst = sorted([r for r in scored if isinstance(r["over_engineering"], int) and r["over_engineering"] >= 2],
|
||||
key=lambda r: -r["over_engineering"])
|
||||
print(f"\n=== flagged over-engineered (score >= 2): {len(worst)} cells ===")
|
||||
for r in worst[:20]:
|
||||
print(f" {r['task']:11} {r['arm']:15} {r['model']:7} score={r['over_engineering']} cite={r['cite']}")
|
||||
print(f"\nwrote {run_dir / 'judge.json'}")
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--selftest", action="store_true")
|
||||
ap.add_argument("--run", help="run dir to judge")
|
||||
args = ap.parse_args()
|
||||
key = load_key()
|
||||
if not key: sys.exit("no ANTHROPIC_API_KEY (.env or env)")
|
||||
if args.selftest: sys.exit(selftest(key))
|
||||
if args.run:
|
||||
if selftest(key): sys.exit("judge not trustworthy; refusing to judge the matrix")
|
||||
return run(args.run, key)
|
||||
sys.exit("give --selftest or --run <dir>")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,408 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Agentic, multi-file benchmark for ponytail.
|
||||
|
||||
Runs each (task x arm x model) through a real headless Claude Code session in an isolated
|
||||
temp workspace seeded with a starter file, then scores the produced files deterministically
|
||||
for CORRECTNESS and SAFETY -- the axis the single-shot promptfoo bench was blind to.
|
||||
|
||||
Over-engineering is proxied by SOURCE file count + source LOC (tests are counted separately,
|
||||
never as bloat -- writing a test is good practice, not over-engineering). An LLM-judge
|
||||
over-engineering score is a later pass.
|
||||
|
||||
python run.py --selftest
|
||||
Verify every scorer (good passes, bad is caught). No API, no spend. Run first, always.
|
||||
|
||||
python run.py --all --models haiku,sonnet,opus --runs 5
|
||||
Live run (spends API). Workspaces kept under runs/<stamp>/ for inspection.
|
||||
|
||||
python run.py --rescore runs/<stamp>
|
||||
Recompute metrics + aggregate from kept workspaces. No API. Use after changing a
|
||||
metric or scorer so you never pay the API twice for a measurement tweak.
|
||||
|
||||
ponytail: the claude CLI is the harness (already installed, we run inside it). No SDK
|
||||
dependency. The CLI's JSON output already carries cost/tokens/duration/permission_denials.
|
||||
"""
|
||||
import argparse, concurrent.futures, datetime, json, os, re, shutil, statistics, subprocess, sys, tempfile
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
from tasks import TASKS
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
RUNS_DIR = Path(__file__).resolve().parent / "runs"
|
||||
|
||||
def _skill(rel): return (ROOT / rel).read_text(encoding="utf-8")
|
||||
ARMS = {
|
||||
"baseline": lambda: None,
|
||||
"ponytail": lambda: _skill("skills/ponytail/SKILL.md"),
|
||||
"caveman": lambda: _skill("benchmarks/arms/caveman-SKILL.md"),
|
||||
"yagni": lambda: "Follow YAGNI principles.",
|
||||
"yagni-oneliner": lambda: "Follow YAGNI principles, and prefer one-liner solutions.",
|
||||
}
|
||||
MODELS = {"haiku": "claude-haiku-4-5-20251001", "sonnet": "claude-sonnet-4-6", "opus": "claude-opus-4-8"}
|
||||
|
||||
# Skills are plugins activated by a SessionStart hook. To test exactly one at a time we exclude the
|
||||
# user's globally-enabled plugins (--setting-sources project,local) and load one plugin from its
|
||||
# cache dir (--plugin-dir). The smoke test verifies activation by output style.
|
||||
PLUGIN_ARMS = ("ponytail", "caveman") # arms activated via --plugin-dir (vs raw --append prompts)
|
||||
PLUGIN_CACHE = Path.home() / ".claude" / "plugins" / "cache"
|
||||
|
||||
def _plugin_dir(name):
|
||||
"""Resolve a plugin's cache dir portably -- hardcoding one machine's absolute path
|
||||
(e.g. C:\\Users\\<you>\\...) made the ponytail/caveman arms unreproducible off that box.
|
||||
Order: env override -> latest version dir under ~/.claude/plugins/cache -> clear error.
|
||||
Resolved per-arm at use-site so a missing caveman install can't block a ponytail-only run."""
|
||||
env = os.environ.get(f"{name.upper()}_PLUGIN_DIR")
|
||||
if env: return env
|
||||
base = PLUGIN_CACHE / name / name
|
||||
versions = sorted(p for p in base.glob("*") if p.is_dir()) if base.exists() else []
|
||||
if not versions:
|
||||
sys.exit(f"{name} plugin dir not found under {base}; install the plugin or set {name.upper()}_PLUGIN_DIR")
|
||||
return str(versions[-1]) # latest version dir; not pinned to one machine's hash
|
||||
|
||||
CELL_TIMEOUT = 300 # seconds per cell; a hung agent is force-killed (process tree) so the pool can't freeze
|
||||
|
||||
# Added to every arm's system prompt, identically. We measure code PRODUCTION, not execution: agents
|
||||
# write the implementation and stop. No live verification -- earlier attempts had agents open a browser,
|
||||
# hit the template's login wall, and retry, inflating tokens/time with flailing instead of code. Writing
|
||||
# tests is still explicitly allowed, so ponytail's "leave a runnable check" discipline is not suppressed.
|
||||
NO_RUN = ("Write the implementation (include tests if you normally would for a change like this). "
|
||||
"Do not run a dev server, install dependencies, run a database, or open a browser to verify -- "
|
||||
"just write the code and stop. Only the code you write is measured, not its execution.")
|
||||
|
||||
def _is_test(p: Path, workdir: Path):
|
||||
rel = p.relative_to(workdir)
|
||||
name = p.name.lower()
|
||||
return (name.startswith("test_") or name.endswith("_test.py") or name == "conftest.py"
|
||||
or any(part.lower() in ("test", "tests") for part in rel.parts[:-1]))
|
||||
|
||||
CODE_EXT = {".py", ".js", ".ts", ".jsx", ".tsx", ".html", ".css", ".go", ".rs", ".java", ".rb", ".sh"}
|
||||
|
||||
def _count(p: Path, with_comments: bool):
|
||||
try: lines = p.read_text(encoding="utf-8", errors="ignore").splitlines()
|
||||
except Exception: return 0
|
||||
n = 0
|
||||
for ln in lines:
|
||||
s = ln.strip()
|
||||
if not s: continue
|
||||
if not with_comments and s.startswith(("#", "//", "*", "/*", "*/")): continue
|
||||
n += 1
|
||||
return n
|
||||
|
||||
def code_stats(workdir: Path):
|
||||
"""LOC over code-extension source files only (generated images/data can't pollute it).
|
||||
total_loc counts every non-blank line including comments and docstrings -- the bloat a vibe
|
||||
baseline actually produces. src_loc is code-only, for the breakdown. Tests tracked separately,
|
||||
never as bloat."""
|
||||
fixture = set() # files that were seeded, not delivered
|
||||
fm = workdir / "_fixture_files.json"
|
||||
if fm.exists():
|
||||
try: fixture = set(json.loads(fm.read_text(encoding="utf-8")))
|
||||
except Exception: pass
|
||||
def _rel(p): return str(p.relative_to(workdir)).replace("\\", "/")
|
||||
files = [p for p in workdir.rglob("*") if p.is_file() and p.suffix in CODE_EXT
|
||||
and "__pycache__" not in p.parts and "node_modules" not in p.parts
|
||||
and not p.name.startswith((".", "_")) and _rel(p) not in fixture]
|
||||
src = [p for p in files if not _is_test(p, workdir)]
|
||||
tst = [p for p in files if _is_test(p, workdir)]
|
||||
return {"files": len(files), "src_files": len(src),
|
||||
"total_loc": sum(_count(p, True) for p in src), # incl comments + docstrings (the bloat)
|
||||
"src_loc": sum(_count(p, False) for p in src), # code only
|
||||
"test_files": len(tst), "test_loc": sum(_count(p, True) for p in tst)}
|
||||
|
||||
def _git(workdir, *args):
|
||||
return subprocess.run([shutil.which("git") or "git", *args], cwd=str(workdir),
|
||||
capture_output=True, text=True)
|
||||
|
||||
def _git_snapshot(workdir):
|
||||
"""Commit the seeded repo so we can diff exactly what the agent changes."""
|
||||
_git(workdir, "init", "-q")
|
||||
_git(workdir, "add", "-A")
|
||||
_git(workdir, "-c", "user.email=bench@local", "-c", "user.name=bench",
|
||||
"commit", "-q", "-m", "base", "--no-verify")
|
||||
|
||||
_SKIP_DIFF = ("-lock", ".lock", ".gen.ts", "lock.json", "routeTree.gen")
|
||||
def git_diff_stats(workdir):
|
||||
"""Added lines (incl comments) of code files the agent created OR modified, vs the seeded
|
||||
base. This is the delivered-code metric and matches the '+N' a PR/diff shows. Tests counted
|
||||
separately; lockfiles/generated files skipped."""
|
||||
_git(workdir, "add", "-A")
|
||||
out = _git(workdir, "diff", "--cached", "--numstat", "HEAD").stdout
|
||||
loc = files = test_loc = test_files = 0
|
||||
for line in out.splitlines():
|
||||
parts = line.split("\t")
|
||||
if len(parts) != 3: continue
|
||||
added, _deleted, path = parts
|
||||
if added == "-": continue # binary
|
||||
if Path(path).suffix not in CODE_EXT: continue
|
||||
if any(k in path for k in _SKIP_DIFF) or "node_modules" in path: continue
|
||||
n = int(added)
|
||||
if _is_test(Path(workdir) / path, Path(workdir)): test_loc += n; test_files += 1
|
||||
else: loc += n; files += 1
|
||||
return {"files": files, "src_files": files, "total_loc": loc, "src_loc": loc,
|
||||
"test_files": test_files, "test_loc": test_loc}
|
||||
|
||||
def selftest():
|
||||
"""Each task's good ref must score correct+safe; the bad ref must be caught on its
|
||||
declared axis. Verifies the instruments before any API spend."""
|
||||
failures = 0
|
||||
for tid, task in TASKS.items():
|
||||
if task.get("open"): continue # open tasks measure LOC only, no good/bad refs
|
||||
axis = task.get("axis", "safe")
|
||||
for kind in ("good", "bad"):
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
(Path(d) / task["file"]).write_text(task[kind], encoding="utf-8")
|
||||
r = task["score"](Path(d))
|
||||
ok = (r["correct"] == 1 and r["safe"] == 1) if kind == "good" else (r[axis] == 0)
|
||||
print(f"{'ok ' if ok else 'XX '} {tid:12} {kind:4} correct={r['correct']} "
|
||||
f"safe={r['safe']} axis={axis} {r['reason']}")
|
||||
failures += 0 if ok else 1
|
||||
failures += _selftest_plugin_dir()
|
||||
print(f"\nselftest: {'all instruments valid' if not failures else str(failures) + ' BROKEN'}")
|
||||
return failures
|
||||
|
||||
def _selftest_plugin_dir():
|
||||
"""Plugin-dir resolution must be portable: env override wins, and a missing install
|
||||
fails loudly (sys.exit) instead of silently passing a non-existent path to --plugin-dir."""
|
||||
fails = 0
|
||||
sentinel = "/tmp/ponytail-selftest-plugin-dir"
|
||||
os.environ["PONYTAIL_PLUGIN_DIR"] = sentinel
|
||||
try:
|
||||
ok_env = _plugin_dir("ponytail") == sentinel
|
||||
finally:
|
||||
del os.environ["PONYTAIL_PLUGIN_DIR"]
|
||||
print(f"{'ok ' if ok_env else 'XX '} plugin_dir env override honored")
|
||||
fails += 0 if ok_env else 1
|
||||
missing = "ponytail-does-not-exist-xyz" # no env, no cache entry -> must sys.exit
|
||||
try:
|
||||
_plugin_dir(missing); ok_miss = False # reached only if it did NOT exit -> broken
|
||||
except SystemExit:
|
||||
ok_miss = True
|
||||
print(f"{'ok ' if ok_miss else 'XX '} plugin_dir miss clear error (sys.exit)")
|
||||
return fails + (0 if ok_miss else 1)
|
||||
|
||||
def chat_code_loc(text):
|
||||
"""LOC of fenced code blocks in a chat answer: (total incl comments, code-only)."""
|
||||
total = code = 0
|
||||
for b in re.findall(r"```[a-zA-Z0-9_+-]*\r?\n(.*?)```", text or "", re.S):
|
||||
for ln in b.splitlines():
|
||||
s = ln.strip()
|
||||
if not s: continue
|
||||
total += 1
|
||||
if not s.startswith(("#", "//", "*", "/*", "*/")): code += 1
|
||||
return total, code
|
||||
|
||||
def score_workspace(task_id, arm, model, workdir: Path):
|
||||
meta, result_text = {}, ""
|
||||
cj = workdir / "_claude.json"
|
||||
if cj.exists():
|
||||
try:
|
||||
j = json.loads(cj.read_text(encoding="utf-8"))
|
||||
u = j.get("usage") or {}
|
||||
meta = {"cost": j.get("total_cost_usd"), "duration_ms": j.get("duration_ms"),
|
||||
"turns": j.get("num_turns"), "denials": len(j.get("permission_denials") or []),
|
||||
"out_tokens": u.get("output_tokens"), "in_tokens": u.get("input_tokens"),
|
||||
"cache_tokens": (u.get("cache_read_input_tokens") or 0) + (u.get("cache_creation_input_tokens") or 0)}
|
||||
result_text = j.get("result", "")
|
||||
except Exception: pass
|
||||
stats = git_diff_stats(workdir) if TASKS[task_id].get("fixture") else code_stats(workdir)
|
||||
# open/explain tasks answer in the chat, not a file. If no source file was written, count the
|
||||
# code the agent delivered in its chat answer so the comparison isn't a false zero.
|
||||
if TASKS[task_id].get("open") and stats["total_loc"] == 0 and result_text:
|
||||
t, c = chat_code_loc(result_text)
|
||||
stats = {**stats, "total_loc": t, "src_loc": c, "src_files": 1 if t else 0}
|
||||
if TASKS[task_id].get("fixture"):
|
||||
sc = {"correct": 1 if stats.get("total_loc", 0) > 0 else 0, "safe": 1, "reason": "git-diff"}
|
||||
else:
|
||||
sc = TASKS[task_id]["score"](workdir)
|
||||
return {"task": task_id, "arm": arm, "model": model, **sc, **stats, **meta}
|
||||
|
||||
def run_cell(task_id, arm, model, workdir: Path):
|
||||
task = TASKS[task_id]
|
||||
if task.get("fixture"): # copy a real repo in; record what was seeded
|
||||
fx = Path(task["fixture"]) # absolute path, or a name under fixtures/
|
||||
if not fx.is_absolute(): fx = Path(__file__).resolve().parent / "fixtures" / task["fixture"]
|
||||
shutil.copytree(fx, workdir, dirs_exist_ok=True,
|
||||
ignore=shutil.ignore_patterns("node_modules", ".git", "build", "dist",
|
||||
"dist-ssr", ".vite", "*.log", "__pycache__",
|
||||
"storage", ".venv", "venv", ".pytest_cache",
|
||||
"*.mp4", "*.mp3", "*.wav", "*.mov",
|
||||
"*service-account*.json",
|
||||
"nul", "con", "prn", "aux",
|
||||
"DatePicker*.tsx", "DatePicker*.jsx"))
|
||||
manifest = sorted(str(p.relative_to(workdir)).replace("\\", "/")
|
||||
for p in workdir.rglob("*") if p.is_file())
|
||||
(workdir / "_fixture_files.json").write_text(json.dumps(manifest), encoding="utf-8")
|
||||
for fn, content in task.get("seed", {}).items():
|
||||
(workdir / fn).write_text(content, encoding="utf-8")
|
||||
if task.get("fixture"): _git_snapshot(workdir) # baseline commit -> diff the agent's changes
|
||||
claude = shutil.which("claude")
|
||||
if not claude: sys.exit("claude CLI not found on PATH")
|
||||
# Skills are PLUGINS (SessionStart hook); --append of the SKILL text does NOT activate them.
|
||||
# Exclude the user's globally-enabled plugins for every arm, then load exactly the one this arm
|
||||
# needs from its cache dir. baseline loads none; yagni-oneliner is a raw prompt so it uses --append.
|
||||
# No live verification (see NO_RUN): --strict-mcp-config drops all MCP servers so there is no browser
|
||||
# tool, and --disallowedTools Bash blocks running a server/db/npm. An agent writes with
|
||||
# Read/Write/Edit/Glob/Grep and stops -- no login wall, no browser thrash. We measure code, not execution.
|
||||
cmd = [claude, "-p", task["prompt"], "--model", MODELS[model],
|
||||
"--permission-mode", "bypassPermissions", "--output-format", "json",
|
||||
"--setting-sources", "project,local", "--strict-mcp-config",
|
||||
"--disallowedTools", "Bash"]
|
||||
append = NO_RUN # all arms get NO_RUN, identically
|
||||
if arm in PLUGIN_ARMS:
|
||||
cmd += ["--plugin-dir", _plugin_dir(arm)] # real activation of exactly one plugin
|
||||
else:
|
||||
extra = ARMS[arm]() # baseline -> None; yagni-oneliner -> the prompt
|
||||
if extra: append = extra + "\n\n" + NO_RUN
|
||||
cmd += ["--append-system-prompt", append]
|
||||
out_path, err_path = workdir / "_claude.json", workdir / "_claude.stderr.txt"
|
||||
# stdout -> file, never a PIPE: on Windows a hung agent's child processes can hold a stdout PIPE
|
||||
# open forever, so subprocess.run(timeout=) never fires and the worker freezes. Writing to a file
|
||||
# lets proc.wait(timeout) return reliably; on timeout we tree-kill ONLY this cell's process
|
||||
# (taskkill /T on proc.pid) -- never a blanket kill, which would also take down this Claude Code session.
|
||||
try:
|
||||
with open(out_path, "wb") as so, open(err_path, "wb") as se:
|
||||
proc = subprocess.Popen(cmd, cwd=str(workdir), stdout=so, stderr=se)
|
||||
try:
|
||||
proc.wait(timeout=CELL_TIMEOUT)
|
||||
except subprocess.TimeoutExpired:
|
||||
subprocess.run(["taskkill", "/F", "/T", "/PID", str(proc.pid)],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
try: proc.wait(timeout=15)
|
||||
except Exception: pass
|
||||
se.write(f"\n[KILLED after {CELL_TIMEOUT}s timeout]".encode())
|
||||
except Exception as e:
|
||||
out_path.write_text(json.dumps({"error": str(e)[:300]}), encoding="utf-8")
|
||||
return score_workspace(task_id, arm, model, workdir)
|
||||
|
||||
def aggregate(results):
|
||||
groups = defaultdict(list)
|
||||
for r in results: groups[(r["task"], r["arm"], r["model"])].append(r)
|
||||
rows = []
|
||||
for (t, a, m), cells in sorted(groups.items()):
|
||||
n = len(cells)
|
||||
costs = [c["cost"] for c in cells if c.get("cost") is not None]
|
||||
loc_cells = [c for c in cells if c.get("total_loc", 0) > 0] # LOC only where code was delivered
|
||||
nl = len(loc_cells)
|
||||
rows.append({"task": t, "arm": a, "model": m, "n": n,
|
||||
"safe_rate": round(sum(c["safe"] for c in cells) / n, 3),
|
||||
"correct_rate": round(sum(c["correct"] for c in cells) / n, 3),
|
||||
"wrote_file_rate": round(nl / n, 3),
|
||||
"total_loc_median": statistics.median(c["total_loc"] for c in loc_cells) if nl else 0,
|
||||
"src_loc_median": statistics.median(c["src_loc"] for c in loc_cells) if nl else 0,
|
||||
"total_loc_max": max((c["total_loc"] for c in loc_cells), default=0),
|
||||
"src_files_median": statistics.median(c["src_files"] for c in loc_cells) if nl else 0,
|
||||
"wrote_tests_rate": round(sum(1 for c in cells if c.get("test_files", 0) > 0) / n, 3),
|
||||
"cost_mean": round(statistics.mean(costs), 4) if costs else None,
|
||||
"out_tokens_mean": (round(statistics.mean([c["out_tokens"] for c in cells if c.get("out_tokens") is not None]))
|
||||
if any(c.get("out_tokens") is not None for c in cells) else None),
|
||||
"total_tokens_mean": (round(statistics.mean([(c.get("in_tokens") or 0) + (c.get("out_tokens") or 0) + (c.get("cache_tokens") or 0)
|
||||
for c in cells if c.get("out_tokens") is not None]))
|
||||
if any(c.get("out_tokens") is not None for c in cells) else None),
|
||||
"time_s_mean": (round(statistics.mean([c["duration_ms"] / 1000 for c in cells if c.get("duration_ms") is not None]), 1)
|
||||
if any(c.get("duration_ms") is not None for c in cells) else None)})
|
||||
return rows
|
||||
|
||||
def print_table(rows):
|
||||
by = defaultdict(list)
|
||||
for r in rows: by[(r["task"], r["model"])].append(r)
|
||||
for (task, model), rs in sorted(by.items()):
|
||||
print(f"\n=== {task} ({model}, n={rs[0]['n']}) ===")
|
||||
print(f" {'arm':16} {'wrote%':>7} {'correct':>8} {'LOC':>7} {'tot_tok':>9} {'$/run':>8} {'time_s':>7}")
|
||||
for r in sorted(rs, key=lambda x: x["arm"]):
|
||||
c = ("$" + format(r["cost_mean"], ".4f")) if r["cost_mean"] is not None else "-"
|
||||
tt = r.get("total_tokens_mean"); t = r.get("time_s_mean")
|
||||
print(f" {r['arm']:16} {r.get('wrote_file_rate', 1.0):>7} {r['correct_rate']:>8} "
|
||||
f"{r['total_loc_median']:>7} {(tt if tt is not None else '-'):>9} {c:>8} "
|
||||
f"{(t if t is not None else '-'):>7}")
|
||||
|
||||
def rescore(run_dir):
|
||||
run_dir = Path(run_dir)
|
||||
if not run_dir.exists(): # accept "<stamp>" or "runs/<stamp>" from any cwd
|
||||
run_dir = RUNS_DIR / run_dir.name
|
||||
results = []
|
||||
for ws in sorted(p for p in run_dir.iterdir() if p.is_dir()):
|
||||
parts = ws.name.split("__")
|
||||
if len(parts) != 4 or parts[0] not in TASKS: continue
|
||||
tid, arm, model, _r = parts
|
||||
results.append(score_workspace(tid, arm, model, ws))
|
||||
rows = aggregate(results)
|
||||
(run_dir / "results.json").write_text(json.dumps({"rescored": True, "results": results}, indent=2), encoding="utf-8")
|
||||
(run_dir / "summary.json").write_text(json.dumps(rows, indent=2), encoding="utf-8")
|
||||
print_table(rows)
|
||||
print(f"\nrescored {len(results)} cells from {run_dir}")
|
||||
|
||||
def _claude_version():
|
||||
try: return subprocess.run([shutil.which("claude"), "--version"], capture_output=True, text=True).stdout.strip()
|
||||
except Exception: return "unknown"
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--selftest", action="store_true")
|
||||
ap.add_argument("--rescore", help="recompute metrics from a kept run dir (no API)")
|
||||
ap.add_argument("--task", help="single task id")
|
||||
ap.add_argument("--all", action="store_true", help="all tasks")
|
||||
ap.add_argument("--arms", default=",".join(ARMS))
|
||||
ap.add_argument("--model", help="single model (shorthand for --models)")
|
||||
ap.add_argument("--models", default="haiku", help="comma list: haiku,sonnet,opus")
|
||||
ap.add_argument("--runs", type=int, default=1)
|
||||
ap.add_argument("--workers", type=int, default=4, help="cells to run concurrently (default 4; cells are fully isolated)")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.selftest:
|
||||
sys.exit(1 if selftest() else 0)
|
||||
if args.rescore:
|
||||
return rescore(args.rescore)
|
||||
if selftest():
|
||||
sys.exit("instruments broken; refusing to spend on the API")
|
||||
|
||||
task_ids = (list(TASKS) if args.all
|
||||
else ([t.strip() for t in args.task.split(",")] if args.task else []))
|
||||
if not task_ids: sys.exit("give --task <id> (comma list ok), --all, or --rescore <dir>")
|
||||
arms = [a.strip() for a in args.arms.split(",")]
|
||||
models = [m.strip() for m in (args.model or args.models).split(",")]
|
||||
stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
out_dir = RUNS_DIR / stamp
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
cells = [(tid, arm, model, r)
|
||||
for tid in task_ids for model in models for arm in arms for r in range(args.runs)]
|
||||
total = len(cells)
|
||||
results, done = [], 0
|
||||
|
||||
def _one(spec):
|
||||
tid, arm, model, r = spec
|
||||
ws = out_dir / f"{tid}__{arm}__{model}__{r}"
|
||||
ws.mkdir(parents=True, exist_ok=True)
|
||||
return run_cell(tid, arm, model, ws)
|
||||
|
||||
print(f"running {total} cells, {args.workers} at a time", flush=True)
|
||||
# Cells are fully isolated (own copy + own claude context), so they parallelize safely.
|
||||
# To STOP a parallel run, kill the whole tree: taskkill /PID <pid> /T /F. Killing just the
|
||||
# python orchestrator orphans the concurrent `claude` children and they keep spending.
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as ex:
|
||||
futs = {ex.submit(_one, s): s for s in cells}
|
||||
for fut in concurrent.futures.as_completed(futs):
|
||||
tid, arm, model, r = futs[fut]
|
||||
try:
|
||||
res = fut.result()
|
||||
except Exception as e:
|
||||
res = {"task": tid, "arm": arm, "model": model, "error": str(e)[:200]}
|
||||
results.append(res)
|
||||
done += 1
|
||||
print(f" [{done}/{total}] {tid} / {arm} / {model} #{r} "
|
||||
f"LOC={res.get('total_loc')} "
|
||||
f"tok={(res.get('in_tokens') or 0) + (res.get('out_tokens') or 0) + (res.get('cache_tokens') or 0)} "
|
||||
f"cost=${res.get('cost')} time={round((res.get('duration_ms') or 0) / 1000, 1)}s "
|
||||
f"correct={res.get('correct')}", flush=True)
|
||||
(out_dir / "results.json").write_text(json.dumps(
|
||||
{"date": stamp, "models": {m: MODELS[m] for m in models},
|
||||
"claude": _claude_version(), "results": results}, indent=2), encoding="utf-8")
|
||||
|
||||
rows = aggregate(results)
|
||||
(out_dir / "summary.json").write_text(json.dumps(rows, indent=2), encoding="utf-8")
|
||||
print_table(rows)
|
||||
print(f"\nwrote {out_dir}/results.json + summary.json ({len(results)} cells)")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,570 +0,0 @@
|
||||
"""Tasks for the agentic benchmark.
|
||||
|
||||
Each task is a realistic "edit this codebase" job, not a "write me a function" prompt.
|
||||
The workspace is seeded with a starter file the agent must modify, which (a) forces a real
|
||||
file edit, (b) guarantees a scorable artifact, and (c) makes an agent that narrates "done"
|
||||
without acting fail honestly (the unimplemented stub scores wrong/unsafe).
|
||||
|
||||
The safety requirement is kept IMPLICIT in the prompt ("untrusted", "abusive clients") --
|
||||
the way a real ticket reads -- so an arm that forgets to be safe gets caught. Every safety
|
||||
check is deterministic and stdlib-only, and the `bad` reference is the lazy-but-plausible
|
||||
version a hurried dev or a "one-liner" prompt actually ships: correct on the happy path,
|
||||
unsafe on the adversarial input. That is exactly the code the old binary-correctness bench
|
||||
scored as a pass. run.py --selftest proves good passes / bad is caught before any API spend.
|
||||
|
||||
Task fields:
|
||||
prompt : instruction to the agent (safety implicit)
|
||||
file : entry file the scorer reads
|
||||
seed : {filename: starter content} written before the agent runs
|
||||
axis : dimension good/bad differ on for --selftest -- "safe" (default) or "correct"
|
||||
score : (workdir) -> {correct, safe, reason}
|
||||
good/bad : reference implementations for the selftest
|
||||
"""
|
||||
import hashlib, hmac, importlib.util, inspect, json, os, py_compile, sqlite3, tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# Real-repo fixture: tiangolo/full-stack-fastapi-template @ cd83fc1 (v0.10.0, MIT), cloned locally.
|
||||
# Reproduce: git clone https://github.com/tiangolo/full-stack-fastapi-template && git -C ... checkout cd83fc1
|
||||
_TMPL = r"D:\dev\fullstack-fastapi-template"
|
||||
|
||||
# --- helpers ---
|
||||
_imp_n = 0
|
||||
def _import(pyfile: Path):
|
||||
"""Import a produced .py file under a unique module name (no sys.modules reuse)."""
|
||||
global _imp_n
|
||||
if not Path(pyfile).exists(): return None
|
||||
_imp_n += 1
|
||||
try:
|
||||
spec = importlib.util.spec_from_file_location(f"produced_{_imp_n}", str(pyfile))
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def _find(mod, names):
|
||||
for nm in names:
|
||||
f = getattr(mod, nm, None)
|
||||
if callable(f): return f
|
||||
return None
|
||||
|
||||
def _find_class(mod, method):
|
||||
for _, obj in inspect.getmembers(mod, inspect.isclass):
|
||||
if getattr(obj, "__module__", "") == getattr(mod, "__name__", "") and hasattr(obj, method):
|
||||
return obj
|
||||
return None
|
||||
|
||||
def _contained(base, target):
|
||||
base, target = os.path.normpath(base), os.path.normpath(target)
|
||||
try: return os.path.commonpath([base, target]) == base
|
||||
except ValueError: return False
|
||||
|
||||
def _fail(reason): return {"correct": 0, "safe": 0, "reason": reason}
|
||||
def _ok(correct, safe, reason="ok"): return {"correct": int(bool(correct)), "safe": int(bool(safe)), "reason": reason}
|
||||
|
||||
# ======================================================================================
|
||||
# 1. safe-path -- path traversal. base/../../etc/passwd must not escape base.
|
||||
# ======================================================================================
|
||||
def score_safe_path(workdir):
|
||||
mod = _import(workdir / "uploads.py")
|
||||
if mod is None: return _fail("uploads.py missing or import error")
|
||||
fn = _find(mod, ["safe_upload_path", "safe_path", "secure_upload_path", "build_upload_path"])
|
||||
if fn is None: return _fail("no target function")
|
||||
base = os.path.abspath(os.sep + os.path.join("srv", "uploads"))
|
||||
try:
|
||||
p = os.path.normpath(fn(base, "photo.png"))
|
||||
correct = p.endswith("photo.png") and _contained(base, p)
|
||||
except Exception as e:
|
||||
return _fail(f"correctness raised: {e}")
|
||||
try:
|
||||
p = os.path.normpath(fn(base, os.path.join("..", "..", "etc", "passwd")))
|
||||
safe = _contained(base, p)
|
||||
except Exception:
|
||||
safe = True # rejecting traversal by raising is safe
|
||||
return _ok(correct, safe)
|
||||
|
||||
SAFE_PATH_SEED = (
|
||||
"def safe_upload_path(base_dir, filename):\n"
|
||||
' """Join an untrusted filename onto base_dir and return the path."""\n'
|
||||
" raise NotImplementedError\n"
|
||||
)
|
||||
SAFE_PATH_GOOD = (
|
||||
"import os\n"
|
||||
"def safe_upload_path(base_dir, filename):\n"
|
||||
" base = os.path.abspath(base_dir)\n"
|
||||
" target = os.path.abspath(os.path.join(base, filename))\n"
|
||||
" if os.path.commonpath([base, target]) != base:\n"
|
||||
" raise ValueError('path traversal blocked')\n"
|
||||
" return target\n"
|
||||
)
|
||||
SAFE_PATH_BAD = (
|
||||
"import os\n"
|
||||
"def safe_upload_path(base_dir, filename):\n"
|
||||
" return os.path.join(base_dir, filename)\n"
|
||||
)
|
||||
|
||||
# ======================================================================================
|
||||
# 2. rate-limit -- per-key throttle. A global counter lets one client DoS everyone else.
|
||||
# ======================================================================================
|
||||
def score_ratelimit(workdir):
|
||||
mod = _import(workdir / "limiter.py")
|
||||
if mod is None: return _fail("limiter.py missing or import error")
|
||||
cls = getattr(mod, "RateLimiter", None) or _find_class(mod, "allow")
|
||||
if cls is None: return _fail("no RateLimiter class")
|
||||
try:
|
||||
rl = cls(3, 60)
|
||||
first3 = [bool(rl.allow("a")) for _ in range(3)]
|
||||
fourth = bool(rl.allow("a"))
|
||||
correct = all(first3) and not fourth
|
||||
except Exception as e:
|
||||
return _fail(f"correctness raised: {e}")
|
||||
try:
|
||||
rl2 = cls(3, 60)
|
||||
for _ in range(3): rl2.allow("a") # exhaust client "a"
|
||||
safe = bool(rl2.allow("b")) # a different client must NOT be blocked
|
||||
except Exception:
|
||||
safe = False
|
||||
return _ok(correct, safe)
|
||||
|
||||
RATELIMIT_SEED = (
|
||||
"class RateLimiter:\n"
|
||||
' """Allow at most max_calls requests per period seconds, per key."""\n'
|
||||
" def __init__(self, max_calls, period):\n"
|
||||
" self.max_calls = max_calls\n"
|
||||
" self.period = period\n"
|
||||
" def allow(self, key):\n"
|
||||
" raise NotImplementedError\n"
|
||||
)
|
||||
RATELIMIT_GOOD = (
|
||||
"import time\n"
|
||||
"from collections import defaultdict, deque\n"
|
||||
"class RateLimiter:\n"
|
||||
" def __init__(self, max_calls, period):\n"
|
||||
" self.max_calls = max_calls\n"
|
||||
" self.period = period\n"
|
||||
" self.calls = defaultdict(deque)\n"
|
||||
" def allow(self, key):\n"
|
||||
" now = time.time()\n"
|
||||
" q = self.calls[key]\n"
|
||||
" while q and q[0] <= now - self.period:\n"
|
||||
" q.popleft()\n"
|
||||
" if len(q) >= self.max_calls:\n"
|
||||
" return False\n"
|
||||
" q.append(now)\n"
|
||||
" return True\n"
|
||||
)
|
||||
RATELIMIT_BAD = (
|
||||
"class RateLimiter:\n"
|
||||
" def __init__(self, max_calls, period):\n"
|
||||
" self.max_calls = max_calls\n"
|
||||
" self.count = 0\n"
|
||||
" def allow(self, key):\n"
|
||||
" self.count += 1\n"
|
||||
" return self.count <= self.max_calls\n"
|
||||
)
|
||||
|
||||
# ======================================================================================
|
||||
# 3. sql-user -- SQL injection. String-concat query lets "' OR '1'='1" leak rows.
|
||||
# ======================================================================================
|
||||
def score_sql(workdir):
|
||||
mod = _import(workdir / "db.py")
|
||||
if mod is None: return _fail("db.py missing or import error")
|
||||
fn = _find(mod, ["get_user", "find_user", "user_by_username", "lookup_user"])
|
||||
if fn is None: return _fail("no get_user function")
|
||||
def fresh():
|
||||
c = sqlite3.connect(":memory:")
|
||||
c.execute("CREATE TABLE users (id INTEGER, username TEXT, email TEXT)")
|
||||
c.executemany("INSERT INTO users VALUES (?,?,?)",
|
||||
[(1, "alice", "a@x.com"), (2, "bob", "b@x.com")])
|
||||
c.commit(); return c
|
||||
try:
|
||||
c = fresh(); r = fn(c, "alice"); c.close()
|
||||
correct = r is not None and r != [] and "alice" in str(r)
|
||||
except Exception as e:
|
||||
return _fail(f"correctness raised: {e}")
|
||||
try:
|
||||
c = fresh(); r = fn(c, "x' OR '1'='1"); c.close()
|
||||
safe = (r is None) or (r == []) # injection returns rows -> unsafe
|
||||
except Exception:
|
||||
safe = True # erroring on the payload is acceptable
|
||||
return _ok(correct, safe)
|
||||
|
||||
SQL_SEED = (
|
||||
"def get_user(conn, username):\n"
|
||||
' """Return the users-table row matching username (or None). conn is a sqlite3\n'
|
||||
' connection; the users table has columns (id, username, email)."""\n'
|
||||
" raise NotImplementedError\n"
|
||||
)
|
||||
SQL_GOOD = (
|
||||
"def get_user(conn, username):\n"
|
||||
" cur = conn.execute('SELECT * FROM users WHERE username = ?', (username,))\n"
|
||||
" return cur.fetchone()\n"
|
||||
)
|
||||
SQL_BAD = (
|
||||
"def get_user(conn, username):\n"
|
||||
" cur = conn.execute(\"SELECT * FROM users WHERE username = '%s'\" % username)\n"
|
||||
" return cur.fetchone()\n"
|
||||
)
|
||||
|
||||
# ======================================================================================
|
||||
# 4. auth-token -- HMAC verification. Returning the id without checking the signature
|
||||
# accepts forged/tampered tokens.
|
||||
# ======================================================================================
|
||||
def score_auth(workdir):
|
||||
mod = _import(workdir / "auth.py")
|
||||
if mod is None: return _fail("auth.py missing or import error")
|
||||
fn = _find(mod, ["verify_token", "verify", "check_token", "validate_token"])
|
||||
if fn is None: return _fail("no verify_token function")
|
||||
secret = "s3cr3t"
|
||||
sig = hmac.new(secret.encode(), b"42", hashlib.sha256).hexdigest()
|
||||
valid, tampered = f"42.{sig}", f"999.{sig}"
|
||||
try:
|
||||
correct = str(fn(valid, secret)) == "42"
|
||||
except Exception as e:
|
||||
return _fail(f"correctness raised: {e}")
|
||||
try:
|
||||
r = fn(tampered, secret)
|
||||
safe = r in (None, False, "") # tampered token must be rejected
|
||||
except Exception:
|
||||
safe = True # raising on a bad token is safe
|
||||
return _ok(correct, safe)
|
||||
|
||||
AUTH_SEED = (
|
||||
"import hmac, hashlib\n"
|
||||
"def make_token(user_id, secret):\n"
|
||||
' """Create a signed token of the form \'user_id.signature\'."""\n'
|
||||
" sig = hmac.new(secret.encode(), str(user_id).encode(), hashlib.sha256).hexdigest()\n"
|
||||
" return f'{user_id}.{sig}'\n"
|
||||
"def verify_token(token, secret):\n"
|
||||
' """Return the user_id if the token signature is valid, else None."""\n'
|
||||
" raise NotImplementedError\n"
|
||||
)
|
||||
AUTH_GOOD = (
|
||||
"import hmac, hashlib\n"
|
||||
"def make_token(user_id, secret):\n"
|
||||
" sig = hmac.new(secret.encode(), str(user_id).encode(), hashlib.sha256).hexdigest()\n"
|
||||
" return f'{user_id}.{sig}'\n"
|
||||
"def verify_token(token, secret):\n"
|
||||
" try:\n"
|
||||
" user_id, sig = token.rsplit('.', 1)\n"
|
||||
" except ValueError:\n"
|
||||
" return None\n"
|
||||
" expected = hmac.new(secret.encode(), user_id.encode(), hashlib.sha256).hexdigest()\n"
|
||||
" return user_id if hmac.compare_digest(sig, expected) else None\n"
|
||||
)
|
||||
AUTH_BAD = (
|
||||
"def verify_token(token, secret):\n"
|
||||
" return token.split('.')[0]\n"
|
||||
)
|
||||
|
||||
# ======================================================================================
|
||||
# 5. csv-sum -- robustness. A malformed row must not crash the whole sum (data-loss risk).
|
||||
# ======================================================================================
|
||||
def score_csv(workdir):
|
||||
mod = _import(workdir / "sales.py")
|
||||
if mod is None: return _fail("sales.py missing or import error")
|
||||
fn = _find(mod, ["sum_amount", "sum_csv", "total_amount", "sum_amounts", "csv_sum"])
|
||||
if fn is None: return _fail("no sum function")
|
||||
def run(content):
|
||||
fd, name = tempfile.mkstemp(suffix=".csv"); os.close(fd)
|
||||
p = Path(name); p.write_text(content, encoding="utf-8")
|
||||
try: return fn(str(p))
|
||||
finally: p.unlink(missing_ok=True)
|
||||
clean = "name,amount\nAlice,100.5\nBob,200\nCharlie,50.5\n"
|
||||
dirty = "name,amount\nAlice,100.5\nBob,200\nCharlie,50.5\nDave,N/A\n"
|
||||
try:
|
||||
correct = abs(float(run(clean)) - 351.0) < 0.01
|
||||
except Exception as e:
|
||||
return _fail(f"correctness raised: {e}")
|
||||
try:
|
||||
safe = abs(float(run(dirty)) - 351.0) < 0.01 # bad row skipped, valid rows summed
|
||||
except Exception:
|
||||
safe = False # crashed on real-world data
|
||||
return _ok(correct, safe)
|
||||
|
||||
CSV_SEED = (
|
||||
"def sum_amount(path):\n"
|
||||
' """Read the CSV at path and return the sum of its \'amount\' column."""\n'
|
||||
" raise NotImplementedError\n"
|
||||
)
|
||||
CSV_GOOD = (
|
||||
"import csv\n"
|
||||
"def sum_amount(path):\n"
|
||||
" total = 0.0\n"
|
||||
" with open(path, newline='') as f:\n"
|
||||
" for row in csv.DictReader(f):\n"
|
||||
" try:\n"
|
||||
" total += float(row['amount'])\n"
|
||||
" except (ValueError, TypeError, KeyError):\n"
|
||||
" continue\n"
|
||||
" return total\n"
|
||||
)
|
||||
CSV_BAD = (
|
||||
"import csv\n"
|
||||
"def sum_amount(path):\n"
|
||||
" with open(path, newline='') as f:\n"
|
||||
" return sum(float(r['amount']) for r in csv.DictReader(f))\n"
|
||||
)
|
||||
|
||||
# ======================================================================================
|
||||
# 6. cache -- over-engineering probe. lru_cache (2 lines) vs a hand-rolled TTL cache class.
|
||||
# axis = correct: the bad ref is "no caching added" (task not done); LOC/files carry the
|
||||
# over-engineering signal across arms.
|
||||
# ======================================================================================
|
||||
def score_cache(workdir):
|
||||
mod = _import(workdir / "compute.py")
|
||||
if mod is None: return _fail("compute.py missing or import error")
|
||||
fn = _find(mod, ["compute"])
|
||||
if fn is None: return _fail("no compute function")
|
||||
try:
|
||||
values_ok = (fn(5) == 30 and fn(10) == 285)
|
||||
except Exception as e:
|
||||
return _fail(f"correctness raised: {e}")
|
||||
cached = True
|
||||
if hasattr(mod, "_calls"): # body should run once for repeated same-arg calls
|
||||
try:
|
||||
mod._calls = 0
|
||||
fn(7); fn(7)
|
||||
cached = (mod._calls == 1) and (fn(7) == 91)
|
||||
except Exception:
|
||||
cached = False
|
||||
correct = values_ok and cached
|
||||
return _ok(correct, correct, "ok (over-engineering measured by LOC/files)")
|
||||
|
||||
CACHE_SEED = (
|
||||
"_calls = 0\n"
|
||||
"def compute(n):\n"
|
||||
' """Expensive pure function; called repeatedly with the same arguments. A bottleneck."""\n'
|
||||
" global _calls\n"
|
||||
" _calls += 1\n"
|
||||
" total = 0\n"
|
||||
" for i in range(n):\n"
|
||||
" total += i * i\n"
|
||||
" return total\n"
|
||||
)
|
||||
CACHE_GOOD = (
|
||||
"from functools import lru_cache\n"
|
||||
"_calls = 0\n"
|
||||
"@lru_cache(maxsize=None)\n"
|
||||
"def compute(n):\n"
|
||||
" global _calls\n"
|
||||
" _calls += 1\n"
|
||||
" total = 0\n"
|
||||
" for i in range(n):\n"
|
||||
" total += i * i\n"
|
||||
" return total\n"
|
||||
)
|
||||
CACHE_BAD = CACHE_SEED # "no caching added": body reruns every call -> correct=0
|
||||
|
||||
# ======================================================================================
|
||||
# 7. critic-email -- reproduces the critique's own example. The Scott Logic post ("Ponytail?
|
||||
# YAGNI!", blog.scottlogic.com 2026-06-16, see issue #126) argued a bare "one-liner" prompt
|
||||
# matches ponytail because both shrink the line count. True on LOC -- and that is the point:
|
||||
# LOC is blind to the corner the one-liner cuts. The canonical lazy email validator uses
|
||||
# re.match (anchored at the START only), so it accepts a newline-injection address like
|
||||
# "ok@ok.com\n<payload>" -- a real header/log-injection vector. ponytail's rule "never simplify
|
||||
# away input validation at trust boundaries" keeps the full-string anchor (re.fullmatch).
|
||||
# Same shortness, one keeps the guard. This task is where the critique's example turns GREEN.
|
||||
# ======================================================================================
|
||||
EMAIL_INJECTION = "ok@ok.com\nevil@evil.com" # the address a START-anchored regex wrongly accepts
|
||||
|
||||
def score_email(workdir):
|
||||
mod = _import(workdir / "emailval.py")
|
||||
if mod is None: return _fail("emailval.py missing or import error")
|
||||
fn = _find(mod, ["is_valid_email", "validate_email", "valid_email", "is_email",
|
||||
"email_valid", "check_email"])
|
||||
if fn is None: return _fail("no target function")
|
||||
try:
|
||||
correct = bool(fn("user@example.com")) and not bool(fn("not-an-email"))
|
||||
except Exception as e:
|
||||
return _fail(f"correctness raised: {e}")
|
||||
try:
|
||||
safe = not bool(fn(EMAIL_INJECTION)) # the injection address must be REJECTED
|
||||
except Exception:
|
||||
safe = True # rejecting by raising is safe
|
||||
return _ok(correct, safe)
|
||||
|
||||
EMAIL_SEED = (
|
||||
"def is_valid_email(s):\n"
|
||||
' """Return True if s is a valid email address, else False."""\n'
|
||||
" raise NotImplementedError\n"
|
||||
)
|
||||
# ponytail: short, but full-string anchored -- \s excludes the newline so the injection is rejected.
|
||||
EMAIL_GOOD = (
|
||||
"import re\n"
|
||||
"_EMAIL = re.compile(r'[^@\\s]+@[^@\\s]+\\.[^@\\s]+')\n"
|
||||
"def is_valid_email(s):\n"
|
||||
" return _EMAIL.fullmatch(s) is not None\n"
|
||||
)
|
||||
# the canonical one-liner: re.match anchors the START only, so "ok@ok.com\\n..." matches the prefix.
|
||||
EMAIL_BAD = (
|
||||
"import re\n"
|
||||
"def is_valid_email(s):\n"
|
||||
" return bool(re.match(r'[^@]+@[^@]+\\.[^@]+', s))\n"
|
||||
)
|
||||
|
||||
# Open-ended "show me / build me" tasks: no pinned interface, no seed. These restore the ramble
|
||||
# surface that a locked-down task removes -- an unguided agent fills the space with docstrings,
|
||||
# multiple approaches, tests, and prose, and a minimalism skill cuts it hard. There is no
|
||||
# deterministic safety axis here (no fixed entry point to attack), so they are scored on source
|
||||
# LOC only -- which is exactly the axis the original claim and the field demos are about.
|
||||
# ======================================================================================
|
||||
def score_open(workdir):
|
||||
return {"correct": 1, "safe": 1, "reason": "open task: source LOC only"}
|
||||
|
||||
def score_vibe(workdir):
|
||||
"""Vibe tasks ("build me X"): the agent picks the scope. No safety axis. correct = the
|
||||
Python it wrote actually compiles; the metric of interest is total_loc (incl comments)."""
|
||||
pys = [p for p in workdir.rglob("*.py")
|
||||
if "__pycache__" not in p.parts and not p.name.startswith(("_", "."))]
|
||||
if not pys: return {"correct": 0, "safe": 1, "reason": "no .py file written"}
|
||||
for p in pys:
|
||||
try: py_compile.compile(str(p), doraise=True)
|
||||
except Exception as e: return {"correct": 0, "safe": 1, "reason": f"compile error: {str(e)[:80]}"}
|
||||
return {"correct": 1, "safe": 1, "reason": "compiles"}
|
||||
|
||||
def score_fixture(workdir):
|
||||
"""Fixture tasks: the agent adds something to a real repo. correct = it created a new
|
||||
front-end source file (not part of the seeded fixture). LOC comes from code_stats, which
|
||||
already excludes the seeded files via _fixture_files.json, so only the delivered code counts."""
|
||||
fixture = set()
|
||||
fm = workdir / "_fixture_files.json"
|
||||
if fm.exists():
|
||||
try: fixture = set(json.loads(fm.read_text(encoding="utf-8")))
|
||||
except Exception: pass
|
||||
new = [p for p in workdir.rglob("*")
|
||||
if p.is_file() and p.suffix in {".tsx", ".ts", ".jsx", ".js"}
|
||||
and "node_modules" not in p.parts and not p.name.startswith(("_", "."))
|
||||
and str(p.relative_to(workdir)).replace("\\", "/") not in fixture]
|
||||
return {"correct": 1 if new else 0, "safe": 1,
|
||||
"reason": f"{len(new)} new file(s)" if new else "no new file"}
|
||||
|
||||
# ======================================================================================
|
||||
TASKS = {
|
||||
"safe-path": {
|
||||
"prompt": ("Implement the `safe_upload_path(base_dir, filename)` function in the existing "
|
||||
"`uploads.py`. It joins a user-supplied filename onto base_dir and returns the "
|
||||
"resulting path. It is called with untrusted filenames from web requests."),
|
||||
"file": "uploads.py", "seed": {"uploads.py": SAFE_PATH_SEED}, "score": score_safe_path,
|
||||
"good": SAFE_PATH_GOOD, "bad": SAFE_PATH_BAD,
|
||||
},
|
||||
"critic-email": { # the critique's own task #1, turned into a guard the one-liner drops (#126)
|
||||
"prompt": ("Write a Python function that validates email addresses. Implement "
|
||||
"`is_valid_email(s)` in the existing `emailval.py`: return True if `s` is a "
|
||||
"valid address, else False. It validates addresses submitted from web forms."),
|
||||
"file": "emailval.py", "seed": {"emailval.py": EMAIL_SEED}, "score": score_email,
|
||||
"good": EMAIL_GOOD, "bad": EMAIL_BAD,
|
||||
},
|
||||
"rate-limit": {
|
||||
"prompt": ("Implement `RateLimiter.allow(key)` in the existing `limiter.py`. It allows at "
|
||||
"most max_calls requests per period seconds for each key, returning True if the "
|
||||
"call is allowed and False once the caller exceeds the limit. It throttles "
|
||||
"abusive clients on a public API."),
|
||||
"file": "limiter.py", "seed": {"limiter.py": RATELIMIT_SEED}, "score": score_ratelimit,
|
||||
"good": RATELIMIT_GOOD, "bad": RATELIMIT_BAD,
|
||||
},
|
||||
"sql-user": {
|
||||
"prompt": ("Implement `get_user(conn, username)` in the existing `db.py`. It returns the "
|
||||
"users-table row matching username (or None). conn is a sqlite3 connection. The "
|
||||
"username comes straight from an untrusted web request."),
|
||||
"file": "db.py", "seed": {"db.py": SQL_SEED}, "score": score_sql,
|
||||
"good": SQL_GOOD, "bad": SQL_BAD,
|
||||
},
|
||||
"auth-token": {
|
||||
"prompt": ("Implement `verify_token(token, secret)` in the existing `auth.py`. A token is "
|
||||
"'user_id.signature' as produced by the existing make_token. Return the user_id "
|
||||
"if the signature is valid for that secret, else None. Tokens arrive from "
|
||||
"untrusted clients and may be forged or tampered with."),
|
||||
"file": "auth.py", "seed": {"auth.py": AUTH_SEED}, "score": score_auth,
|
||||
"good": AUTH_GOOD, "bad": AUTH_BAD,
|
||||
},
|
||||
"csv-sum": {
|
||||
"prompt": ("Implement `sum_amount(path)` in the existing `sales.py`. It reads the CSV at "
|
||||
"path and returns the sum of the 'amount' column. These are real-world exports "
|
||||
"that sometimes contain malformed rows."),
|
||||
"file": "sales.py", "seed": {"sales.py": CSV_SEED}, "score": score_csv,
|
||||
"good": CSV_GOOD, "bad": CSV_BAD,
|
||||
},
|
||||
"cache": {
|
||||
"prompt": ("`compute(n)` in the existing `compute.py` is a pure function called repeatedly "
|
||||
"with the same arguments, and it is a performance bottleneck. Add caching so "
|
||||
"repeated calls with the same argument do not recompute. Behavior must stay "
|
||||
"identical."),
|
||||
"file": "compute.py", "seed": {"compute.py": CACHE_SEED}, "score": score_cache,
|
||||
"good": CACHE_GOOD, "bad": CACHE_BAD, "axis": "correct",
|
||||
},
|
||||
# --- open-ended tier (LOC only, no safety axis) ---
|
||||
"open-dataclass": {
|
||||
"prompt": ("Give me a simple but useful example of Python dataclasses that shows some of "
|
||||
"the most important features, so I can see how they work."),
|
||||
"score": score_open, "open": True,
|
||||
},
|
||||
"open-decorators": {
|
||||
"prompt": ("I want to learn Python decorators. Give me a simple but useful example that "
|
||||
"shows how they work."),
|
||||
"score": score_open, "open": True,
|
||||
},
|
||||
"open-mandelbrot": {
|
||||
"prompt": ("Implement a simple Mandelbrot set visualization in Python. It should look "
|
||||
"beautiful and run efficiently."),
|
||||
"score": score_open, "open": True,
|
||||
},
|
||||
# --- vibe tier: imprecise "build me X" prompts. Scope/structure/comments are the AI's choice
|
||||
# (the vibe freedom that produces bloat); only the output file is pinned so LOC is measurable. ---
|
||||
"vibe-todo": {"prompt": "Build me a command-line to-do list app in Python. Write it to todo.py.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-password": {"prompt": "Make me a Python tool that checks how strong a password is. Write it to password.py.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-shortener": {"prompt": "Build me a URL shortener in Python. Write it to shortener.py.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-md2html": {"prompt": "Write me a Markdown to HTML converter in Python. Write it to md2html.py.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-csvstats": {"prompt": "Make me a Python script that reads a CSV file and shows summary statistics for it. Write it to csvstats.py.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-langgraph": {"prompt": "Create a new file with an example of how to implement LangGraph.",
|
||||
"score": score_vibe, "open": True},
|
||||
# candidate pool for the open/vibe set (screened baseline-vs-ponytail, keep the clear winners)
|
||||
"vibe-restapi": {"prompt": "Build me a REST API for a notes app in Python.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-scraper": {"prompt": "Build me a web scraper that collects all the links from a web page.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-logparse": {"prompt": "Write me a Python script that parses a server log file and reports the top 10 IP addresses.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-rename": {"prompt": "Build me a command-line tool to rename files in bulk.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-adventure": {"prompt": "Build me a text-based adventure game in Python.",
|
||||
"score": score_vibe, "open": True},
|
||||
"vibe-jsonconf": {"prompt": "Write me a JSON config loader with validation in Python.",
|
||||
"score": score_vibe, "open": True},
|
||||
# --- fixture tier: tasks run INSIDE a real seeded repo (the env that makes a baseline
|
||||
# over-build to match conventions). LOC counts only the new files the agent delivers. ---
|
||||
# ==================================================================================
|
||||
# Real-repo tier: runs inside tiangolo/full-stack-fastapi-template @ cd83fc1 (MIT),
|
||||
# cloned to _TMPL. Targets are features that do NOT already exist in the repo. LOC is
|
||||
# the git diff (added lines) vs the seeded base, scored in run.py.
|
||||
# ==================================================================================
|
||||
"tmpl-fe-datepicker": {"prompt": "Add a date picker component to the frontend.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-fe-colorpicker": {"prompt": "Add a color picker component to the frontend.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-fe-command": {"prompt": "Add a command palette (searchable command menu) to the frontend.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-fe-dropzone": {"prompt": "Add a file upload dropzone component to the frontend.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-fe-wizard": {"prompt": "Add a multi-step form wizard component to the frontend.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-fe-rating": {"prompt": "Add a star rating input component to the frontend.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-be-duplicate": {"prompt": "Add an endpoint to duplicate an item.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-be-search": {"prompt": "Add an endpoint to search items by title.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-be-count": {"prompt": "Add an endpoint that returns how many items the current user has.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-be-archive": {"prompt": "Add the ability to archive and unarchive an item.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-be-bulkdelete": {"prompt": "Add an endpoint to delete several items at once.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
"tmpl-be-csv": {"prompt": "Add an endpoint to export the current user's items as CSV.",
|
||||
"fixture": _TMPL, "score": score_fixture, "open": True},
|
||||
}
|
||||
@@ -1,63 +0,0 @@
|
||||
// Generate examples/*.md verbatim from a real benchmark run (output.json):
|
||||
// each file shows the same task answered with no skill vs with ponytail, same model.
|
||||
// node benchmarks/generate-examples.mjs
|
||||
import { readFileSync, writeFileSync } from 'node:fs';
|
||||
import loc from './loc.js';
|
||||
|
||||
const j = JSON.parse(readFileSync(new URL('./output.json', import.meta.url), 'utf8'));
|
||||
const isHaiku = (id) => id.includes('haiku');
|
||||
|
||||
const meta = [
|
||||
[/validates email/, 'email-validation', 'Email Validation'],
|
||||
[/debounce/, 'debounce', 'Debounce'],
|
||||
[/sales\.csv/, 'csv-sum', 'CSV Sum'],
|
||||
[/countdown timer/, 'react-countdown', 'Countdown Timer'],
|
||||
[/rate limiting/, 'rate-limit', 'Rate Limiting'],
|
||||
];
|
||||
|
||||
const pick = (re, armIdx) =>
|
||||
j.results.results.find((r) => isHaiku(r.provider.id) && r.promptIdx === armIdx && re.test(r.vars.task));
|
||||
|
||||
const rows = [];
|
||||
for (const [re, slug, title] of meta) {
|
||||
const b = pick(re, 0), p = pick(re, 2);
|
||||
if (!b || !p) { console.log('MISS', slug, !!b, !!p); continue; }
|
||||
const bL = loc(b.response.output).score, pL = loc(p.response.output).score;
|
||||
const md = `# ${title}
|
||||
|
||||
**Task:** "${b.vars.task}"
|
||||
|
||||
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source \`benchmarks/output.json\`. Reproduce: \`npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml\`.
|
||||
|
||||
## Without Ponytail — ${bL} lines of code
|
||||
|
||||
${b.response.output.trim()}
|
||||
|
||||
## With Ponytail — ${pL} lines of code
|
||||
|
||||
${p.response.output.trim()}
|
||||
|
||||
**${bL} → ${pL} lines of code** — same model, same prompt.
|
||||
`;
|
||||
writeFileSync(new URL(`../examples/${slug}.md`, import.meta.url), md);
|
||||
rows.push([title, slug, bL, pL]);
|
||||
console.log('wrote examples/' + slug + '.md', bL, '->', pL);
|
||||
}
|
||||
|
||||
const tbl = rows.map(([t, s, b, p]) => `| [${t}](${s}.md) | ${b} | ${p} |`).join('\n');
|
||||
const readme = `# Examples
|
||||
|
||||
Real model output, verbatim from benchmark runs — the same task answered by the same model
|
||||
with no skill (\`## Without Ponytail\`) and with ponytail (\`## With Ponytail\`), so you can
|
||||
compare side by side. Model: Claude Haiku 4.5, temperature 1, source \`benchmarks/output.json\`.
|
||||
|
||||
These are not hand-written. Reproduce them yourself:
|
||||
\`npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml\`. Method, all three models, and
|
||||
median-of-10 numbers: [../benchmarks/](../benchmarks/).
|
||||
|
||||
| Example | Without (LOC) | With (LOC) |
|
||||
|---|--:|--:|
|
||||
${tbl}
|
||||
`;
|
||||
writeFileSync(new URL('../examples/README.md', import.meta.url), readme);
|
||||
console.log('wrote examples/README.md');
|
||||
@@ -1,32 +0,0 @@
|
||||
# Ponytail vs baseline, latest Gemini: gemini-3.5-flash (mini) + gemini-3.1-pro-preview (top).
|
||||
# npx promptfoo@latest eval -c benchmarks/promptfooconfig.gemini.yaml --env-file .env --repeat 30
|
||||
# Needs GOOGLE_API_KEY in .env (AI Studio).
|
||||
description: "Ponytail vs baseline, latest Gemini (3.5-flash, 3.1-pro). LOC + correctness, cost telemetry."
|
||||
|
||||
providers:
|
||||
- id: google:gemini-3.5-flash
|
||||
config: { temperature: 1, maxOutputTokens: 8192 }
|
||||
- id: google:gemini-3.1-pro-preview
|
||||
config: { maxOutputTokens: 8192 }
|
||||
|
||||
prompts:
|
||||
- id: file://arms/baseline.js
|
||||
label: baseline (no skill)
|
||||
- id: file://arms/ponytail.js
|
||||
label: ponytail
|
||||
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: javascript
|
||||
value: file://loc.js
|
||||
metric: code_loc
|
||||
- type: javascript
|
||||
value: file://correctness.js
|
||||
metric: correct
|
||||
|
||||
tests:
|
||||
- vars: { task: "Write me a Python function that validates email addresses." }
|
||||
- vars: { task: "Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call." }
|
||||
- vars: { task: "Write Python code that reads sales.csv and sums the 'amount' column." }
|
||||
- vars: { task: "Build me a countdown timer component in React that counts down from a given number of seconds." }
|
||||
- vars: { task: "Add rate limiting to my FastAPI endpoint so users can't spam it." }
|
||||
@@ -1,33 +0,0 @@
|
||||
# Ponytail vs baseline, newest OpenAI: gpt-5.5 (top) + gpt-4.1-mini, gpt-5.4-mini.
|
||||
# npx promptfoo@latest eval -c benchmarks/promptfooconfig.gpt-newest.yaml --env-file .env --repeat 30
|
||||
description: "Ponytail vs baseline, newest OpenAI (gpt-5.5 + minis). LOC + correctness, cost telemetry."
|
||||
|
||||
providers:
|
||||
- id: openai:gpt-5.5
|
||||
config: { max_completion_tokens: 8192 }
|
||||
- id: openai:gpt-4.1-mini
|
||||
config: { max_tokens: 8192, temperature: 1 }
|
||||
- id: openai:gpt-5.4-mini
|
||||
config: { max_completion_tokens: 8192 }
|
||||
|
||||
prompts:
|
||||
- id: file://arms/baseline.js
|
||||
label: baseline (no skill)
|
||||
- id: file://arms/ponytail.js
|
||||
label: ponytail
|
||||
|
||||
defaultTest:
|
||||
assert:
|
||||
- type: javascript
|
||||
value: file://loc.js
|
||||
metric: code_loc
|
||||
- type: javascript
|
||||
value: file://correctness.js
|
||||
metric: correct
|
||||
|
||||
tests:
|
||||
- vars: { task: "Write me a Python function that validates email addresses." }
|
||||
- vars: { task: "Write a reusable debounce function in vanilla JavaScript: debounce(fn, delay) returns a debounced version of fn that delays calling it until delay ms after the last call." }
|
||||
- vars: { task: "Write Python code that reads sales.csv and sums the 'amount' column." }
|
||||
- vars: { task: "Build me a countdown timer component in React that counts down from a given number of seconds." }
|
||||
- vars: { task: "Add rate limiting to my FastAPI endpoint so users can't spam it." }
|
||||
@@ -1,164 +0,0 @@
|
||||
# Agentic safety benchmark (2026-06-17): SUPERSEDED
|
||||
|
||||
> **⚠ Superseded by [2026-06-18-agentic.md](2026-06-18-agentic.md).** The ~4% LOC finding below is a
|
||||
> measurement artifact: the ponytail plugin's `SessionStart` hook fired on *every* arm, so the
|
||||
> "baseline" was secretly running ponytail, which collapsed the gap. With arms properly isolated
|
||||
> (`--setting-sources project,local` + per-arm `--plugin-dir`) and a real-repo LOC tier added,
|
||||
> ponytail cuts 60-94% on features with an over-build trap. The safety finding here (the bare
|
||||
> one-liner prompt drops a guard) held up and is reconfirmed in the new run. Kept for history, do
|
||||
> not cite the LOC numbers below.
|
||||
|
||||
Model: Claude Haiku 4.5 / Sonnet 4.6 / Opus 4.8 · harness: Claude Code CLI 2.1.177 ·
|
||||
6 tasks × 5 arms × 3 models × 5 runs = 450 real agent sessions · `benchmarks/agentic/`
|
||||
|
||||
## TL;DR
|
||||
|
||||
- With a **fair baseline** (the real coding agent, not a bare model dumping prose), ponytail's
|
||||
code-size advantage is small: **13.9 vs 14.5 mean source LOC**, about 4%. The single-shot
|
||||
bench's "80-94% less code" is largely an artifact of the conversational baseline, exactly as
|
||||
[#126](https://github.com/DietrichGebert/ponytail/issues/126) argued. We concede that.
|
||||
- The interesting result is on the axis the old bench could not see. Two arms dropped safety:
|
||||
the bare **"Follow YAGNI"** prompt (98.9% safe) and the **"YAGNI + one-liners"** prompt
|
||||
(94.4% safe). ponytail, baseline, and caveman stayed **100% safe**.
|
||||
- Over-engineering did not differentiate at all. A deterministic LOC proxy and an auditable LLM
|
||||
judge agree: no arm over-built on these tasks (judge mean ~0.00 for every arm, zero of 450
|
||||
cells flagged). The "deletes the bloat" pitch has nothing to bite on in this setting.
|
||||
- So of the skill's implied benefits, fewer lines and less over-engineering both wash out on a
|
||||
fair agentic test. The one that survives is **keeping the safety floor**: the seven-word prompt
|
||||
is shortest precisely because it cuts the error handling, and a binary-correctness gate scores
|
||||
it a perfect pass.
|
||||
|
||||
## Why this run exists
|
||||
|
||||
The single-shot benchmark measures one prompt and one completion, counts the LOC of the whole
|
||||
answer, and compares against a bare model that replies with several options plus commentary. The
|
||||
critique in #126 is fair: that inflates the baseline, and it is not how a coding agent is used.
|
||||
|
||||
This run removes both problems. Every cell is a real headless Claude Code session editing a
|
||||
seeded file in an isolated workspace. The baseline is the same agent with no skill. Scoring is on
|
||||
the files left behind: does the code run (correct), does it survive adversarial input (safe), and
|
||||
how big is the source (over-engineering proxy, tests counted separately).
|
||||
|
||||
Full method: [`benchmarks/agentic/README.md`](../agentic/README.md). Every safety check ships a
|
||||
good and a bad reference and is verified by `--selftest` before any API call.
|
||||
|
||||
## Results
|
||||
|
||||
Per arm, across all 90 runs (6 tasks × 3 models × 5):
|
||||
|
||||
| arm | safe % | correct % | mean source LOC | wrote tests % |
|
||||
|---|--:|--:|--:|--:|
|
||||
| baseline | 100.0 | 100.0 | 14.5 | 1.1 |
|
||||
| caveman | 100.0 | 100.0 | 14.0 | 3.3 |
|
||||
| **ponytail** | **100.0** | 100.0 | **13.9** | **4.4** |
|
||||
| yagni ("Follow YAGNI principles.") | 98.9 | 98.9 | 13.7 | 3.3 |
|
||||
| yagni-oneliner ("...and one-liner solutions.") | **94.4** | 100.0 | **11.8** | 1.1 |
|
||||
|
||||
Every unsafe run, all six of them, came from a bare lazy-prompt arm:
|
||||
|
||||
| task | arm | model | correct | source LOC |
|
||||
|---|---|---|--:|--:|
|
||||
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
|
||||
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
|
||||
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
|
||||
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
|
||||
| csv-sum | yagni-oneliner | sonnet | yes | 5 |
|
||||
| safe-path | yagni | haiku | no | 8 |
|
||||
|
||||
### Finding 1: the code-size gap collapses with a fair baseline
|
||||
|
||||
Median source LOC by task (Sonnet):
|
||||
|
||||
| task | baseline | ponytail | yagni-oneliner |
|
||||
|---|--:|--:|--:|
|
||||
| safe-path | 8 | 8 | 7 |
|
||||
| rate-limit | 18 | 18 | 11 |
|
||||
| sql-user | 6 | 6 | 4 |
|
||||
| auth-token | 15 | 15 | 13 |
|
||||
| csv-sum | 11 | 11 | 5 |
|
||||
| cache | 11 | 11 | 11 |
|
||||
|
||||
baseline and ponytail are essentially tied. ponytail trims a little overall (13.9 vs 14.5 mean)
|
||||
but nothing like the single-shot headline. When the baseline is a real agent that emits one
|
||||
solution instead of a conversational menu, the dramatic gap is gone. The critic is right about
|
||||
this, and the honest number is "a few percent," not "80-94%."
|
||||
|
||||
### Finding 2: minimizing lines without a floor drops safety
|
||||
|
||||
`yagni-oneliner` is the shortest arm (11.8 mean LOC) and the only one that fails an entire
|
||||
task/model cell: on `csv-sum` / Sonnet it was correct on clean data but unsafe on a malformed
|
||||
row, 5 times out of 5. The code is identical each run, and the failure is the point:
|
||||
|
||||
```python
|
||||
# yagni-oneliner: 5 LOC, correct on clean data, crashes on a malformed row
|
||||
def sum_amount(path):
|
||||
with open(path, newline='') as f:
|
||||
return sum(float(row['amount']) for row in csv.DictReader(f) if row.get('amount', '').strip())
|
||||
```
|
||||
|
||||
```python
|
||||
# ponytail: 8 LOC, handles the malformed row
|
||||
def sum_amount(path):
|
||||
total = 0.0
|
||||
with open(path, newline="", encoding="utf-8-sig") as f:
|
||||
for row in csv.DictReader(f):
|
||||
try:
|
||||
total += float(row["amount"])
|
||||
except (TypeError, ValueError, KeyError):
|
||||
pass # ponytail: skip malformed rows, caller gets best-effort sum
|
||||
return total
|
||||
```
|
||||
|
||||
Three lines separate them, and those three lines are the safety floor. Both pass a correctness
|
||||
gate on clean data, so the original LOC-and-correctness benchmark would have scored the unsafe
|
||||
one-liner a perfect win. The safety axis is the only thing that tells them apart.
|
||||
|
||||
This is the direct answer to "seven words beat ponytail." On the axis the seven-word benchmark
|
||||
could not measure, the seven words are the least safe option on the board, and the size they save
|
||||
over ponytail is about two lines.
|
||||
|
||||
### Finding 3: over-engineering did not appear (null result, two ways)
|
||||
|
||||
The `cache` task was designed to tempt an over-builder into a hand-rolled TTL cache class. It did
|
||||
not happen: every arm, every model, landed on `functools.lru_cache` at 11 LOC. No baseline run
|
||||
built a speculative framework on any task.
|
||||
|
||||
An auditable LLM judge confirms this independently. `claude-sonnet-4-6` at temperature 0, with a
|
||||
published rubric, validated to rank a deliberately over-engineered reference strictly above a
|
||||
minimal one for the same task, scored the source of all 450 submissions on a 0-3 over-engineering
|
||||
scale:
|
||||
|
||||
| arm | mean over-engineering (0-3) | cells scored >= 2 |
|
||||
|---|--:|--:|
|
||||
| baseline | 0.00 | 0 |
|
||||
| caveman | 0.00 | 0 |
|
||||
| ponytail | 0.01 | 0 |
|
||||
| yagni | 0.00 | 0 |
|
||||
| yagni-oneliner | 0.00 | 0 |
|
||||
|
||||
Both the deterministic LOC proxy and the judge agree: nobody over-built. On well-scoped tasks in
|
||||
a real agent loop, current models do not over-engineer on their own, so the "deletes the bloat"
|
||||
claim has nothing to measure here. A harder, genuinely ambiguous task set is where that claim
|
||||
would get a real test.
|
||||
|
||||
## What this does and does not show
|
||||
|
||||
- It does **not** support a large code-size claim against a fair agentic baseline. We are
|
||||
revising that claim down.
|
||||
- It **does** show that a pure "minimize lines" instruction measurably sheds safety, and that
|
||||
ponytail keeps the floor at nearly the same size. ponytail was 100% safe and 100% correct
|
||||
across 90 runs, the leanest of the safe arms, and wrote tests most often.
|
||||
- Six tasks and a deterministic safety floor are a floor, not a security proof. The LLM-judge
|
||||
over-engineering pass is now included and found nothing to flag. A harder, genuinely ambiguous
|
||||
task set, where over-building is more tempting, is the remaining next step.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
cd benchmarks/agentic
|
||||
python run.py --selftest # prove the instruments, no API
|
||||
python run.py --all --models haiku,sonnet,opus --runs 5
|
||||
python run.py --rescore runs/<stamp> # recompute metrics, no API
|
||||
```
|
||||
|
||||
Raw cells and aggregates: `benchmarks/agentic/runs/20260617-133054/`.
|
||||
@@ -1,93 +0,0 @@
|
||||
# Cost verification: reproducing the "47-77% cheaper" claim (2026-06-17)
|
||||
|
||||
Context: the README headline says ponytail is "47-77% cheaper." This is a fresh
|
||||
reproduction to back that number with current data: three pooled 10-run evals on Claude
|
||||
(30 reps per cell), plus OpenAI and Gemini arms to test how far the claim travels.
|
||||
|
||||
## TL;DR
|
||||
|
||||
- On Claude, ponytail is **42-75% cheaper** than no-skill across Haiku, Sonnet, and Opus
|
||||
(pooled 30 reps). The published 47-77% is close but a few points optimistic at both ends:
|
||||
the reproduced floor is 42% (Opus) and the ceiling 75% (Sonnet).
|
||||
- The cost win is **Claude-specific**. On OpenAI it mostly reverses: gpt-4.1-mini is 40%
|
||||
cheaper, but gpt-5.4-mini is **26% more expensive** and the newest top model **gpt-5.5 is
|
||||
39% more expensive** and not faster. On the reasoning models the always-on ruleset (large
|
||||
input, plus extra reasoning tokens) outweighs the shorter code.
|
||||
- Latency holds on Claude: **3.1-5.8x faster**, inside the README's "3-6x". On OpenAI it is
|
||||
mixed (2.5x on gpt-4.1-mini, down to 0.9x on gpt-5.5).
|
||||
- Correctness is not hurt anywhere: ponytail scores **100%** on every Claude and OpenAI
|
||||
model tested. The no-skill baseline drops to 76% on Claude Sonnet (a real over-engineering
|
||||
bug, a dict returned instead of a bool).
|
||||
- Gemini (gemini-3.5-flash, gemini-3.1-pro-preview) is pending: the run hit the Google AI
|
||||
Studio 600/day cap and is deferred to a fresh-quota day.
|
||||
|
||||
## Method
|
||||
|
||||
Three arms (no skill, caveman, ponytail) on Claude; baseline vs ponytail on OpenAI. Five
|
||||
everyday tasks, `--repeat 10` per run. Cost comes from promptfoo API telemetry
|
||||
(`response.cost`). Per task we take the median cost across reps, then sum the five
|
||||
task-medians for the "5 tasks" figure.
|
||||
|
||||
- Claude: three runs pooled to **30 reps per cell**.
|
||||
- OpenAI: **10 reps**. Runs 2 and 3 could not be pooled because OpenAI's automatic prompt
|
||||
caching collapsed the token telemetry on identical repeated prompts (reported as
|
||||
`cached`, with `prompt`/`completion`/`cost` zeroed), so only run 1 has valid cost. The
|
||||
10-rep numbers are stable: an independent earlier 10-rep run agrees within ~4 points
|
||||
(gpt-4.1-mini 35.7% vs 39.6%, gpt-5.4-mini 28.7% vs 26.2% more expensive). Claude pooled
|
||||
cleanly because Anthropic caching is opt-in and never triggered.
|
||||
|
||||
Reproduce:
|
||||
|
||||
```bash
|
||||
npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml --env-file .env --repeat 10
|
||||
npx promptfoo@latest eval -c benchmarks/promptfooconfig.gpt-newest.yaml --env-file .env --repeat 10
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
### Claude (pooled, 30 reps, USD for 5 tasks)
|
||||
|
||||
| model | baseline | caveman | ponytail | ponytail vs baseline |
|
||||
|---|--:|--:|--:|--:|
|
||||
| Haiku | 0.0299 | 0.0139 | 0.0110 | **63.1% cheaper** |
|
||||
| Sonnet | 0.1367 | 0.0458 | 0.0348 | **74.5% cheaper** |
|
||||
| Opus | 0.1368 | 0.0724 | 0.0789 | **42.3% cheaper** |
|
||||
|
||||
**Range: 42-75% cheaper** (vs the published 47-77%). Latency 3.1-5.8x faster; ponytail
|
||||
correctness 100% on all three.
|
||||
|
||||
### OpenAI (10 reps, USD for 5 tasks)
|
||||
|
||||
| model | baseline | ponytail | ponytail vs baseline | latency | correctness |
|
||||
|---|--:|--:|--:|--:|--:|
|
||||
| gpt-4.1-mini | 0.0026 | 0.0015 | **39.6% cheaper** | 2.5x faster | 100% |
|
||||
| gpt-5.4-mini | 0.0060 | 0.0075 | **26.2% more expensive** | 1.5x faster | 100% |
|
||||
| gpt-5.5 | 0.0714 | 0.0990 | **38.7% more expensive** | 0.9x (slower) | 100% |
|
||||
|
||||
The reasoning models (gpt-5.4-mini, gpt-5.5) cost more under ponytail: the ruleset is
|
||||
re-sent as input every call and the baseline output is already terse, so the input and
|
||||
reasoning-token overhead outweighs the lines saved. Effective per-token rates derived from
|
||||
run 1: gpt-5.5 ~$5/$30 per M in/out, gpt-5.4-mini $0.75/$4.50, gpt-4.1-mini ~$0.13/$1.61.
|
||||
|
||||
### Gemini
|
||||
|
||||
Pending. The 30-rep run hit the Google AI Studio free-tier 600 requests/day cap mid-run, so
|
||||
results are polluted. Rerun on a fresh-quota day: gemini-3.5-flash (mini) and
|
||||
gemini-3.1-pro-preview (top), baseline vs ponytail.
|
||||
|
||||
## Takeaway
|
||||
|
||||
The Claude claim holds in direction but is a few points high: the reproduced, pooled range
|
||||
is **42-75% cheaper on Claude**, faster on every Claude model, with no correctness cost.
|
||||
Recommend changing the README headline from "47-77% cheaper" to **42-75% cheaper** and
|
||||
keeping the "Claude" scope, because cross-provider the picture flips: on OpenAI's reasoning
|
||||
models, including the newest top model gpt-5.5, ponytail costs more, not less. The number is
|
||||
about code generation cost on Claude, not a universal or cross-provider promise.
|
||||
|
||||
## Notes
|
||||
|
||||
- About 22 of 1350 Claude reps dropped on transient empty responses; excluded from medians,
|
||||
immaterial at this n. OpenAI runs were 100% complete.
|
||||
- Reproduce from the committed configs: `promptfooconfig.yaml` (Claude),
|
||||
`promptfooconfig.gpt-newest.yaml` (OpenAI), `promptfooconfig.gemini.yaml` (Gemini). The
|
||||
raw eval JSON is gitignored and regenerable.
|
||||
@@ -1,219 +0,0 @@
|
||||
# Agentic benchmark: does ponytail cut code without cutting safety?
|
||||
|
||||
*2026-06-18. Haiku 4.5. Real Claude Code sessions on a real open-source repo.*
|
||||
|
||||
This is a rebuilt benchmark written in direct response to Colin Eberhardt's critique in
|
||||
[issue #126](https://github.com/DietrichGebert/ponytail/issues/126). His points were fair, so
|
||||
this run is built to be able to *disprove* ponytail, not just flatter it.
|
||||
|
||||
## The critique, restated honestly
|
||||
|
||||
The original ponytail benchmark was single-shot: one prompt, one completion, count the lines.
|
||||
Colin argued, correctly, that:
|
||||
|
||||
1. **A single completion is not how a coding agent is used.** Real work is an agent editing a
|
||||
real codebase over many turns.
|
||||
2. **The baseline was a bare, chatty model.** It emitted prose, caveats, and multiple options, so
|
||||
"lines of the answer" counted commentary, not code. That inflates the baseline and flatters the
|
||||
skill. The 80–94% reductions were partly a conversational-baseline artifact.
|
||||
3. **"Prefer one-liners" might trade away safety.** If the discipline is "write less," does it drop
|
||||
input validation and error handling to get there?
|
||||
4. A short prompt ("Follow YAGNI principles, and prefer one-liner solutions") might do the same job
|
||||
as a whole skill.
|
||||
|
||||
All four are reasonable. This benchmark answers them.
|
||||
|
||||
## What changed
|
||||
|
||||
| | single-shot (old) | agentic (this) |
|
||||
|---|---|---|
|
||||
| unit of work | one prompt → one completion | a **real headless Claude Code session** in a temp workspace |
|
||||
| baseline | bare API model (emits prose + options) | the **same Claude Code agent with no skill** |
|
||||
| task | "write me X" | a real ticket against a real repo, or "implement this function" |
|
||||
| LOC counted | whole answer incl. commentary | **`git diff` added lines** of the files the agent leaves behind |
|
||||
| arms | ponytail vs bare model | baseline · ponytail · caveman · **Colin's own one-liner prompt** |
|
||||
| safety | not measured | **measured: the produced code is executed against adversarial input** |
|
||||
|
||||
The baseline here is Claude Code doing the job properly. Any difference is the skill's effect, not
|
||||
the model being chatty. That is the core of Colin's critique, and it is now controlled for.
|
||||
|
||||
### A contamination bug we found in our own numbers
|
||||
|
||||
An earlier agentic run showed a tiny ~4% gap and we nearly published it. It was wrong: ponytail and
|
||||
caveman are Claude Code **plugins** that fire a `SessionStart` hook, and that hook was firing on
|
||||
*every* arm, including the baseline, so the baseline was secretly running ponytail. Fixed by
|
||||
isolating each arm: `--setting-sources project,local` excludes the user's global plugins, and
|
||||
exactly one plugin is loaded per arm via `--plugin-dir`. We mention this because it is the kind of
|
||||
error that makes a benchmark lie, and finding it is the reason to trust the rest.
|
||||
|
||||
## Setup
|
||||
|
||||
- **Engine:** Claude Code `2.1.177`, headless (`claude -p`), `--output-format json`. Not a bare
|
||||
API model, the same product people actually use.
|
||||
- **Model:** Haiku 4.5 (`claude-haiku-4-5-20251001`). One model is enough to make the point; the
|
||||
harness supports Sonnet/Opus.
|
||||
- **Repo:** [`tiangolo/full-stack-fastapi-template`](https://github.com/fastapi/full-stack-fastapi-template)
|
||||
@ `cd83fc1` (MIT). A real, popular FastAPI + React codebase. Public and pinned, so anyone can
|
||||
reproduce.
|
||||
- **Arms:**
|
||||
- `baseline`: no skill.
|
||||
- `ponytail`: the skill, loaded as its real plugin.
|
||||
- `caveman`: a *terse-prose* skill (talks short, builds normally). A control: if ponytail's
|
||||
effect were just "be brief," caveman would match it.
|
||||
- `yagni-oneliner`: Colin's seven words: *"Follow YAGNI principles, and prefer one-liner
|
||||
solutions."* appended to the system prompt. The direct test of point (4).
|
||||
- **Isolation:** every cell gets its own fresh copy of the repo and its own fresh agent context
|
||||
(separate process, no shared history). `n=4` runs per (task, arm). Nothing carries between runs.
|
||||
- **Metric:** LOC is `git diff` added lines (comments included) of the files the agent writes.
|
||||
We do **not** run a server or a browser, agents only write code; we measure the code. (The safety
|
||||
tasks are the exception: their scorer executes the produced function directly.)
|
||||
|
||||
Two axes, because the tasks split into two kinds:
|
||||
|
||||
- **Over-build room**: open features in the real repo, where the agent chooses how much to build.
|
||||
- **Surgical room**: "implement this one function," little room to over-build, where the question
|
||||
is whether minimizing drops a *guard*.
|
||||
|
||||
## Axis 1: lines of code on real features (12 tasks)
|
||||
|
||||
Each task is a one-line ticket against the template. LOC is the mean of 4 runs.
|
||||
|
||||
**Frontend**
|
||||
|
||||
| task (ticket) | baseline | caveman | **ponytail** | yagni-oneliner |
|
||||
|---|--:|--:|--:|--:|
|
||||
| date picker | 404 | 202 | **23** | 162 |
|
||||
| color picker | 287 | 188 | **23** | 25 |
|
||||
| file dropzone | 251 | 226 | **95** | 175 |
|
||||
| multi-step wizard | 571 | 492 | **312** | 406 |
|
||||
| star rating | 103 | 95 | **70** | 101 |
|
||||
| command palette | 268 | 260 | **233** | 285 |
|
||||
|
||||
**Backend**
|
||||
|
||||
| task (ticket) | baseline | caveman | **ponytail** | yagni-oneliner |
|
||||
|---|--:|--:|--:|--:|
|
||||
| archive/unarchive item | 175 | 197 | **116** | 147 |
|
||||
| search items by title | 44 | 44 | **44** | 43 |
|
||||
| export items as CSV | 36 | 36 | **33** | 32 |
|
||||
| bulk-delete items | 33 | 29 | **26** | 24 |
|
||||
| duplicate an item | 24 | 24 | **23** | 20 |
|
||||
| count user's items | 21 | 20 | **17** | 18 |
|
||||
|
||||
What this says, including where ponytail does **not** win:
|
||||
|
||||
1. **Big wins are exactly where a native platform feature replaces a custom build.** Date picker
|
||||
−94%, color picker −92%, dropzone −62%. The baseline hand-builds a component; ponytail reaches
|
||||
for `<input type="date">`, `<input type="color">`, `<input type="file">`. This is the discipline
|
||||
working as designed, not a chatty-baseline artifact, the baseline here is real Claude Code.
|
||||
2. **On irreducible code the arms converge.** Backend CRUD endpoints and the command palette are
|
||||
near-identical across all arms. ponytail trims a little and never bloats, but it does not invent
|
||||
savings where there are none. An honest benchmark has to show this, and it does.
|
||||
3. **caveman lands between baseline and ponytail.** Terseness alone explains part of the gap but
|
||||
not most of it. The effect is the lazy-*code* discipline, not short talk.
|
||||
4. **Colin's one-liner prompt is erratic.** Brilliant on the color picker (25), but near or *above*
|
||||
baseline on the date picker (162), wizard (406), and command palette (285 > baseline's 268). The
|
||||
plugin is consistent; the seven-word prompt is not. That is the answer to point (4): the prompt
|
||||
sometimes lands and sometimes doesn't, the skill lands every time.
|
||||
|
||||
Bonus: where ponytail cuts code it is also cheaper and faster (date picker: ~$0.06 / 49s vs the
|
||||
baseline's ~$0.15 / 88s), fewer lines is fewer tokens.
|
||||
|
||||
## Axis 2: does minimizing drop a guard? (6 tasks)
|
||||
|
||||
Each task seeds a starter file and asks for one function. The safety requirement is left **implicit**,
|
||||
the way a real ticket reads. The scorer then **executes the produced function against adversarial
|
||||
input** (deterministic, stdlib-only): path traversal, SQL injection, a forged token, a malformed CSV
|
||||
row, a quota-exhausting client. The `bad` reference for each is the lazy-but-plausible version:
|
||||
correct on the happy path, unsafe on the adversarial one, exactly what a one-liner is tempted to write.
|
||||
|
||||
**Safe rate (5 security tasks × 4 runs = 20 runs per arm):**
|
||||
|
||||
| arm | safe | LOC where it matters |
|
||||
|---|--:|---|
|
||||
| baseline | 100% (20/20) | - |
|
||||
| caveman | 100% (20/20) | - |
|
||||
| **ponytail** | **100% (20/20)** | safe-path 9.5, sql-user 4.5 |
|
||||
| yagni-oneliner | **95% (19/20)** | safe-path **6** |
|
||||
|
||||
The whole thesis is in one task. On `safe-path` (join an untrusted filename onto a base directory):
|
||||
|
||||
- **yagni-oneliner** wrote the fewest lines (6) and went unsafe **once in four**, a `../../`
|
||||
filename escaped the directory.
|
||||
- **ponytail** wrote ~9.5 lines and was safe **4/4**.
|
||||
|
||||
The ~3 lines ponytail kept *were the path-traversal check*. "Write less" without judgment cuts the
|
||||
guard; ponytail's rule, *never simplify away input validation at trust boundaries*, keeps it. That
|
||||
is the difference between lazy and careless, and it is the answer to point (3).
|
||||
|
||||
Honest caveat: at Haiku scale the safety gap is small, one slip in twenty. It is a floor, not a
|
||||
dramatic result, and a deterministic check is not a proof of security. But the direction is exactly
|
||||
the design hypothesis, and the only arm that dropped a guard was the bare one-liner prompt.
|
||||
|
||||
## Summary: percent change vs baseline (all metrics)
|
||||
|
||||
Mean across each tier's tasks (every task averaged over 4 runs), relative to the no-skill baseline.
|
||||
Negative is less code / cheaper / faster.
|
||||
|
||||
**12 feature tasks** (baseline absolute, per task: 191 LOC, 349k tokens, $0.097, 69s):
|
||||
|
||||
| arm | LOC | tokens | cost | time |
|
||||
|---|--:|--:|--:|--:|
|
||||
| caveman | −20% | +7% | +3% | +2% |
|
||||
| **ponytail** | **−54%** | **−22%** | **−20%** | **−27%** |
|
||||
| yagni-oneliner | −33% | −14% | −21% | −30% |
|
||||
|
||||
**6 safety tasks** (baseline absolute, per task: 12 LOC, 104k tokens, $0.038, 22s):
|
||||
|
||||
| arm | LOC | tokens | cost | time | safe |
|
||||
|---|--:|--:|--:|--:|--:|
|
||||
| caveman | −4% | −8% | −4% | +12% | 100% |
|
||||
| **ponytail** | **−5%** | **−18%** | **−7%** | **−1%** | **100%** |
|
||||
| yagni-oneliner | −18% | −4% | −8% | +3% | **95%** |
|
||||
|
||||
Reading it:
|
||||
|
||||
- **ponytail is the only arm that cuts every metric** on the feature tasks, and the only large code
|
||||
cut (−54%). caveman writes less code but spends *more* tokens (+7%), terse output, same
|
||||
deliberation, so it is not cheaper. yagni-oneliner is cheap and fast but cuts less code than
|
||||
ponytail and is the one arm that dropped a safety guard.
|
||||
- The **−54% LOC is the across-task aggregate**; per task it runs from ~0% (irreducible backend
|
||||
CRUD) to −94% (date picker). The average is pulled down by tasks with no bloat to cut, this is the
|
||||
honest aggregate, not the cherry-picked peak.
|
||||
- On the surgical safety tasks the code is tiny for everyone (10–12 lines), so size barely moves;
|
||||
there the signal is the safe rate, where only yagni-oneliner slips.
|
||||
|
||||
## Limitations (so this can't be the next thing someone debunks)
|
||||
|
||||
- **One model.** Haiku 4.5 only. Bigger models may close the over-build gap (they need less hand-
|
||||
holding) or widen it. The harness runs Sonnet/Opus; we stopped at Haiku for cost.
|
||||
- **Safety is a floor.** Six surgical tasks, deterministic checks. It shows whether an arm drops a
|
||||
*known* guard, not that the code is secure.
|
||||
- **`yagni-oneliner` is our paraphrase** of Colin's argument, not a claim about his exact intent.
|
||||
It is the strongest short-prompt version we could write for the comparison.
|
||||
- **Nondeterminism.** `n=4`. Frontend LOC varies run to run (a custom build is 300–570 lines); the
|
||||
means are stable but not tight. Backend and safety LOC are tight.
|
||||
- **Four of 192 LOC cells** hit a Windows process-timeout bug mid-run and were force-killed; their
|
||||
LOC still counted (the files were written) but cost/time did not. Every (task, arm) kept ≥2 of 4
|
||||
runs. The bug is fixed in the harness.
|
||||
|
||||
## Conclusion
|
||||
|
||||
On a real repo, with the real agent, measured by `git diff`:
|
||||
|
||||
- ponytail **cuts 60–94% of the code** on features that have an over-build trap (custom component
|
||||
vs native input), and is a wash on code that is already minimal. It never writes more.
|
||||
- It does this **without dropping a safety guard** (100% safe), while the bare "one-liner" prompt
|
||||
was the only arm that did (95%), and was also the inconsistent one on size.
|
||||
|
||||
The original 80–94% single-shot numbers were inflated by a chatty baseline, Colin was right. The
|
||||
honest number on real tickets is "huge where there's bloat to cut, nothing where there isn't, and
|
||||
not at the cost of safety." That is a smaller and more defensible claim, and it is the one ponytail
|
||||
was actually built to make.
|
||||
|
||||
## Reproduce
|
||||
|
||||
See [`benchmarks/agentic/README.md`](../agentic/README.md). Short version: clone the template at
|
||||
`cd83fc1`, then `python run.py --selftest` (no API), then the run command in that README. Every
|
||||
workspace is preserved under `runs/<stamp>/` so any metric can be recomputed offline with
|
||||
`--rescore`.
|
||||
@@ -1,2 +0,0 @@
|
||||
description = "Show ponytail's measured impact scoreboard (less code, cost, time)"
|
||||
prompt = "Show the ponytail gain scoreboard. One shot, change nothing: do not switch mode, write flag files, or persist anything. Render the published benchmark medians (5 everyday tasks; models Haiku, Sonnet, Opus; source benchmarks/ and the README) as plain ASCII bars: Lines of code, no-skill 100% vs ponytail 6-20% (down 80-94%); Cost, no-skill 100% vs ponytail 23-53% (down 47-77%); Speed, ponytail 3-6x faster. The bar length shows the measured range, the label carries the exact figure. These are benchmark medians, not this repo. NEVER print a per-repo savings number: the unbuilt version was never written, so there is no real baseline to subtract from in a live repo. For real per-repo figures, point to /ponytail-debt (the counted shortcut ledger) and /ponytail-audit (what is still cuttable). Report only."
|
||||
@@ -1,2 +1,2 @@
|
||||
description = "Quick reference for ponytail levels, skills, and commands"
|
||||
prompt = "Show the ponytail quick reference. One shot, change nothing: do not switch mode, write flag files, or persist anything. Levels: /ponytail lite (build what's asked, name the lazier alternative in one line), /ponytail (full, the default ladder: YAGNI then stdlib then native then one line then minimum), /ponytail ultra (deletion before addition, challenges the requirement before building). Commands: /ponytail-review (over-engineering review of the current changes), /ponytail-audit (whole-repo over-engineering audit), /ponytail-debt (harvest ponytail: comments into a tracked ledger), /ponytail-gain (measured-impact scoreboard from the benchmark), /ponytail-help (this card). Deactivate with 'stop ponytail', 'normal mode', or /ponytail off; resume anytime with /ponytail. Default mode is full; change it with the PONYTAIL_DEFAULT_MODE environment variable (off|lite|full|ultra) or a config file at ~/.config/ponytail/config.json (Windows: %APPDATA%\\ponytail\\config.json) with {\"defaultMode\": \"lite\"}. Resolution order: env var, then config file, then full."
|
||||
prompt = "Show the ponytail quick reference. One shot, change nothing: do not switch mode, write flag files, or persist anything. Levels: /ponytail lite (build what's asked, name the lazier alternative in one line), /ponytail (full, the default ladder: YAGNI then stdlib then native then one line then minimum), /ponytail ultra (deletion before addition, challenges the requirement before building). Commands: /ponytail-review (over-engineering review of the current changes), /ponytail-audit (whole-repo over-engineering audit), /ponytail-debt (harvest ponytail: comments into a tracked ledger), /ponytail-help (this card). Deactivate with 'stop ponytail', 'normal mode', or /ponytail off; resume anytime with /ponytail. Default mode is full; change it with the PONYTAIL_DEFAULT_MODE environment variable (off|lite|full|ultra) or a config file at ~/.config/ponytail/config.json (Windows: %APPDATA%\\ponytail\\config.json) with {\"defaultMode\": \"lite\"}. Resolution order: env var, then config file, then full."
|
||||
|
||||
@@ -8,18 +8,17 @@ to load in a given agent.
|
||||
|
||||
| Host | Files | Notes |
|
||||
|------|-------|-------|
|
||||
| Claude Code | `.claude-plugin/plugin.json`, `commands/`, `hooks/claude-codex-hooks.json`, `hooks/` | Full plugin install with session activation, mode tracking, commands, and statusline support. |
|
||||
| Codex | `.codex-plugin/plugin.json`, `hooks/claude-codex-hooks.json`, `hooks/`, `skills/` | Plugin install with the same skills plus lifecycle hooks for activation and mode tracking. |
|
||||
| Claude Code | `.claude-plugin/`, `commands/`, `hooks/` | Full plugin install with session activation, mode tracking, commands, and statusline support. |
|
||||
| Codex | `.codex-plugin/plugin.json`, `hooks/hooks.json`, `hooks/`, `skills/` | Plugin install with the same skills plus lifecycle hooks for activation and mode tracking. |
|
||||
| OpenCode | `.opencode/plugins/ponytail.mjs`, `.opencode/command/`, `hooks/`, `skills/` | Server plugin injects the ruleset each turn via `experimental.chat.system.transform` and persists `/ponytail` switches; reuses the shared instruction builder. |
|
||||
| pi | `pi-extension/`, `skills/`, `hooks/` | Package extension: injects the ruleset each turn through the shared instruction builder and registers the `/ponytail` commands. |
|
||||
| Gemini CLI | `gemini-extension.json`, `AGENTS.md`, `commands/`, `skills/` | Extension manifest points `contextFileName` at `AGENTS.md` for always-on rules, and reuses the existing `commands/*.toml` and `skills/`, which Gemini CLI auto-discovers. The Claude/Codex hook map is not placed at Gemini's auto-discovered `hooks/hooks.json` path. |
|
||||
| Gemini CLI | `gemini-extension.json`, `AGENTS.md`, `commands/`, `skills/` | Extension manifest points `contextFileName` at `AGENTS.md` for always-on rules, and reuses the existing `commands/*.toml` and `skills/`, which Gemini CLI auto-discovers. |
|
||||
| Cursor | `.cursor/rules/ponytail.mdc` | Always-on project rule. |
|
||||
| Windsurf | `.windsurf/rules/ponytail.md` | Project rule. |
|
||||
| Cline | `.clinerules/ponytail.md` | Project rule. |
|
||||
| GitHub Copilot | `.github/copilot-instructions.md` | Repository instruction file. |
|
||||
| GitHub Copilot CLI | `.github/plugin/`, `AGENTS.md`, `.github/copilot-instructions.md`, `~/.copilot/copilot-instructions.md` | Plugin-supported (`copilot plugin marketplace add DietrichGebert/ponytail` + `copilot plugin install ponytail@ponytail`). Fallback instruction mode remains: per-project from `AGENTS.md` or `.github/copilot-instructions.md`, or globally from `~/.copilot/copilot-instructions.md` (instruction-tier, no `/ponytail` levels or hooks). |
|
||||
| Antigravity | `AGENTS.md` | Reads `AGENTS.md` at the repo root as always-on rules (like `.cursorrules`/`CLAUDE.md`); `.agents/rules/` also works for workspace rules. Instruction-tier. |
|
||||
| CodeWhale | `AGENTS.md` | Reads `AGENTS.md` from the repo root as project instructions; also reads `CLAUDE.md` and `.claude/instructions.md` as fallbacks. Instruction-tier. |
|
||||
| VS Code + Codex extension | `AGENTS.md` | The Codex extension reads `AGENTS.md` (repo root, or `~/.codex/AGENTS.md` globally). Instruction-tier; the full Codex plugin row above adds `/ponytail` levels and hooks. |
|
||||
| Kiro | `.kiro/steering/ponytail.md` | Steering rule; copy globally or into a project. |
|
||||
| Generic agents | `AGENTS.md` or `skills/*/SKILL.md` | Copy the compact rule file or load the skill files directly. |
|
||||
@@ -36,6 +35,5 @@ instructions, keep its copied rule text aligned with `AGENTS.md`.
|
||||
- `skills/ponytail-review/SKILL.md`: over-engineering review
|
||||
- `skills/ponytail-audit/SKILL.md`: whole-repo over-engineering audit
|
||||
- `skills/ponytail-debt/SKILL.md`: harvest `ponytail:` shortcuts into a tracked ledger
|
||||
- `skills/ponytail-gain/SKILL.md`: measured-impact scoreboard from the benchmark
|
||||
- `skills/ponytail-help/SKILL.md`: quick reference
|
||||
- `AGENTS.md`: compact always-on instruction set for agents without skill support
|
||||
|
||||
@@ -1,173 +0,0 @@
|
||||
# Platform-Native Solutions
|
||||
|
||||
The lazy senior dev's first question is always: *does the platform already do this?*
|
||||
|
||||
This document answers that question for the most common cases. Before reaching for a package, scan here. The platform ships with your app for free, doesn't break on updates, and was written by people whose job is exactly that problem.
|
||||
|
||||
---
|
||||
|
||||
## HTML Elements
|
||||
|
||||
Things the browser already has as a form control.
|
||||
|
||||
| You think you need | What the platform has |
|
||||
|---|---|
|
||||
| Date picker library | `<input type="date">` |
|
||||
| Time picker library | `<input type="time">` |
|
||||
| Color picker library | `<input type="color">` |
|
||||
| Range slider library | `<input type="range">` |
|
||||
| Progress bar component | `<progress value="70" max="100">` |
|
||||
| Meter/gauge component | `<meter value="0.7">` |
|
||||
| Modal/dialog library | `<dialog>` + `dialog.showModal()` |
|
||||
| Accordion/FAQ component | `<details><summary>Title</summary>…</details>` |
|
||||
| Tooltip library | `title` attribute + CSS `::before`/`::after` |
|
||||
| Searchable dropdown | `<input list="id"> <datalist id="id">` |
|
||||
| Auto-growing textarea | `field-sizing: content` (CSS) |
|
||||
| Sticky header | `position: sticky; top: 0` (CSS) |
|
||||
|
||||
---
|
||||
|
||||
## CSS Capabilities
|
||||
|
||||
Things developers reach for JavaScript to do.
|
||||
|
||||
| You think you need JS for | What CSS has |
|
||||
|---|---|
|
||||
| Responsive font size | `font-size: clamp(1rem, 2.5vw, 2rem)` |
|
||||
| Fluid spacing | `padding: clamp(1rem, 5vw, 3rem)` |
|
||||
| Dark mode | `@media (prefers-color-scheme: dark)` |
|
||||
| Reduced motion | `@media (prefers-reduced-motion: reduce)` |
|
||||
| Responsive layout without breakpoints | `grid-template-columns: repeat(auto-fill, minmax(250px, 1fr))` |
|
||||
| Component-level responsive design | `@container` queries |
|
||||
| Global design tokens / theming | CSS custom properties (`--color-primary: #7c3aed`) |
|
||||
| Smooth scroll | `scroll-behavior: smooth` |
|
||||
| Scroll-snap carousel | `scroll-snap-type: x mandatory` + `scroll-snap-align: start` |
|
||||
| Aspect ratio enforcement | `aspect-ratio: 16 / 9` |
|
||||
| Truncate text with ellipsis | `overflow: hidden; text-overflow: ellipsis; white-space: nowrap` |
|
||||
| Multi-line text clamp | `-webkit-line-clamp: 3` |
|
||||
| CSS cascade layers (style isolation) | `@layer base, components, utilities` |
|
||||
| Nested CSS selectors | Native CSS nesting (no preprocessor needed) |
|
||||
| `has()` parent selector | `:has(input:checked)` |
|
||||
|
||||
---
|
||||
|
||||
## JavaScript / Browser APIs
|
||||
|
||||
Libraries people install that the runtime already ships.
|
||||
|
||||
| You think you need | What the platform has |
|
||||
|---|---|
|
||||
| `query-string` / `qs` | `new URLSearchParams(location.search)` |
|
||||
| `lodash.clonedeep` | `structuredClone(obj)` |
|
||||
| `lodash.groupby` | `Object.groupBy(arr, fn)` |
|
||||
| `lodash.debounce` | — see debounce one-liner below |
|
||||
| `numeral` / `accounting` | `new Intl.NumberFormat("en-US", { style: "currency", currency: "USD" })` |
|
||||
| `date-fns` format | `new Intl.DateTimeFormat("en-US", { dateStyle: "long" }).format(date)` |
|
||||
| `date-fns` relative time | `new Intl.RelativeTimeFormat("en", { numeric: "auto" }).format(-3, "day")` |
|
||||
| `plural` / `i18n` plurals | `new Intl.PluralRules("en-US").select(count)` |
|
||||
| `clipboard.js` | `navigator.clipboard.writeText(text)` |
|
||||
| `uuid` (v4) | `crypto.randomUUID()` |
|
||||
| Infinite scroll library | `new IntersectionObserver(cb).observe(sentinel)` |
|
||||
| Resize listener library | `new ResizeObserver(cb).observe(element)` |
|
||||
| DOM mutation watcher | `new MutationObserver(cb).observe(el, options)` |
|
||||
| `uuid-validate` | `/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(id)` |
|
||||
| `is-online` / `connectivity check` | `navigator.onLine` + `online`/`offline` events |
|
||||
| `sharesheet` library | `navigator.share({ title, text, url })` |
|
||||
| `store.js` / `localForage` (simple case) | `localStorage.setItem(key, JSON.stringify(val))` |
|
||||
| Abort fetch on timeout | `AbortSignal.timeout(5000)` passed to `fetch` |
|
||||
| Custom event bus | `new EventTarget()` / `dispatchEvent(new CustomEvent("x", { detail }))` |
|
||||
|
||||
**Debounce one-liner** (no library):
|
||||
```js
|
||||
// ponytail: 3 lines beats a dependency
|
||||
let t;
|
||||
const debounce = (fn, ms) => (...args) => { clearTimeout(t); t = setTimeout(() => fn(...args), ms); };
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Node.js Standard Library
|
||||
|
||||
Packages that wrap Node built-ins.
|
||||
|
||||
| You think you need | What Node has |
|
||||
|---|---|
|
||||
| `mkdirp` | `fs.mkdirSync(path, { recursive: true })` |
|
||||
| `rimraf` | `fs.rmSync(path, { recursive: true, force: true })` |
|
||||
| `make-dir` | `fs.mkdirSync(path, { recursive: true })` |
|
||||
| `slash` (win paths) | `path.posix` or `path.normalize()` |
|
||||
| `uuid` (v4) | `crypto.randomUUID()` |
|
||||
| `ms` (parse duration strings) | — keep `ms`, it's genuinely useful and tiny |
|
||||
| `is-stream` | `val instanceof stream.Readable` |
|
||||
| `object-assign` | `Object.assign()` / spread |
|
||||
| `array-uniq` | `[...new Set(arr)]` |
|
||||
| `array-flatten` | `arr.flat(Infinity)` |
|
||||
| `flat` | `arr.flat(depth)` |
|
||||
| `path-exists` | `fs.existsSync(path)` |
|
||||
| `load-json-file` | `JSON.parse(fs.readFileSync(path, "utf8"))` |
|
||||
| `write-json-file` | `fs.writeFileSync(path, JSON.stringify(obj, null, 2))` |
|
||||
| `pkg-dir` | `path.resolve(__dirname, "..")` / `import.meta.dirname` |
|
||||
|
||||
---
|
||||
|
||||
## Python Standard Library
|
||||
|
||||
Packages that wrap what Python already ships.
|
||||
|
||||
| You think you need | What Python has |
|
||||
|---|---|
|
||||
| `python-dateutil` (basic parsing) | `datetime.fromisoformat()` (Python 3.7+) |
|
||||
| `pytz` | `zoneinfo.ZoneInfo("America/New_York")` (Python 3.9+) |
|
||||
| `attrs` (simple data classes) | `@dataclass` |
|
||||
| `six` | — drop it, Python 2 is gone |
|
||||
| `pathlib2` | `pathlib.Path` (built-in since Python 3.4) |
|
||||
| `enum34` | `enum.Enum` (built-in since Python 3.4) |
|
||||
| `typing_extensions` (common types) | `from __future__ import annotations` + built-in generics |
|
||||
| `simplejson` (basic use) | `json` (stdlib) |
|
||||
| `requests` (simple GET) | `urllib.request.urlopen(url)` — `requests` for anything real |
|
||||
| `click` (single command) | `argparse` (stdlib) |
|
||||
| `mergedeep` | `dict \| other_dict` (Python 3.9+) |
|
||||
| `more-itertools` (basic) | `itertools` (stdlib): `chain`, `islice`, `groupby`, `product` |
|
||||
| `toolz` (basic) | `functools`: `lru_cache`, `partial`, `reduce` |
|
||||
| `tabulate` (dev/debug only) | `pprint.pprint()` for quick inspection |
|
||||
|
||||
---
|
||||
|
||||
## Database
|
||||
|
||||
Things the application layer implements that the database already does.
|
||||
|
||||
| You think you need app code for | What the database has |
|
||||
|---|---|
|
||||
| Pagination offset/limit | `LIMIT 20 OFFSET 40` |
|
||||
| Running totals | `SUM(...) OVER (ORDER BY date)` (window function) |
|
||||
| Rank within group | `RANK() OVER (PARTITION BY category ORDER BY score DESC)` |
|
||||
| Pivot / cross-tab | `FILTER (WHERE ...)` + conditional aggregation |
|
||||
| Deduplication | `SELECT DISTINCT` / `ON CONFLICT DO NOTHING` |
|
||||
| Soft-delete filtering | Generated column + partial index |
|
||||
| Tree traversal | Recursive CTE (`WITH RECURSIVE`) |
|
||||
| Full-text search (basic) | `tsvector` / `MATCH AGAINST` / `FTS5` |
|
||||
| JSON storage + query | `jsonb` (Postgres) / `JSON_EXTRACT` (SQLite/MySQL) |
|
||||
| UUID generation | `gen_random_uuid()` (Postgres) / `UUID()` (MySQL) |
|
||||
| Timestamps on insert/update | `DEFAULT now()` + trigger or `ON UPDATE CURRENT_TIMESTAMP` |
|
||||
| Enforce uniqueness | `UNIQUE` constraint — not application-level checks |
|
||||
| Enforce referential integrity | `FOREIGN KEY` — not application-level checks |
|
||||
| Enforce value ranges | `CHECK (price > 0)` — not application-level validation |
|
||||
|
||||
---
|
||||
|
||||
## The Pattern
|
||||
|
||||
Across every layer, the pattern is the same:
|
||||
|
||||
```
|
||||
Platform team spends years solving the problem.
|
||||
Package author wraps it.
|
||||
You install the wrapper.
|
||||
The wrapper goes unmaintained.
|
||||
You debug the wrapper.
|
||||
```
|
||||
|
||||
Skip the wrapper. The platform ships with your app for free.
|
||||
|
||||
When the native solution is genuinely insufficient (old browser support, edge cases it doesn't handle, ergonomics that matter at scale), the library earns its place. Install it then — not before.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Examples
|
||||
|
||||
Real model output, verbatim from benchmark runs — the same task answered by the same model
|
||||
with no skill (`## Without Ponytail`) and with ponytail (`## With Ponytail`), so you can
|
||||
compare side by side. Model: Claude Haiku 4.5, temperature 1, source `benchmarks/output.json`.
|
||||
|
||||
These are not hand-written. Reproduce them yourself:
|
||||
`npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`. Method, all three models, and
|
||||
median-of-10 numbers: [../benchmarks/](../benchmarks/).
|
||||
|
||||
| Example | Without (LOC) | With (LOC) |
|
||||
|---|--:|--:|
|
||||
| [Email Validation](email-validation.md) | 75 | 3 |
|
||||
| [Debounce](debounce.md) | 116 | 10 |
|
||||
| [CSV Sum](csv-sum.md) | 20 | 3 |
|
||||
| [Countdown Timer](react-countdown.md) | 267 | 9 |
|
||||
| [Rate Limiting](rate-limit.md) | 128 | 10 |
|
||||
@@ -0,0 +1,68 @@
|
||||
# API Endpoint
|
||||
|
||||
**Task:** "Add an endpoint that returns a user by id."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```
|
||||
app/
|
||||
├── controllers/user_controller.py
|
||||
├── services/user_service.py
|
||||
├── repositories/user_repository.py
|
||||
├── schemas/user_schemas.py
|
||||
└── exceptions/user_exceptions.py
|
||||
```
|
||||
|
||||
```python
|
||||
# schemas/user_schemas.py
|
||||
class UserResponseDTO(BaseModel):
|
||||
id: int
|
||||
name: str
|
||||
email: str
|
||||
|
||||
# repositories/user_repository.py
|
||||
class UserRepository:
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
def find_by_id(self, user_id: int) -> User | None:
|
||||
return self.db.get(User, user_id)
|
||||
|
||||
# services/user_service.py
|
||||
class UserService:
|
||||
def __init__(self, repo: UserRepository):
|
||||
self.repo = repo
|
||||
def get_user(self, user_id: int) -> User:
|
||||
user = self.repo.find_by_id(user_id)
|
||||
if user is None:
|
||||
raise UserNotFoundError(user_id)
|
||||
return user
|
||||
|
||||
# controllers/user_controller.py
|
||||
@router.get("/users/{user_id}", response_model=UserResponseDTO)
|
||||
def get_user(user_id: int, service: UserService = Depends(get_user_service)):
|
||||
try:
|
||||
return service.get_user(user_id)
|
||||
except UserNotFoundError:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
```
|
||||
|
||||
Five files, three classes, a custom exception, and a dependency-injection chain, wrapping one database call.
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```python
|
||||
# ponytail: drop the layers; keep the response schema, it whitelists what leaves the API
|
||||
class UserOut(BaseModel):
|
||||
id: int
|
||||
name: str
|
||||
email: str
|
||||
|
||||
@app.get("/users/{user_id}", response_model=UserOut)
|
||||
def get_user(user_id: int, db: Session = Depends(get_db)):
|
||||
user = db.get(User, user_id)
|
||||
if not user:
|
||||
raise HTTPException(404)
|
||||
return user
|
||||
```
|
||||
|
||||
**5 files → 9 lines.** The repository, service, and custom exception were ceremony. The response schema was not: it whitelists which fields leave the API, so it stays. Returning the raw ORM model (`return user`) would leak every column, including the ones you never meant to expose. That is the line ponytail draws, and it is the same one the skill draws in "when NOT to be lazy": cut the layers, keep the trust boundary. Add a service layer when a second caller shows up, if it ever does.
|
||||
@@ -0,0 +1,74 @@
|
||||
# Caching System
|
||||
|
||||
**Task:** "We should cache these API responses."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```python
|
||||
import time
|
||||
import threading
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Optional
|
||||
|
||||
|
||||
@dataclass
|
||||
class CacheEntry:
|
||||
value: Any
|
||||
expires_at: float
|
||||
|
||||
|
||||
class TTLCache:
|
||||
"""Thread-safe LRU cache with per-entry TTL and max size."""
|
||||
|
||||
def __init__(self, max_size: int = 1000, default_ttl: float = 300.0):
|
||||
self._store: OrderedDict[str, CacheEntry] = OrderedDict()
|
||||
self._lock = threading.Lock()
|
||||
self.max_size = max_size
|
||||
self.default_ttl = default_ttl
|
||||
self.hits = 0
|
||||
self.misses = 0
|
||||
|
||||
def get(self, key: str) -> Optional[Any]:
|
||||
with self._lock:
|
||||
entry = self._store.get(key)
|
||||
if entry is None or entry.expires_at < time.monotonic():
|
||||
self.misses += 1
|
||||
self._store.pop(key, None)
|
||||
return None
|
||||
self._store.move_to_end(key)
|
||||
self.hits += 1
|
||||
return entry.value
|
||||
|
||||
def set(self, key: str, value: Any, ttl: Optional[float] = None) -> None:
|
||||
with self._lock:
|
||||
if len(self._store) >= self.max_size:
|
||||
self._store.popitem(last=False)
|
||||
self._store[key] = CacheEntry(
|
||||
value, time.monotonic() + (ttl or self.default_ttl)
|
||||
)
|
||||
|
||||
# ... plus invalidation, stats endpoint, and unit tests for all of it
|
||||
```
|
||||
|
||||
120 lines of custom infrastructure for a problem nobody has measured yet.
|
||||
|
||||
## With Ponytail
|
||||
|
||||
First question: **do you actually need a cache?**
|
||||
|
||||
- **Unsure?** Ship without it. Add it when you measure the problem. (YAGNI)
|
||||
- **Pure function, hot path?** The standard library has it:
|
||||
|
||||
```python
|
||||
# ponytail: stdlib covers this
|
||||
from functools import lru_cache
|
||||
|
||||
@lru_cache(maxsize=1000)
|
||||
def fetch(key): ...
|
||||
```
|
||||
|
||||
- **Real distributed caching needs?** Use Redis / memcached / your platform's
|
||||
cache. Infrastructure problems get infrastructure, not a homemade class.
|
||||
|
||||
**120 lines → 0–3 lines.** The fastest cache is the one you didn't have to debug.
|
||||
@@ -1,71 +0,0 @@
|
||||
# CSV Sum
|
||||
|
||||
**Task:** "Write Python code that reads sales.csv and sums the 'amount' column."
|
||||
|
||||
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
## Without Ponytail — 20 lines of code
|
||||
|
||||
# Python code to read sales.csv and sum the 'amount' column
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
# Read the CSV file
|
||||
df = pd.read_csv('sales.csv')
|
||||
|
||||
# Sum the 'amount' column
|
||||
total_amount = df['amount'].sum()
|
||||
|
||||
# Print the result
|
||||
print(f"Total amount: ${total_amount:,.2f}")
|
||||
```
|
||||
|
||||
## Alternative methods:
|
||||
|
||||
**Using CSV module (without pandas):**
|
||||
```python
|
||||
import csv
|
||||
|
||||
total_amount = 0
|
||||
|
||||
with open('sales.csv', 'r') as file:
|
||||
reader = csv.DictReader(file)
|
||||
for row in reader:
|
||||
total_amount += float(row['amount'])
|
||||
|
||||
print(f"Total amount: ${total_amount:,.2f}")
|
||||
```
|
||||
|
||||
**Using pandas with error handling:**
|
||||
```python
|
||||
import pandas as pd
|
||||
|
||||
try:
|
||||
df = pd.read_csv('sales.csv')
|
||||
total_amount = df['amount'].sum()
|
||||
print(f"Total amount: ${total_amount:,.2f}")
|
||||
except FileNotFoundError:
|
||||
print("Error: sales.csv not found")
|
||||
except KeyError:
|
||||
print("Error: 'amount' column not found")
|
||||
```
|
||||
|
||||
**The pandas method is recommended** because it's:
|
||||
- More concise and readable
|
||||
- Handles data types automatically
|
||||
- Provides built-in error handling
|
||||
- Efficient for large files
|
||||
|
||||
## With Ponytail — 3 lines of code
|
||||
|
||||
```python
|
||||
import csv
|
||||
|
||||
total = sum(float(row['amount']) for row in csv.DictReader(open('sales.csv')))
|
||||
print(total)
|
||||
```
|
||||
|
||||
Skipped: pandas, error handling, file closing — add when the CSV is large, malformed, or you need more analysis.
|
||||
|
||||
**20 → 3 lines of code** — same model, same prompt.
|
||||
@@ -0,0 +1,48 @@
|
||||
# Date Picker
|
||||
|
||||
**Task:** "Add a date picker to this form."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```bash
|
||||
npm install flatpickr
|
||||
```
|
||||
|
||||
```jsx
|
||||
import flatpickr from "flatpickr";
|
||||
import "flatpickr/dist/flatpickr.min.css";
|
||||
import { useEffect, useRef } from "react";
|
||||
|
||||
export default function DatePicker({ value, onChange, minDate, maxDate }) {
|
||||
const inputRef = useRef(null);
|
||||
const instanceRef = useRef(null);
|
||||
|
||||
useEffect(() => {
|
||||
instanceRef.current = flatpickr(inputRef.current, {
|
||||
defaultDate: value,
|
||||
minDate,
|
||||
maxDate,
|
||||
dateFormat: "Y-m-d",
|
||||
onChange: ([date]) => onChange(date),
|
||||
});
|
||||
return () => instanceRef.current?.destroy();
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
instanceRef.current?.setDate(value, false);
|
||||
}, [value]);
|
||||
|
||||
return <input ref={inputRef} className="date-picker" />;
|
||||
}
|
||||
```
|
||||
|
||||
One dependency, one wrapper component, two `useEffect` hooks, a cleanup function, and a CSS import, to pick a date.
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```html
|
||||
<!-- ponytail: browser has one -->
|
||||
<input type="date">
|
||||
```
|
||||
|
||||
**1 dependency + 30 lines → 0 dependencies + 1 line.** Native, accessible, localized, keyboard-navigable, mobile-friendly. The browser team already did the work.
|
||||
@@ -1,211 +0,0 @@
|
||||
# Debounce
|
||||
|
||||
**Task:** "Add debounce to a search input in vanilla JavaScript. It currently fires an API call on every keystroke."
|
||||
|
||||
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
## Without Ponytail — 116 lines of code
|
||||
|
||||
# Debounce Search Input
|
||||
|
||||
Here's a practical implementation to prevent excessive API calls:
|
||||
|
||||
## Basic Debounce Function
|
||||
|
||||
```javascript
|
||||
function debounce(func, delay) {
|
||||
let timeoutId;
|
||||
return function (...args) {
|
||||
clearTimeout(timeoutId);
|
||||
timeoutId = setTimeout(() => func(...args), delay);
|
||||
};
|
||||
}
|
||||
|
||||
// Usage
|
||||
const searchInput = document.getElementById('search');
|
||||
const resultsContainer = document.getElementById('results');
|
||||
|
||||
const handleSearch = async (query) => {
|
||||
if (!query.trim()) {
|
||||
resultsContainer.innerHTML = '';
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const response = await fetch(`/api/search?q=${encodeURIComponent(query)}`);
|
||||
const data = await response.json();
|
||||
displayResults(data);
|
||||
} catch (error) {
|
||||
console.error('Search failed:', error);
|
||||
}
|
||||
};
|
||||
|
||||
// Debounce the search function (300ms delay)
|
||||
const debouncedSearch = debounce(handleSearch, 300);
|
||||
|
||||
searchInput.addEventListener('input', (e) => {
|
||||
debouncedSearch(e.target.value);
|
||||
});
|
||||
```
|
||||
|
||||
## Enhanced Version with Loading State
|
||||
|
||||
```javascript
|
||||
const searchInput = document.getElementById('search');
|
||||
const resultsContainer = document.getElementById('results');
|
||||
const loadingIndicator = document.getElementById('loading');
|
||||
|
||||
const handleSearch = async (query) => {
|
||||
if (!query.trim()) {
|
||||
resultsContainer.innerHTML = '';
|
||||
loadingIndicator.style.display = 'none';
|
||||
return;
|
||||
}
|
||||
|
||||
loadingIndicator.style.display = 'block';
|
||||
|
||||
try {
|
||||
const response = await fetch(`/api/search?q=${encodeURIComponent(query)}`);
|
||||
if (!response.ok) throw new Error('Search failed');
|
||||
|
||||
const data = await response.json();
|
||||
displayResults(data);
|
||||
} catch (error) {
|
||||
resultsContainer.innerHTML = '<p class="error">Search failed. Try again.</p>';
|
||||
console.error('Search error:', error);
|
||||
} finally {
|
||||
loadingIndicator.style.display = 'none';
|
||||
}
|
||||
};
|
||||
|
||||
const debouncedSearch = debounce(handleSearch, 300);
|
||||
|
||||
searchInput.addEventListener('input', (e) => {
|
||||
debouncedSearch(e.target.value);
|
||||
});
|
||||
|
||||
function displayResults(data) {
|
||||
resultsContainer.innerHTML = data
|
||||
.map(item => `<div class="result">${item.title}</div>`)
|
||||
.join('');
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced: Debounce with Cancel & Immediate Options
|
||||
|
||||
```javascript
|
||||
function debounce(func, delay, options = {}) {
|
||||
let timeoutId;
|
||||
let lastArgs;
|
||||
|
||||
const debounced = function (...args) {
|
||||
lastArgs = args;
|
||||
clearTimeout(timeoutId);
|
||||
|
||||
if (options.immediate && !timeoutId) {
|
||||
func(...args);
|
||||
}
|
||||
|
||||
timeoutId = setTimeout(() => {
|
||||
if (!options.immediate) {
|
||||
func(...lastArgs);
|
||||
}
|
||||
timeoutId = null;
|
||||
}, delay);
|
||||
};
|
||||
|
||||
// Allow manual cancellation
|
||||
debounced.cancel = () => clearTimeout(timeoutId);
|
||||
|
||||
return debounced;
|
||||
}
|
||||
|
||||
// Usage with options
|
||||
const debouncedSearch = debounce(handleSearch, 300, { immediate: false });
|
||||
|
||||
// Cancel pending search if needed
|
||||
searchInput.addEventListener('blur', () => {
|
||||
debouncedSearch.cancel();
|
||||
});
|
||||
```
|
||||
|
||||
## HTML Example
|
||||
|
||||
```html
|
||||
<div class="search-container">
|
||||
<input
|
||||
type="text"
|
||||
id="search"
|
||||
placeholder="Search..."
|
||||
autocomplete="off"
|
||||
>
|
||||
<div id="loading" style="display: none;">Loading...</div>
|
||||
<div id="results"></div>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.search-container {
|
||||
max-width: 500px;
|
||||
margin: 20px auto;
|
||||
}
|
||||
|
||||
#search {
|
||||
width: 100%;
|
||||
padding: 10px;
|
||||
font-size: 16px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
#results {
|
||||
margin-top: 10px;
|
||||
}
|
||||
|
||||
.result {
|
||||
padding: 10px;
|
||||
border-bottom: 1px solid #eee;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.result:hover {
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
|
||||
.error {
|
||||
color: red;
|
||||
padding: 10px;
|
||||
}
|
||||
</style>
|
||||
```
|
||||
|
||||
## Key Benefits
|
||||
|
||||
| Feature | Benefit |
|
||||
|---------|---------|
|
||||
| **Debounce** | Reduces API calls from 100+ to 1-2 per search |
|
||||
| **300ms delay** | Balances responsiveness with performance |
|
||||
| **Loading state** | Provides user feedback |
|
||||
| **Error handling** | Graceful failure management |
|
||||
| **Cancel method** | Cleanup when needed |
|
||||
|
||||
**Typical keystroke reduction:** 10 characters = 10 API calls → 1-2 API calls with debounce ✅
|
||||
|
||||
## With Ponytail — 10 lines of code
|
||||
|
||||
```javascript
|
||||
const searchInput = document.querySelector('input[type="search"]');
|
||||
let debounceTimer;
|
||||
|
||||
searchInput.addEventListener('input', (e) => {
|
||||
clearTimeout(debounceTimer);
|
||||
debounceTimer = setTimeout(() => {
|
||||
fetch(`/api/search?q=${encodeURIComponent(e.target.value)}`)
|
||||
.then(r => r.json())
|
||||
.then(data => console.log(data)); // replace with your render
|
||||
}, 300);
|
||||
});
|
||||
```
|
||||
|
||||
Skipped: debounce utility function, class wrapper, config object — `setTimeout` + `clearTimeout` is the debounce. Add a utility when you need it on 3+ inputs.
|
||||
|
||||
**116 → 10 lines of code** — same model, same prompt.
|
||||
@@ -1,31 +0,0 @@
|
||||
# Deep Clone
|
||||
|
||||
**Task:** "Deep clone this object."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```bash
|
||||
npm install lodash
|
||||
```
|
||||
|
||||
```js
|
||||
import { cloneDeep } from "lodash";
|
||||
|
||||
const copy = cloneDeep(original);
|
||||
```
|
||||
|
||||
Or the classic hack:
|
||||
|
||||
```js
|
||||
// fragile: loses Date, undefined, Map, Set, circular refs, functions
|
||||
const copy = JSON.parse(JSON.stringify(original));
|
||||
```
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```js
|
||||
// ponytail: structuredClone does this
|
||||
const copy = structuredClone(original);
|
||||
```
|
||||
|
||||
**1 dependency (or a fragile hack) → 1 built-in.** `structuredClone` handles `Date`, `Map`, `Set`, `ArrayBuffer`, `RegExp`, circular references, and more — everything `JSON.parse/stringify` silently drops. Available in every browser since 2022 and Node.js since v17. Pull lodash in when you need the rest of it, not for one function.
|
||||
+33
-138
@@ -1,156 +1,51 @@
|
||||
# Email Validation
|
||||
|
||||
**Task:** "Write me a Python function that validates email addresses."
|
||||
**Task:** "Validate an email address in Python."
|
||||
|
||||
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
## Without Ponytail — 75 lines of code
|
||||
|
||||
# Email Validation Function
|
||||
|
||||
Here's a practical Python function that validates email addresses:
|
||||
## Without Ponytail
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
EMAIL_PATTERN = re.compile(
|
||||
r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
)
|
||||
|
||||
class EmailValidator:
|
||||
"""Validates email addresses against RFC-like rules."""
|
||||
|
||||
def __init__(self, pattern: re.Pattern = EMAIL_PATTERN):
|
||||
self.pattern = pattern
|
||||
|
||||
def validate(self, email: str) -> bool:
|
||||
if not isinstance(email, str):
|
||||
raise TypeError("email must be a string")
|
||||
email = email.strip()
|
||||
if not email:
|
||||
return False
|
||||
return bool(self.pattern.match(email))
|
||||
|
||||
|
||||
def validate_email(email: str) -> bool:
|
||||
"""
|
||||
Validates an email address using a regular expression.
|
||||
|
||||
Args:
|
||||
email (str): The email address to validate
|
||||
|
||||
Returns:
|
||||
bool: True if valid, False otherwise
|
||||
"""
|
||||
# Regular expression pattern for email validation
|
||||
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
|
||||
return re.match(pattern, email) is not None
|
||||
|
||||
|
||||
# Test cases
|
||||
if __name__ == "__main__":
|
||||
test_emails = [
|
||||
"user@example.com", # Valid
|
||||
"john.doe@company.co.uk", # Valid
|
||||
"test+tag@domain.org", # Valid
|
||||
"invalid.email@", # Invalid
|
||||
"missing@domain", # Invalid
|
||||
"@nodomain.com", # Invalid
|
||||
"spaces in@email.com", # Invalid
|
||||
"user@domain..com", # Invalid (double dot)
|
||||
]
|
||||
|
||||
for email in test_emails:
|
||||
result = validate_email(email)
|
||||
print(f"{email:<30} -> {result}")
|
||||
"""Convenience wrapper around EmailValidator."""
|
||||
return EmailValidator().validate(email)
|
||||
```
|
||||
|
||||
## More Robust Version (with additional checks)
|
||||
A class, a wrapper, a regex that still rejects valid addresses and accepts invalid ones. Regex cannot validate email. Only a delivery attempt can.
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def validate_email_advanced(email: str) -> tuple[bool, str]:
|
||||
"""
|
||||
Advanced email validation with detailed feedback.
|
||||
|
||||
Args:
|
||||
email (str): The email address to validate
|
||||
|
||||
Returns:
|
||||
tuple: (is_valid, message)
|
||||
"""
|
||||
# Basic checks
|
||||
if not email or not isinstance(email, str):
|
||||
return False, "Email must be a non-empty string"
|
||||
|
||||
email = email.strip()
|
||||
|
||||
if len(email) > 254:
|
||||
return False, "Email is too long (max 254 characters)"
|
||||
|
||||
# Regular expression pattern
|
||||
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
|
||||
if not re.match(pattern, email):
|
||||
return False, "Email format is invalid"
|
||||
|
||||
# Additional checks
|
||||
local_part, domain = email.rsplit('@', 1)
|
||||
|
||||
if len(local_part) > 64:
|
||||
return False, "Local part is too long (max 64 characters)"
|
||||
|
||||
if local_part.startswith('.') or local_part.endswith('.'):
|
||||
return False, "Local part cannot start or end with a dot"
|
||||
|
||||
if '..' in email:
|
||||
return False, "Email cannot contain consecutive dots"
|
||||
|
||||
return True, "Email is valid"
|
||||
|
||||
|
||||
# Test the advanced version
|
||||
if __name__ == "__main__":
|
||||
test_emails = [
|
||||
"user@example.com",
|
||||
"invalid..email@domain.com",
|
||||
".invalid@domain.com",
|
||||
"user@domain",
|
||||
]
|
||||
|
||||
for email in test_emails:
|
||||
is_valid, message = validate_email_advanced(email)
|
||||
print(f"{email:<35} -> {is_valid:5} ({message})")
|
||||
# ponytail: good enough, real validation is sending the mail
|
||||
"@" in email and "." in email.split("@")[-1]
|
||||
```
|
||||
|
||||
## Using a Third-Party Library (Recommended for Production)
|
||||
Or, if it must be thorough, the standard library has it:
|
||||
|
||||
```python
|
||||
from email_validator import validate_email, EmailNotValidError
|
||||
|
||||
def validate_email_production(email: str) -> bool:
|
||||
"""
|
||||
Validates email using the email_validator library.
|
||||
More reliable for production use.
|
||||
|
||||
Install with: pip install email-validator
|
||||
"""
|
||||
try:
|
||||
# Validate and normalize the email
|
||||
valid = validate_email(email)
|
||||
return True
|
||||
except EmailNotValidError:
|
||||
return False
|
||||
|
||||
|
||||
# Usage
|
||||
if __name__ == "__main__":
|
||||
print(validate_email_production("user@example.com")) # True
|
||||
print(validate_email_production("invalid@")) # False
|
||||
# ponytail: stdlib covers this
|
||||
from email.utils import parseaddr
|
||||
"@" in parseaddr(email)[1]
|
||||
```
|
||||
|
||||
## Comparison
|
||||
|
||||
| Method | Pros | Cons |
|
||||
|--------|------|------|
|
||||
| **Regex (Simple)** | Fast, no dependencies | Limited validation |
|
||||
| **Regex (Advanced)** | More thorough | Still not RFC-compliant |
|
||||
| **email-validator** | RFC-compliant, reliable | External dependency |
|
||||
|
||||
**Recommendation**: Use the `email-validator` library for production applications, as it properly handles RFC 5321/5322 standards. Use the simple regex version for quick scripts or when you want to avoid dependencies.
|
||||
|
||||
## With Ponytail — 3 lines of code
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def is_valid_email(email: str) -> bool:
|
||||
return bool(re.match(r'^[^@]+@[^@]+\.[^@]+$', email))
|
||||
```
|
||||
|
||||
Skipped: RFC 5322 parser, DNS MX lookup, confirmation email. Add when you actually need to reject `user+tag@sub.domain.co.uk` or catch typos — until then, this catches 99% of "oops I fat-fingered it" cases.
|
||||
|
||||
**75 → 3 lines of code** — same model, same prompt.
|
||||
**27 lines → 1 line.** And the honest answer: let the confirmation email reject it. That's what confirmation emails are for.
|
||||
|
||||
@@ -1,35 +0,0 @@
|
||||
# Group By
|
||||
|
||||
**Task:** "Group this array of objects by a key."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```bash
|
||||
npm install lodash
|
||||
```
|
||||
|
||||
```js
|
||||
import { groupBy } from "lodash";
|
||||
|
||||
const byStatus = groupBy(orders, "status");
|
||||
// → { pending: [...], shipped: [...], delivered: [...] }
|
||||
```
|
||||
|
||||
Or the hand-rolled version:
|
||||
|
||||
```js
|
||||
const byStatus = orders.reduce((acc, order) => {
|
||||
(acc[order.status] ??= []).push(order);
|
||||
return acc;
|
||||
}, {});
|
||||
```
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```js
|
||||
// ponytail: Object.groupBy does this
|
||||
const byStatus = Object.groupBy(orders, order => order.status);
|
||||
// → { pending: [...], shipped: [...], delivered: [...] }
|
||||
```
|
||||
|
||||
**1 dependency (or a reduce) → 1 built-in.** `Object.groupBy` shipped in Chrome 117, Firefox 119, Safari 17.4, Node.js 21. If you need a `Map` instead of a plain object: `Map.groupBy(orders, o => o.status)`. Check your target runtime; if you need IE11 or old Node, the `reduce` one-liner is still the right call — not lodash.
|
||||
@@ -1,58 +0,0 @@
|
||||
# Infinite Scroll
|
||||
|
||||
**Task:** "Load more items when the user scrolls to the bottom."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```bash
|
||||
npm install react-infinite-scroll-component
|
||||
```
|
||||
|
||||
```jsx
|
||||
import InfiniteScroll from "react-infinite-scroll-component";
|
||||
|
||||
export function Feed({ items, fetchMore, hasMore }) {
|
||||
return (
|
||||
<InfiniteScroll
|
||||
dataLength={items.length}
|
||||
next={fetchMore}
|
||||
hasMore={hasMore}
|
||||
loader={<Spinner />}
|
||||
endMessage={<p>No more items</p>}
|
||||
scrollThreshold={0.9}
|
||||
>
|
||||
{items.map(item => <Card key={item.id} item={item} />)}
|
||||
</InfiniteScroll>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
A dependency to watch scroll position and fire a callback.
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```jsx
|
||||
// ponytail: IntersectionObserver does this, no scroll listener needed
|
||||
import { useEffect, useRef } from "react";
|
||||
|
||||
export function Feed({ items, fetchMore, hasMore }) {
|
||||
const sentinel = useRef(null);
|
||||
|
||||
useEffect(() => {
|
||||
const observer = new IntersectionObserver(([entry]) => {
|
||||
if (entry.isIntersecting && hasMore) fetchMore();
|
||||
});
|
||||
if (sentinel.current) observer.observe(sentinel.current);
|
||||
return () => observer.disconnect();
|
||||
}, [hasMore, fetchMore]);
|
||||
|
||||
return (
|
||||
<>
|
||||
{items.map(item => <Card key={item.id} item={item} />)}
|
||||
<div ref={sentinel} />
|
||||
</>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
**1 dependency → 0 dependencies.** `IntersectionObserver` fires only when the sentinel enters the viewport — no scroll event, no throttling, no jank. Ships in every browser. The library wraps exactly this API.
|
||||
@@ -1,62 +0,0 @@
|
||||
# Modal Dialog
|
||||
|
||||
**Task:** "Add a modal dialog for the delete confirmation."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```bash
|
||||
npm install @radix-ui/react-dialog
|
||||
# or: npm install react-modal
|
||||
```
|
||||
|
||||
```jsx
|
||||
import * as Dialog from "@radix-ui/react-dialog";
|
||||
import { useState } from "react";
|
||||
|
||||
export function DeleteModal({ onConfirm, onCancel }) {
|
||||
return (
|
||||
<Dialog.Root>
|
||||
<Dialog.Trigger asChild>
|
||||
<button className="btn-danger">Delete</button>
|
||||
</Dialog.Trigger>
|
||||
<Dialog.Portal>
|
||||
<Dialog.Overlay className="dialog-overlay" />
|
||||
<Dialog.Content className="dialog-content">
|
||||
<Dialog.Title>Confirm deletion</Dialog.Title>
|
||||
<Dialog.Description>This action cannot be undone.</Dialog.Description>
|
||||
<div className="dialog-actions">
|
||||
<Dialog.Close asChild>
|
||||
<button onClick={onCancel}>Cancel</button>
|
||||
</Dialog.Close>
|
||||
<button className="btn-danger" onClick={onConfirm}>Delete</button>
|
||||
</div>
|
||||
</Dialog.Content>
|
||||
</Dialog.Portal>
|
||||
</Dialog.Root>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
A dependency, a portal, an overlay, a root, a trigger, a content wrapper — to show a box with two buttons.
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```html
|
||||
<!-- ponytail: browser has one, with focus trapping and backdrop built in -->
|
||||
<dialog id="confirm-delete">
|
||||
<p>This action cannot be undone.</p>
|
||||
<button id="cancel">Cancel</button>
|
||||
<button id="confirm">Delete</button>
|
||||
</dialog>
|
||||
```
|
||||
|
||||
```js
|
||||
const dialog = document.getElementById("confirm-delete");
|
||||
document.getElementById("cancel").onclick = () => dialog.close();
|
||||
document.getElementById("confirm").onclick = () => { onConfirm(); dialog.close(); };
|
||||
|
||||
// Open it:
|
||||
dialog.showModal();
|
||||
```
|
||||
|
||||
**1 dependency + 30 lines → 0 dependencies + 8 lines.** The native `<dialog>` traps focus automatically, closes on Escape, renders a backdrop via `::backdrop`, and is accessible by default. All browsers since 2022. The library was solving a problem the platform solved.
|
||||
@@ -1,37 +0,0 @@
|
||||
# Number Formatting
|
||||
|
||||
**Task:** "Format numbers as currency and with thousand separators."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```bash
|
||||
npm install numeral
|
||||
# or: npm install accounting
|
||||
```
|
||||
|
||||
```js
|
||||
import numeral from "numeral";
|
||||
|
||||
numeral(1234567.89).format("$1,234.00"); // "$1,234,567.89"
|
||||
numeral(0.745).format("0.0%"); // "74.5%"
|
||||
numeral(1500).format("0.0a"); // "1.5k"
|
||||
```
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```js
|
||||
// ponytail: Intl.NumberFormat does this, locale-aware
|
||||
new Intl.NumberFormat("en-US", { style: "currency", currency: "USD" })
|
||||
.format(1234567.89);
|
||||
// → "$1,234,567.89"
|
||||
|
||||
new Intl.NumberFormat("en-US", { style: "percent" })
|
||||
.format(0.745);
|
||||
// → "74.5%"
|
||||
|
||||
new Intl.NumberFormat("en-US", { notation: "compact" })
|
||||
.format(1500);
|
||||
// → "1.5K"
|
||||
```
|
||||
|
||||
**1 dependency → 0 dependencies.** `Intl.NumberFormat` is built into every JS runtime, handles every locale correctly, and gets currency symbols, decimal separators, and grouping right for any market without a lookup table. A library that hardcodes formats will always be wrong for someone.
|
||||
@@ -1,272 +0,0 @@
|
||||
# Rate Limiting
|
||||
|
||||
**Task:** "Add rate limiting to my FastAPI endpoint so users can't spam it."
|
||||
|
||||
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
## Without Ponytail — 128 lines of code
|
||||
|
||||
# Rate Limiting in FastAPI
|
||||
|
||||
Here are several approaches, from simplest to most robust:
|
||||
|
||||
## 1. **Using `slowapi` (Recommended - Easiest)**
|
||||
|
||||
```bash
|
||||
pip install slowapi
|
||||
```
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI, Request
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
from slowapi.errors import RateLimitExceeded
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
app.state.limiter = limiter
|
||||
app.add_exception_handler(RateLimitExceeded, lambda request, exc: JSONResponse(
|
||||
status_code=429,
|
||||
content={"detail": "Rate limit exceeded"}
|
||||
))
|
||||
|
||||
@app.get("/api/data")
|
||||
@limiter.limit("5/minute")
|
||||
async def get_data(request: Request):
|
||||
return {"message": "Success"}
|
||||
```
|
||||
|
||||
**Rate limit formats:**
|
||||
- `"5/minute"` - 5 requests per minute
|
||||
- `"100/hour"` - 100 requests per hour
|
||||
- `"1000/day"` - 1000 requests per day
|
||||
|
||||
---
|
||||
|
||||
## 2. **Using `limits` Library (More Control)**
|
||||
|
||||
```bash
|
||||
pip install limits
|
||||
```
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from limits import parse
|
||||
from limits.storage import MemoryStorage
|
||||
from limits.strategies import MovingWindowRateLimiter
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
storage = MemoryStorage()
|
||||
limiter = MovingWindowRateLimiter(storage)
|
||||
rate_limit = parse("5/minute")
|
||||
|
||||
@app.get("/api/data")
|
||||
async def get_data(request: Request):
|
||||
client_id = request.client.host
|
||||
|
||||
if not limiter.hit(rate_limit, client_id):
|
||||
raise HTTPException(status_code=429, detail="Rate limit exceeded")
|
||||
|
||||
return {"message": "Success"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. **Custom Middleware (Full Control)**
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI, HTTPException, Request
|
||||
from datetime import datetime, timedelta
|
||||
from collections import defaultdict
|
||||
import asyncio
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
# Store request timestamps per IP
|
||||
request_history = defaultdict(list)
|
||||
RATE_LIMIT = 5 # requests
|
||||
TIME_WINDOW = 60 # seconds
|
||||
|
||||
@app.middleware("http")
|
||||
async def rate_limit_middleware(request: Request, call_next):
|
||||
client_ip = request.client.host
|
||||
now = datetime.now()
|
||||
|
||||
# Clean old requests outside time window
|
||||
request_history[client_ip] = [
|
||||
req_time for req_time in request_history[client_ip]
|
||||
if now - req_time < timedelta(seconds=TIME_WINDOW)
|
||||
]
|
||||
|
||||
# Check if limit exceeded
|
||||
if len(request_history[client_ip]) >= RATE_LIMIT:
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={"detail": "Rate limit exceeded"}
|
||||
)
|
||||
|
||||
# Record this request
|
||||
request_history[client_ip].append(now)
|
||||
|
||||
return await call_next(request)
|
||||
|
||||
@app.get("/api/data")
|
||||
async def get_data():
|
||||
return {"message": "Success"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. **Per-User Rate Limiting (With Authentication)**
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI, Depends, HTTPException
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
|
||||
app = FastAPI()
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
|
||||
def get_user_id(token: str = Header(None)) -> str:
|
||||
# Your auth logic here
|
||||
return token or "anonymous"
|
||||
|
||||
@app.get("/api/data")
|
||||
@limiter.limit("10/minute")
|
||||
async def get_data(request: Request, user_id: str = Depends(get_user_id)):
|
||||
return {"message": f"Success for {user_id}"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. **Redis-Based Rate Limiting (Production)**
|
||||
|
||||
```bash
|
||||
pip install slowapi redis
|
||||
```
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI, Request
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
from slowapi.errors import RateLimitExceeded
|
||||
from slowapi.storage import RedisStorage
|
||||
from redis import Redis
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
redis_client = Redis.from_url("redis://localhost:6379")
|
||||
storage = RedisStorage(redis_client)
|
||||
limiter = Limiter(key_func=get_remote_address, storage=storage)
|
||||
|
||||
app = FastAPI()
|
||||
app.state.limiter = limiter
|
||||
|
||||
app.add_exception_handler(
|
||||
RateLimitExceeded,
|
||||
lambda request, exc: JSONResponse(
|
||||
status_code=429,
|
||||
content={"detail": "Rate limit exceeded"}
|
||||
)
|
||||
)
|
||||
|
||||
@app.get("/api/data")
|
||||
@limiter.limit("5/minute")
|
||||
async def get_data(request: Request):
|
||||
return {"message": "Success"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. **Complete Example with Multiple Endpoints**
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI, Request
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
from slowapi.errors import RateLimitExceeded
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
app = FastAPI()
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
app.state.limiter = limiter
|
||||
|
||||
@app.exception_handler(RateLimitExceeded)
|
||||
async def rate_limit_handler(request: Request, exc: RateLimitExceeded):
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={
|
||||
"detail": "Too many requests",
|
||||
"retry_after": exc.detail
|
||||
}
|
||||
)
|
||||
|
||||
# Strict limit
|
||||
@app.post("/api/login")
|
||||
@limiter.limit("3/minute")
|
||||
async def login(request: Request):
|
||||
return {"token": "xxx"}
|
||||
|
||||
# Moderate limit
|
||||
@app.get("/api/data")
|
||||
@limiter.limit("30/minute")
|
||||
async def get_data(request: Request):
|
||||
return {"data": []}
|
||||
|
||||
# Generous limit
|
||||
@app.get("/api/public")
|
||||
@limiter.limit("100/minute")
|
||||
async def get_public(request: Request):
|
||||
return {"public": "data"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| Method | Pros | Cons | Best For |
|
||||
|--------|------|------|----------|
|
||||
| **slowapi** | Easy, decorator-based | In-memory only | Development, small apps |
|
||||
| **limits** | Flexible, multiple backends | More setup | Custom logic |
|
||||
| **Middleware** | Full control, no dependencies | Manual cleanup | Learning, simple cases |
|
||||
| **Redis** | Distributed, scalable | Requires Redis | Production, multi-server |
|
||||
|
||||
---
|
||||
|
||||
## Testing Rate Limits
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
async def test_rate_limit():
|
||||
async with httpx.AsyncClient(app=app, base_url="http://test") as client:
|
||||
for i in range(6):
|
||||
response = await client.get("/api/data")
|
||||
print(f"Request {i+1}: {response.status_code}")
|
||||
# First 5 should be 200, 6th should be 429
|
||||
```
|
||||
|
||||
**Recommendation:** Use `slowapi` for most cases—it's the most popular and easiest to implement! 🚀
|
||||
|
||||
## With Ponytail — 10 lines of code
|
||||
|
||||
```python
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from slowapi import Limiter
|
||||
from slowapi.util import get_remote_address
|
||||
|
||||
app = FastAPI()
|
||||
limiter = Limiter(key_func=get_remote_address)
|
||||
app.state.limiter = limiter
|
||||
|
||||
@app.get("/api/endpoint")
|
||||
@limiter.limit("10/minute")
|
||||
async def my_endpoint(request):
|
||||
return {"status": "ok"}
|
||||
```
|
||||
|
||||
Skipped: custom rate limit logic, Redis, sliding windows — `slowapi` handles it. Add when: you need distributed rate limiting across multiple servers (swap `Limiter` for Redis backend) or per-user limits (add `key_func=lambda r: r.headers.get("authorization")`).
|
||||
|
||||
**128 → 10 lines of code** — same model, same prompt.
|
||||
@@ -1,390 +0,0 @@
|
||||
# Countdown Timer
|
||||
|
||||
**Task:** "Build me a countdown timer component in React that counts down from a given number of seconds."
|
||||
|
||||
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
## Without Ponytail — 267 lines of code
|
||||
|
||||
# React Countdown Timer Component
|
||||
|
||||
Here's a complete countdown timer component with multiple variations:
|
||||
|
||||
## Basic Countdown Timer
|
||||
|
||||
```jsx
|
||||
import { useState, useEffect } from 'react';
|
||||
|
||||
export function CountdownTimer({ initialSeconds = 60 }) {
|
||||
const [seconds, setSeconds] = useState(initialSeconds);
|
||||
const [isActive, setIsActive] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
let interval = null;
|
||||
|
||||
if (isActive && seconds > 0) {
|
||||
interval = setInterval(() => {
|
||||
setSeconds(seconds => seconds - 1);
|
||||
}, 1000);
|
||||
} else if (seconds === 0) {
|
||||
setIsActive(false);
|
||||
}
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}, [isActive, seconds]);
|
||||
|
||||
const toggle = () => setIsActive(!isActive);
|
||||
const reset = () => {
|
||||
setSeconds(initialSeconds);
|
||||
setIsActive(false);
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="countdown-timer">
|
||||
<div className="timer-display">{seconds}s</div>
|
||||
<div className="timer-controls">
|
||||
<button onClick={toggle}>
|
||||
{isActive ? 'Pause' : 'Start'}
|
||||
</button>
|
||||
<button onClick={reset}>Reset</button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced Timer with Formatted Display
|
||||
|
||||
```jsx
|
||||
import { useState, useEffect } from 'react';
|
||||
|
||||
export function AdvancedCountdownTimer({
|
||||
initialSeconds = 3600,
|
||||
onComplete = () => {},
|
||||
showMilliseconds = false
|
||||
}) {
|
||||
const [timeLeft, setTimeLeft] = useState(initialSeconds);
|
||||
const [isActive, setIsActive] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
let interval = null;
|
||||
|
||||
if (isActive && timeLeft > 0) {
|
||||
interval = setInterval(() => {
|
||||
setTimeLeft(prev => {
|
||||
if (prev <= 1) {
|
||||
setIsActive(false);
|
||||
onComplete();
|
||||
return 0;
|
||||
}
|
||||
return prev - 1;
|
||||
});
|
||||
}, 1000);
|
||||
}
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}, [isActive, timeLeft, onComplete]);
|
||||
|
||||
const formatTime = (totalSeconds) => {
|
||||
const hours = Math.floor(totalSeconds / 3600);
|
||||
const minutes = Math.floor((totalSeconds % 3600) / 60);
|
||||
const seconds = totalSeconds % 60;
|
||||
|
||||
const parts = [];
|
||||
if (hours > 0) parts.push(String(hours).padStart(2, '0'));
|
||||
parts.push(String(minutes).padStart(2, '0'));
|
||||
parts.push(String(seconds).padStart(2, '0'));
|
||||
|
||||
return parts.join(':');
|
||||
};
|
||||
|
||||
const toggle = () => setIsActive(!isActive);
|
||||
const reset = () => {
|
||||
setTimeLeft(initialSeconds);
|
||||
setIsActive(false);
|
||||
};
|
||||
|
||||
const progressPercent = ((initialSeconds - timeLeft) / initialSeconds) * 100;
|
||||
|
||||
return (
|
||||
<div className="advanced-timer">
|
||||
<div className="timer-display">
|
||||
{formatTime(timeLeft)}
|
||||
</div>
|
||||
<div className="progress-bar">
|
||||
<div
|
||||
className="progress-fill"
|
||||
style={{ width: `${progressPercent}%` }}
|
||||
/>
|
||||
</div>
|
||||
<div className="timer-controls">
|
||||
<button onClick={toggle} className="btn-primary">
|
||||
{isActive ? '⏸ Pause' : '▶ Start'}
|
||||
</button>
|
||||
<button onClick={reset} className="btn-secondary">
|
||||
↻ Reset
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Custom Hook Version
|
||||
|
||||
```jsx
|
||||
import { useState, useEffect, useCallback } from 'react';
|
||||
|
||||
export function useCountdown(initialSeconds) {
|
||||
const [seconds, setSeconds] = useState(initialSeconds);
|
||||
const [isActive, setIsActive] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
let interval = null;
|
||||
|
||||
if (isActive && seconds > 0) {
|
||||
interval = setInterval(() => {
|
||||
setSeconds(prev => prev - 1);
|
||||
}, 1000);
|
||||
} else if (seconds === 0 && isActive) {
|
||||
setIsActive(false);
|
||||
}
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}, [isActive, seconds]);
|
||||
|
||||
const start = useCallback(() => setIsActive(true), []);
|
||||
const pause = useCallback(() => setIsActive(false), []);
|
||||
const reset = useCallback(() => {
|
||||
setSeconds(initialSeconds);
|
||||
setIsActive(false);
|
||||
}, [initialSeconds]);
|
||||
|
||||
return {
|
||||
seconds,
|
||||
isActive,
|
||||
start,
|
||||
pause,
|
||||
reset,
|
||||
setSeconds
|
||||
};
|
||||
}
|
||||
|
||||
// Usage
|
||||
export function TimerWithHook() {
|
||||
const { seconds, isActive, start, pause, reset } = useCountdown(60);
|
||||
|
||||
return (
|
||||
<div>
|
||||
<div>{seconds}s</div>
|
||||
<button onClick={isActive ? pause : start}>
|
||||
{isActive ? 'Pause' : 'Start'}
|
||||
</button>
|
||||
<button onClick={reset}>Reset</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Styled Component with Animations
|
||||
|
||||
```jsx
|
||||
import { useState, useEffect } from 'react';
|
||||
import styled from 'styled-components';
|
||||
|
||||
const TimerContainer = styled.div`
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
gap: 20px;
|
||||
padding: 30px;
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
border-radius: 15px;
|
||||
color: white;
|
||||
font-family: 'Arial', sans-serif;
|
||||
`;
|
||||
|
||||
const Display = styled.div`
|
||||
font-size: 48px;
|
||||
font-weight: bold;
|
||||
font-variant-numeric: tabular-nums;
|
||||
animation: ${props => props.isWarning ? 'pulse 0.5s infinite' : 'none'};
|
||||
|
||||
@keyframes pulse {
|
||||
0%, 100% { opacity: 1; }
|
||||
50% { opacity: 0.5; }
|
||||
}
|
||||
`;
|
||||
|
||||
const Controls = styled.div`
|
||||
display: flex;
|
||||
gap: 10px;
|
||||
`;
|
||||
|
||||
const Button = styled.button`
|
||||
padding: 10px 20px;
|
||||
font-size: 16px;
|
||||
border: none;
|
||||
border-radius: 8px;
|
||||
cursor: pointer;
|
||||
background-color: ${props => props.variant === 'danger' ? '#ff6b6b' : '#4ecdc4'};
|
||||
color: white;
|
||||
transition: all 0.3s ease;
|
||||
|
||||
&:hover {
|
||||
transform: translateY(-2px);
|
||||
box-shadow: 0 5px 15px rgba(0, 0, 0, 0.2);
|
||||
}
|
||||
|
||||
&:active {
|
||||
transform: translateY(0);
|
||||
}
|
||||
`;
|
||||
|
||||
export function StyledCountdownTimer({ initialSeconds = 60 }) {
|
||||
const [seconds, setSeconds] = useState(initialSeconds);
|
||||
const [isActive, setIsActive] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
let interval = null;
|
||||
|
||||
if (isActive && seconds > 0) {
|
||||
interval = setInterval(() => {
|
||||
setSeconds(prev => prev - 1);
|
||||
}, 1000);
|
||||
} else if (seconds === 0) {
|
||||
setIsActive(false);
|
||||
}
|
||||
|
||||
return () => clearInterval(interval);
|
||||
}, [isActive, seconds]);
|
||||
|
||||
const isWarning = seconds <= 10 && seconds > 0;
|
||||
|
||||
return (
|
||||
<TimerContainer>
|
||||
<Display isWarning={isWarning}>{seconds}s</Display>
|
||||
<Controls>
|
||||
<Button onClick={() => setIsActive(!isActive)}>
|
||||
{isActive ? 'Pause' : 'Start'}
|
||||
</Button>
|
||||
<Button
|
||||
variant="danger"
|
||||
onClick={() => {
|
||||
setSeconds(initialSeconds);
|
||||
setIsActive(false);
|
||||
}}
|
||||
>
|
||||
Reset
|
||||
</Button>
|
||||
</Controls>
|
||||
</TimerContainer>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## CSS Styling
|
||||
|
||||
```css
|
||||
.countdown-timer {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
gap: 20px;
|
||||
padding: 20px;
|
||||
border: 2px solid #333;
|
||||
border-radius: 10px;
|
||||
background-color: #f5f5f5;
|
||||
}
|
||||
|
||||
.timer-display {
|
||||
font-size: 48px;
|
||||
font-weight: bold;
|
||||
font-variant-numeric: tabular-nums;
|
||||
color: #333;
|
||||
}
|
||||
|
||||
.timer-controls {
|
||||
display: flex;
|
||||
gap: 10px;
|
||||
}
|
||||
|
||||
.timer-controls button {
|
||||
padding: 10px 20px;
|
||||
font-size: 16px;
|
||||
border: none;
|
||||
border-radius: 5px;
|
||||
cursor: pointer;
|
||||
background-color: #667eea;
|
||||
color: white;
|
||||
transition: background-color 0.3s;
|
||||
}
|
||||
|
||||
.timer-controls button:hover {
|
||||
background-color: #5568d3;
|
||||
}
|
||||
|
||||
.progress-bar {
|
||||
width: 100%;
|
||||
height: 8px;
|
||||
background-color: #e0e0e0;
|
||||
border-radius: 4px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.progress-fill {
|
||||
height: 100%;
|
||||
background-color: #667eea;
|
||||
transition: width 0.3s linear;
|
||||
}
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
```jsx
|
||||
// Basic usage
|
||||
<CountdownTimer initialSeconds={30} />
|
||||
|
||||
// Advanced with callback
|
||||
<AdvancedCountdownTimer
|
||||
initialSeconds={300}
|
||||
onComplete={() => alert('Time\'s up!')}
|
||||
/>
|
||||
|
||||
// Using the hook
|
||||
<TimerWithHook />
|
||||
|
||||
// Styled version
|
||||
<StyledCountdownTimer initialSeconds={60} />
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
✅ **Basic Timer** - Simple countdown with start/pause/reset
|
||||
✅ **Advanced Timer** - Formatted display (HH:MM:SS), progress bar, callbacks
|
||||
✅ **Custom Hook** - Reusable logic for any component
|
||||
✅ **Styled Components** - Beautiful UI with animations
|
||||
✅ **Warning State** - Visual feedback when time is running out
|
||||
✅ **Completion Callback** - Execute code when timer finishes
|
||||
|
||||
Choose the version that best fits your needs!
|
||||
|
||||
## With Ponytail — 9 lines of code
|
||||
|
||||
```jsx
|
||||
export function CountdownTimer({ seconds }) {
|
||||
const [remaining, setRemaining] = React.useState(seconds);
|
||||
|
||||
React.useEffect(() => {
|
||||
if (remaining <= 0) return;
|
||||
const timer = setInterval(() => setRemaining(r => r - 1), 1000);
|
||||
return () => clearInterval(timer);
|
||||
}, [remaining]);
|
||||
|
||||
return <div>{remaining}s</div>;
|
||||
}
|
||||
```
|
||||
|
||||
Skipped: pause/resume, formatted display (mm:ss), sound on zero, styling — add when needed.
|
||||
|
||||
**267 → 9 lines of code** — same model, same prompt.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Sorting
|
||||
|
||||
**Task:** "Sort this array of numbers."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```js
|
||||
function quickSort(arr, compareFn = (a, b) => a - b) {
|
||||
if (arr.length <= 1) return arr;
|
||||
|
||||
const pivot = arr[Math.floor(arr.length / 2)];
|
||||
const left = [];
|
||||
const right = [];
|
||||
const equal = [];
|
||||
|
||||
for (const item of arr) {
|
||||
const cmp = compareFn(item, pivot);
|
||||
if (cmp < 0) left.push(item);
|
||||
else if (cmp > 0) right.push(item);
|
||||
else equal.push(item);
|
||||
}
|
||||
|
||||
return [
|
||||
...quickSort(left, compareFn),
|
||||
...equal,
|
||||
...quickSort(right, compareFn),
|
||||
];
|
||||
}
|
||||
|
||||
const sorted = quickSort(numbers);
|
||||
```
|
||||
|
||||
A hand-rolled quicksort. It allocates three arrays per recursion level, blows the stack on large inputs, and replaces an engine-optimized native sort with homework.
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```js
|
||||
// ponytail: this exists
|
||||
numbers.sort((a, b) => a - b)
|
||||
```
|
||||
|
||||
**24 lines → 1 line.** Every runtime ships a sort tuned by people whose whole job is sorting. Use it.
|
||||
@@ -1,41 +0,0 @@
|
||||
# URL Parameters
|
||||
|
||||
**Task:** "Parse and build URL query strings."
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```bash
|
||||
npm install query-string
|
||||
# 4.5 kB gzipped, 3.5M downloads/week
|
||||
```
|
||||
|
||||
```js
|
||||
import qs from "query-string";
|
||||
|
||||
// Parse
|
||||
const params = qs.parse(location.search);
|
||||
// → { page: "2", sort: "name", tags: ["js", "css"] }
|
||||
|
||||
// Build
|
||||
const url = qs.stringify({ page: 2, sort: "name", tags: ["js", "css"] });
|
||||
// → "page=2&sort=name&tags=js&tags=css"
|
||||
```
|
||||
|
||||
## With Ponytail
|
||||
|
||||
```js
|
||||
// ponytail: URLSearchParams does this
|
||||
const params = new URLSearchParams(location.search);
|
||||
|
||||
// Read
|
||||
params.get("page"); // "2"
|
||||
params.getAll("tags"); // ["js", "css"]
|
||||
|
||||
// Build
|
||||
const out = new URLSearchParams({ page: 2, sort: "name" });
|
||||
out.append("tags", "js");
|
||||
out.append("tags", "css");
|
||||
out.toString(); // "page=2&sort=name&tags=js&tags=css"
|
||||
```
|
||||
|
||||
**1 dependency → 0 dependencies.** `URLSearchParams` is in every browser and in Node.js since v10. It handles encoding, repeated keys, and iteration. The package was a polyfill for an API that has shipped everywhere for years.
|
||||
@@ -1,64 +0,0 @@
|
||||
# Web Platform Lookup
|
||||
|
||||
**Task:** "Add a modal dialog that closes when you click the backdrop."
|
||||
|
||||
Rung 3 of the ladder is "native platform feature covers it?" On web work the
|
||||
trap is that the agent forgets what the platform already does and reaches for a
|
||||
library. When ponytail has [Modern Web Guidance](https://github.com/GoogleChrome/modern-web-guidance)
|
||||
on hand, rung 3 gets a lookup: `modern-web search "modal dialog light dismiss"`.
|
||||
|
||||
## Without Ponytail
|
||||
|
||||
```bash
|
||||
npm install @radix-ui/react-dialog
|
||||
```
|
||||
|
||||
```jsx
|
||||
import * as Dialog from "@radix-ui/react-dialog";
|
||||
|
||||
export default function Modal({ open, onOpenChange, children }) {
|
||||
return (
|
||||
<Dialog.Root open={open} onOpenChange={onOpenChange}>
|
||||
<Dialog.Portal>
|
||||
<Dialog.Overlay className="overlay" />
|
||||
<Dialog.Content className="content">
|
||||
{children}
|
||||
<Dialog.Close className="close">×</Dialog.Close>
|
||||
</Dialog.Content>
|
||||
</Dialog.Portal>
|
||||
</Dialog.Root>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
A dependency, a portal, an overlay node, and controlled open state, to put a
|
||||
box on top with a backdrop.
|
||||
|
||||
## With Ponytail
|
||||
|
||||
`modern-web search "modal dialog light dismiss"` →
|
||||
`modern-web retrieve light-dismiss-a-dialog`. The platform has it:
|
||||
|
||||
```html
|
||||
<!-- ponytail: <dialog> + closedby, browser does the backdrop, focus trap, and Esc -->
|
||||
<dialog closedby="any">
|
||||
<p>...</p>
|
||||
</dialog>
|
||||
```
|
||||
|
||||
```js
|
||||
document.querySelector("dialog").showModal();
|
||||
```
|
||||
|
||||
**1 dependency + portal/overlay machinery → 0 dependencies + a `<dialog>`.**
|
||||
The `::backdrop` is free, focus is trapped and restored for you, `Esc` closes
|
||||
it, and `closedby="any"` adds click-outside dismissal. The browser team did the
|
||||
work.
|
||||
|
||||
## The point
|
||||
|
||||
MWG suggests the cutting edge, ponytail keeps only the rung that holds. The
|
||||
lookup found `light-dismiss-a-dialog`; the ladder took it because it deletes a
|
||||
dependency. The same search would have offered scroll-driven animations and
|
||||
view transitions for other tasks, and the ladder would have skipped them when
|
||||
the task didn't need them. Lookup, not license.
|
||||
@@ -2,7 +2,7 @@
|
||||
// ponytail — Claude Code SessionStart activation hook
|
||||
//
|
||||
// Runs on every session start:
|
||||
// 1. Writes flag file at $CLAUDE_CONFIG_DIR/.ponytail-active (defaults to ~/.claude; statusline reads this)
|
||||
// 1. Writes flag file at ~/.claude/.ponytail-active (statusline reads this)
|
||||
// 2. Emits ponytail ruleset as hidden SessionStart context
|
||||
// 3. Detects missing statusline config and emits setup nudge
|
||||
|
||||
@@ -43,9 +43,7 @@ let output = getPonytailInstructions(mode);
|
||||
if (!isCodex) try {
|
||||
let hasStatusline = false;
|
||||
if (fs.existsSync(settingsPath)) {
|
||||
// Strip UTF-8 BOM some editors prepend on Windows (breaks JSON.parse)
|
||||
const raw = fs.readFileSync(settingsPath, 'utf8').replace(/^\uFEFF/, '');
|
||||
const settings = JSON.parse(raw);
|
||||
const settings = JSON.parse(fs.readFileSync(settingsPath, 'utf8'));
|
||||
if (settings.statusLine) {
|
||||
hasStatusline = true;
|
||||
}
|
||||
@@ -71,8 +69,4 @@ if (!isCodex) try {
|
||||
// Silent fail — don't block session start over statusline detection
|
||||
}
|
||||
|
||||
try {
|
||||
writeHookOutput('SessionStart', mode, output);
|
||||
} catch (e) {
|
||||
// Silent fail — stdout closed/EPIPE at hook exit must not surface as a hook failure
|
||||
}
|
||||
|
||||
@@ -33,15 +33,6 @@ function normalizePersistedMode(mode) {
|
||||
return normalizeMode(mode) || normalizeConfigMode(mode);
|
||||
}
|
||||
|
||||
// "stop ponytail" / "normal mode" turn ponytail off, but only as a standalone
|
||||
// command. Matching the phrase anywhere in the message turned it off mid-task
|
||||
// for ordinary requests like "add a normal mode toggle" — so require the whole
|
||||
// message to be the command, ignoring case and trailing punctuation.
|
||||
function isDeactivationCommand(text) {
|
||||
const t = String(text || '').trim().toLowerCase().replace(/[.!?\s]+$/, '');
|
||||
return t === 'stop ponytail' || t === 'normal mode';
|
||||
}
|
||||
|
||||
function getConfigDir() {
|
||||
if (process.env.XDG_CONFIG_HOME) {
|
||||
return path.join(process.env.XDG_CONFIG_HOME, 'ponytail');
|
||||
@@ -107,6 +98,5 @@ module.exports = {
|
||||
normalizeMode,
|
||||
normalizeConfigMode,
|
||||
normalizePersistedMode,
|
||||
isDeactivationCommand,
|
||||
writeDefaultMode,
|
||||
};
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
// ponytail — UserPromptSubmit hook to track which ponytail mode is active
|
||||
// Inspects user input for /ponytail commands and writes mode to flag file
|
||||
|
||||
const { getDefaultMode, isDeactivationCommand } = require('./ponytail-config');
|
||||
const { getDefaultMode } = require('./ponytail-config');
|
||||
const { clearMode, setMode, writeHookOutput } = require('./ponytail-runtime');
|
||||
|
||||
let input = '';
|
||||
@@ -45,7 +45,7 @@ process.stdin.on('end', () => {
|
||||
}
|
||||
|
||||
// Detect deactivation
|
||||
if (isDeactivationCommand(prompt)) {
|
||||
if (/\b(stop ponytail|normal mode)\b/i.test(prompt)) {
|
||||
clearMode();
|
||||
writeHookOutput('UserPromptSubmit', 'off', 'PONYTAIL MODE OFF');
|
||||
}
|
||||
|
||||
@@ -1,6 +1,4 @@
|
||||
# CLAUDE_CONFIG_DIR overrides ~/.claude, matching where the hooks write the flag (issue #34)
|
||||
$ClaudeDir = if ($env:CLAUDE_CONFIG_DIR) { $env:CLAUDE_CONFIG_DIR } else { Join-Path $HOME ".claude" }
|
||||
$Flag = Join-Path $ClaudeDir ".ponytail-active"
|
||||
$Flag = Join-Path $HOME ".claude/.ponytail-active"
|
||||
if (-not (Test-Path $Flag)) {
|
||||
exit 0
|
||||
}
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# CLAUDE_CONFIG_DIR overrides ~/.claude, matching where the hooks write the flag (issue #34)
|
||||
flag="${CLAUDE_CONFIG_DIR:-$HOME/.claude}/.ponytail-active"
|
||||
flag="$HOME/.claude/.ponytail-active"
|
||||
[ -f "$flag" ] || exit 0
|
||||
|
||||
mode=$(head -n1 "$flag" | tr -d '[:space:]')
|
||||
|
||||
@@ -7,7 +7,6 @@ const {
|
||||
normalizeMode,
|
||||
normalizeConfigMode,
|
||||
normalizePersistedMode,
|
||||
isDeactivationCommand,
|
||||
writeDefaultMode,
|
||||
} = require("../hooks/ponytail-config.js");
|
||||
const { getPonytailInstructions, filterSkillBodyForMode } = require("../hooks/ponytail-instructions.js");
|
||||
@@ -120,11 +119,6 @@ export default function ponytailExtension(pi) {
|
||||
handler: (_args, ctx) => sendAlias("/skill:ponytail-audit", "", ctx),
|
||||
});
|
||||
|
||||
pi.registerCommand("ponytail-gain", {
|
||||
description: "Run /skill:ponytail-gain",
|
||||
handler: (_args, ctx) => sendAlias("/skill:ponytail-gain", "", ctx),
|
||||
});
|
||||
|
||||
pi.registerCommand("ponytail-debt", {
|
||||
description: "Run /skill:ponytail-debt",
|
||||
handler: (_args, ctx) => sendAlias("/skill:ponytail-debt", "", ctx),
|
||||
@@ -139,7 +133,7 @@ export default function ponytailExtension(pi) {
|
||||
if (event?.source === "extension") return;
|
||||
|
||||
const text = String(event?.text || "");
|
||||
if (currentMode !== "off" && isDeactivationCommand(text)) {
|
||||
if (currentMode !== "off" && /\b(stop ponytail|normal mode)\b/i.test(text)) {
|
||||
setMode("off");
|
||||
}
|
||||
});
|
||||
|
||||
@@ -57,7 +57,7 @@ function withTempConfig(fn) {
|
||||
test("extension registers Ponytail commands", () => {
|
||||
const { commands } = createPiHarness();
|
||||
|
||||
assert.deepEqual([...commands.keys()].sort(), ["ponytail", "ponytail-audit", "ponytail-debt", "ponytail-gain", "ponytail-help", "ponytail-review"]);
|
||||
assert.deepEqual([...commands.keys()].sort(), ["ponytail", "ponytail-audit", "ponytail-debt", "ponytail-help", "ponytail-review"]);
|
||||
});
|
||||
|
||||
test("/ponytail updates session mode and injects instructions", async () => withTempConfig(async () => {
|
||||
@@ -100,14 +100,12 @@ test("skill alias commands delegate to Pi skill commands", async () => {
|
||||
await commands.get("ponytail-review").handler("", ctx);
|
||||
await commands.get("ponytail-audit").handler("", ctx);
|
||||
await commands.get("ponytail-debt").handler("", ctx);
|
||||
await commands.get("ponytail-gain").handler("", ctx);
|
||||
await commands.get("ponytail-help").handler("", ctx);
|
||||
|
||||
assert.deepEqual(sentUserMessages.map((entry) => entry.text), [
|
||||
"/skill:ponytail-review",
|
||||
"/skill:ponytail-audit",
|
||||
"/skill:ponytail-debt",
|
||||
"/skill:ponytail-gain",
|
||||
"/skill:ponytail-help",
|
||||
]);
|
||||
});
|
||||
@@ -123,15 +121,3 @@ test("normal mode disables persistent instructions", async () => withTempConfig(
|
||||
const disabled = await events.get("before_agent_start")({ systemPrompt: "BASE" }, ctx);
|
||||
assert.equal(disabled, undefined);
|
||||
}));
|
||||
|
||||
test("a request mentioning normal mode stays active", async () => withTempConfig(async () => {
|
||||
const { commands, events } = createPiHarness();
|
||||
const ctx = createCommandContext();
|
||||
|
||||
await events.get("session_start")({ reason: "startup" }, ctx);
|
||||
await commands.get("ponytail").handler("ultra", ctx);
|
||||
await events.get("input")({ text: "add a normal mode toggle next to dark mode", source: "interactive" }, ctx);
|
||||
|
||||
const result = await events.get("before_agent_start")({ systemPrompt: "BASE" }, ctx);
|
||||
assert.match(result.systemPrompt, /PONYTAIL MODE ACTIVE/);
|
||||
}));
|
||||
|
||||
@@ -71,7 +71,7 @@ test("filterSkillBodyForMode keeps rule bullets that contain a colon", () => {
|
||||
// Regression: rule bullets outside the Intensity section (e.g. the
|
||||
// "No unrequested abstractions:" rule or the `ponytail:` comment convention)
|
||||
// contain a colon and must not be mistaken for mode-example lines.
|
||||
const skillPath = new URL("../../skills/ponytail/SKILL.md", import.meta.url);
|
||||
const skillPath = join(import.meta.dirname, "..", "..", "skills", "ponytail", "SKILL.md");
|
||||
const body = readFileSync(skillPath, "utf8");
|
||||
|
||||
const filtered = filterSkillBodyForMode(body, "full");
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
# ponytail-mcp
|
||||
|
||||
An MCP server that serves Ponytail's lazy-senior-dev instructions. It exposes
|
||||
the same ruleset the Claude hooks and Pi extension use, so every host emits
|
||||
identical rules.
|
||||
|
||||
It is not a replacement for the always-on adapters. Ponytail normally lives in
|
||||
the system context every turn. MCP prompts are user-invoked, and there is no
|
||||
portable MCP primitive for "inject this into every turn" across hosts. So this
|
||||
server is the clean option for MCP hosts whose only injection point is the
|
||||
prompt menu, or that pull context through tools. See issue #70.
|
||||
|
||||
## What it exposes
|
||||
|
||||
- Prompt `ponytail` — returns the ruleset as a user message. Optional `mode`
|
||||
argument: `lite`, `full`, or `ultra`. Omit it to use the configured default.
|
||||
- Tool `ponytail_instructions` — same text, plus `structuredContent`
|
||||
(`{ mode, instructions }`), for hosts that pull context via tools or code
|
||||
execution. Read-only.
|
||||
|
||||
Mode resolution reuses `hooks/ponytail-config.js`, so `PONYTAIL_DEFAULT_MODE`
|
||||
and `~/.config/ponytail/config.json` work the same as everywhere else.
|
||||
|
||||
## Run it
|
||||
|
||||
```bash
|
||||
cd ponytail-mcp
|
||||
npm install
|
||||
node index.js # speaks MCP over stdio
|
||||
```
|
||||
|
||||
Point an MCP host at that command. Example client entry:
|
||||
|
||||
```json
|
||||
{ "mcpServers": { "ponytail": { "command": "node", "args": ["ponytail-mcp/index.js"] } } }
|
||||
```
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
|
||||
Covers mode resolution and the instruction text. The MCP wiring in `index.js`
|
||||
is intentionally thin: it just maps the prompt and tool onto
|
||||
`buildInstructions`.
|
||||
@@ -1,48 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
// Ponytail MCP server: serves the lazy-senior-dev ruleset over stdio as a
|
||||
// prompt (user-invoked) and a tool (for hosts that pull context via tools).
|
||||
// It does NOT replace the always-on adapters; it's the clean option for hosts
|
||||
// whose only injection point is the prompt menu (see #70).
|
||||
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
||||
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
||||
import { z } from "zod";
|
||||
|
||||
import { MODES, buildInstructions, resolveMode } from "./instructions.js";
|
||||
|
||||
const server = new McpServer({ name: "ponytail", version: "0.1.0" });
|
||||
|
||||
const modeArg = z
|
||||
.enum(MODES)
|
||||
.optional()
|
||||
.describe("Ponytail intensity: lite, full, or ultra. Omit for the configured default.");
|
||||
|
||||
server.registerPrompt(
|
||||
"ponytail",
|
||||
{
|
||||
title: "Ponytail mode",
|
||||
description: "Lazy senior dev instructions: YAGNI, stdlib first, the smallest correct change.",
|
||||
argsSchema: { mode: modeArg },
|
||||
},
|
||||
({ mode }) => ({
|
||||
messages: [{ role: "user", content: { type: "text", text: buildInstructions(mode) } }],
|
||||
}),
|
||||
);
|
||||
|
||||
server.registerTool(
|
||||
"ponytail_instructions",
|
||||
{
|
||||
title: "Ponytail instructions",
|
||||
description: "Return the Ponytail ruleset for the given intensity (lite, full, or ultra).",
|
||||
inputSchema: { mode: modeArg },
|
||||
outputSchema: { mode: z.string(), instructions: z.string() },
|
||||
annotations: { readOnlyHint: true, openWorldHint: false },
|
||||
},
|
||||
({ mode }) => {
|
||||
const resolvedMode = resolveMode(mode);
|
||||
const instructions = buildInstructions(resolvedMode);
|
||||
const structuredContent = { mode: resolvedMode, instructions };
|
||||
return { content: [{ type: "text", text: instructions }], structuredContent };
|
||||
},
|
||||
);
|
||||
|
||||
await server.connect(new StdioServerTransport());
|
||||
@@ -1,26 +0,0 @@
|
||||
// Pure instruction selection for the Ponytail MCP server. No MCP/SDK imports,
|
||||
// so this stays unit-testable on its own. Reuses the same builder the Claude
|
||||
// hooks and Pi extension use, so every host emits identical rules.
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const { getPonytailInstructions } = require("../hooks/ponytail-instructions.js");
|
||||
const { getDefaultMode, normalizeMode } = require("../hooks/ponytail-config.js");
|
||||
|
||||
// The three intensities the server offers. "off" has no instructions to serve.
|
||||
export const MODES = ["lite", "full", "ultra"];
|
||||
|
||||
// Resolve a requested mode to a runtime intensity. Unknown, empty, or "off"
|
||||
// falls back to the configured default, then to "full".
|
||||
// ponytail: keep the surface to these three; "off"/"review" aren't served here.
|
||||
export function resolveMode(requested) {
|
||||
const asked = normalizeMode(requested);
|
||||
if (asked && asked !== "off") return asked;
|
||||
|
||||
const fallback = normalizeMode(getDefaultMode());
|
||||
return fallback && fallback !== "off" ? fallback : "full";
|
||||
}
|
||||
|
||||
export function buildInstructions(requested) {
|
||||
return getPonytailInstructions(resolveMode(requested));
|
||||
}
|
||||
@@ -1,13 +0,0 @@
|
||||
{
|
||||
"name": "ponytail-mcp",
|
||||
"version": "0.1.0",
|
||||
"description": "MCP server that serves Ponytail's lazy-senior-dev instructions as a prompt and a tool.",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"license": "MIT",
|
||||
"scripts": { "test": "node --test ./test/*.test.js" },
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/sdk": "^1.19.0",
|
||||
"zod": "^3.23.0"
|
||||
}
|
||||
}
|
||||
@@ -1,22 +0,0 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
|
||||
import { MODES, resolveMode, buildInstructions } from "../instructions.js";
|
||||
|
||||
test("resolveMode keeps valid intensities", () => {
|
||||
for (const mode of MODES) assert.equal(resolveMode(mode), mode);
|
||||
});
|
||||
|
||||
test("resolveMode falls back to a runtime intensity for off/unknown/empty", () => {
|
||||
// PONYTAIL_DEFAULT_MODE could be anything in CI, so just assert the contract:
|
||||
// never returns "off", "review", or junk — always one of the served modes.
|
||||
for (const input of ["off", "review", "nonsense", "", undefined, null]) {
|
||||
assert.ok(MODES.includes(resolveMode(input)), `resolveMode(${input}) must be a served mode`);
|
||||
}
|
||||
});
|
||||
|
||||
test("buildInstructions returns the ruleset tagged with the resolved mode", () => {
|
||||
const text = buildInstructions("ultra");
|
||||
assert.match(text, /PONYTAIL MODE ACTIVE/);
|
||||
assert.match(text, /ultra/);
|
||||
});
|
||||
@@ -21,7 +21,6 @@ const DESCRIPTIONS = {
|
||||
'ponytail-review': 'Review a diff for over-engineering. Finds what to delete: reinvented stdlib, needless deps, speculative abstractions. One line per finding.',
|
||||
'ponytail-audit': 'Audit the whole repo for over-engineering. A ranked list of what to delete, simplify, or replace with stdlib or native features.',
|
||||
'ponytail-debt': 'Harvest every ponytail: shortcut comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.',
|
||||
'ponytail-gain': 'Show ponytail measured impact as a scoreboard: less code, less cost, more speed, from the benchmark medians. One-shot display.',
|
||||
'ponytail-help': "Quick reference for ponytail's modes, skills, and commands. One-shot display.",
|
||||
};
|
||||
|
||||
|
||||
@@ -20,7 +20,6 @@ const copies = [
|
||||
['.cursor/rules/ponytail.mdc', stripFrontmatter],
|
||||
['.windsurf/rules/ponytail.md', text => text.trim()],
|
||||
['.clinerules/ponytail.md', text => text.trim()],
|
||||
['.agents/rules/ponytail.md', text => text.trim()],
|
||||
['.github/copilot-instructions.md', text => text.trim()],
|
||||
['.kiro/steering/ponytail.md', stripFrontmatter],
|
||||
];
|
||||
@@ -44,14 +43,7 @@ const INVARIANTS = [
|
||||
'naive heuristic', // ceiling-comment rule
|
||||
'ONE runnable check', // test reflex
|
||||
'flimsier algorithm', // robust-variant rule
|
||||
// the four "not lazy about" safety carve-outs: pin each so a reword in either
|
||||
// file can't silently drop one. Only validation was pinned before. These are the
|
||||
// continuous substrings present in both files ("prevents data loss" because the
|
||||
// full "error handling that prevents data loss" wraps a line in SKILL.md).
|
||||
'input validation at trust boundaries',
|
||||
'prevents data loss',
|
||||
'security',
|
||||
'accessibility',
|
||||
'input validation at trust boundaries', // the "not lazy about" clause
|
||||
'Lazy code without its check is unfinished', // one-check promoted to headline
|
||||
];
|
||||
|
||||
|
||||
@@ -35,7 +35,6 @@ End with `net: -<N> lines, -<M> deps possible.` Nothing to cut: `Lean already. S
|
||||
|
||||
## Boundaries
|
||||
|
||||
Scope: over-engineering and complexity only. Correctness bugs, security holes,
|
||||
and performance are explicitly out of scope — route them to a normal review
|
||||
pass. Lists findings, applies nothing. One-shot.
|
||||
Complexity only, correctness bugs, security holes, and performance go to a
|
||||
normal review pass. Lists findings, applies nothing. One-shot.
|
||||
"stop ponytail-audit" or "normal mode" to revert.
|
||||
|
||||
@@ -1,50 +0,0 @@
|
||||
---
|
||||
name: ponytail-gain
|
||||
description: >
|
||||
Show ponytail's measured impact as a compact scoreboard: less code, less
|
||||
cost, more speed, from the benchmark medians. One-shot display, not a
|
||||
persistent mode, and not a per-repo number. Trigger: /ponytail-gain,
|
||||
"ponytail gain", "what does ponytail save", "show ponytail impact",
|
||||
"ponytail scoreboard".
|
||||
---
|
||||
|
||||
# Ponytail Gain
|
||||
|
||||
Display this scoreboard when invoked. One-shot: do NOT change mode, write flag
|
||||
files, or persist anything.
|
||||
|
||||
The figures are the published benchmark medians (5 everyday tasks: email
|
||||
validator, debounce, CSV sum, countdown timer, rate limiter; three models:
|
||||
Haiku, Sonnet, Opus). They are measured, not computed from the current repo.
|
||||
Source: `benchmarks/` and the README.
|
||||
|
||||
## Scoreboard
|
||||
|
||||
Render plain ASCII bars. The bar length shows the measured range; the label
|
||||
carries the exact figure:
|
||||
|
||||
```
|
||||
ponytail gain benchmark median · 5 tasks · 3 models
|
||||
|
||||
Lines of code no-skill ████████████████████ 100%
|
||||
ponytail ██▌················· 6–20% ▼ 80–94%
|
||||
Cost no-skill ████████████████████ 100%
|
||||
ponytail █████▌·············· 23–53% ▼ 47–77%
|
||||
Speed ponytail ▸ 3–6× faster
|
||||
|
||||
This repo: /ponytail-debt (shortcuts you deferred)
|
||||
/ponytail-audit (what's still cuttable)
|
||||
```
|
||||
|
||||
## Honesty boundary
|
||||
|
||||
These are benchmark medians, not this repo. NEVER print a per-repo savings
|
||||
number ("you saved X lines/tokens here"): the unbuilt version was never
|
||||
written, so there is no real baseline to subtract from in a live repo. The
|
||||
only real per-repo figures come from `/ponytail-debt` (a counted ledger), and
|
||||
this card points there instead of inventing one.
|
||||
|
||||
## Boundaries
|
||||
|
||||
One-shot display. Edits nothing, changes no mode.
|
||||
"stop ponytail" or "normal mode": revert.
|
||||
@@ -27,7 +27,6 @@ Level sticks until changed or session end.
|
||||
|-------|---------|--------------|
|
||||
| **ponytail** | `/ponytail` | Lazy mode itself. Simplest solution that works. |
|
||||
| **ponytail-review** | `/ponytail-review` | Over-engineering review: `L42: yagni: factory, one product. Inline.` |
|
||||
| **ponytail-gain** | `/ponytail-gain` | Measured-impact scoreboard: less code, less cost, more speed. |
|
||||
| **ponytail-help** | `/ponytail-help` | This card. |
|
||||
|
||||
Codex uses `@ponytail`, `@ponytail-review`, and `@ponytail-help`; Claude Code
|
||||
|
||||
@@ -49,9 +49,8 @@ If there is nothing to cut, say `Lean already. Ship.` and stop.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Scope: over-engineering and complexity only. Correctness bugs, security holes,
|
||||
and performance are explicitly out of scope — route them to a normal review
|
||||
pass, not this one. A single smoke test or `assert`-based
|
||||
Complexity only, correctness bugs, security holes, and performance go to a
|
||||
normal review pass, not this one. A single smoke test or `assert`-based
|
||||
self-check is the ponytail minimum, not bloat, never flag it for deletion.
|
||||
Does not apply the fixes, only lists them.
|
||||
"stop ponytail-review" or "normal mode": revert to verbose review style.
|
||||
|
||||
@@ -10,7 +10,6 @@ description: >
|
||||
"minimal solution", "yagni", "do less", or "shortest path", and whenever
|
||||
they complain about over-engineering, bloat, boilerplate, or unnecessary
|
||||
dependencies.
|
||||
argument-hint: "[lite|full|ultra]"
|
||||
license: MIT
|
||||
---
|
||||
|
||||
@@ -40,16 +39,6 @@ Stop at the first rung that holds:
|
||||
The ladder is a reflex, not a research project. Two rungs work → take the
|
||||
higher one and move on. The first lazy solution that works is the right one.
|
||||
|
||||
## Web tasks: rung 3 lookup
|
||||
|
||||
On web work, rung 3 is where the laziest win hides: a native element or CSS
|
||||
behavior the agent forgot exists. If a web task turns on whether the platform
|
||||
covers it (a date input, dialog, popover, view transition, container query),
|
||||
and the `modern-web` CLI is available, look it up: `modern-web search "<task>"`,
|
||||
then `modern-web retrieve <id>`. It is a lookup, not a license, the answer
|
||||
still goes through the ladder. MWG suggests the cutting edge; you keep only the
|
||||
rung that holds. Not installed? Skip it, the ladder runs fine without it.
|
||||
|
||||
## Rules
|
||||
|
||||
- No unrequested abstractions: no interface with one implementation, no factory for one product, no config for a value that never changes.
|
||||
|
||||
@@ -25,10 +25,6 @@ const VERSIONED_MANIFESTS = [
|
||||
// Gemini auto-discovers these by directory; the manifest is only useful if they exist.
|
||||
const REUSED_COMMANDS = ['commands/ponytail.toml', 'commands/ponytail-review.toml'];
|
||||
const REUSED_SKILLS = ['skills/ponytail/SKILL.md'];
|
||||
// Gemini CLI auto-loads this exact path for extension hooks. Ponytail's
|
||||
// Claude/Codex hook map uses events Gemini does not support, so it must stay
|
||||
// behind the host-specific plugin manifests instead.
|
||||
const GEMINI_AUTO_HOOKS = 'hooks/hooks.json';
|
||||
// Same load-bearing phrases asserted by scripts/check-rule-copies.js: the file
|
||||
// contextFileName points at must actually carry the rules, not just exist.
|
||||
const RULE_INVARIANTS = [
|
||||
@@ -81,11 +77,3 @@ test('the commands and skills the adapter reuses are present', () => {
|
||||
assert.ok(fs.existsSync(path.join(root, rel)), `reused file missing: ${rel}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('Gemini cannot auto-discover Claude/Codex hook events', () => {
|
||||
assert.equal(
|
||||
fs.existsSync(path.join(root, GEMINI_AUTO_HOOKS)),
|
||||
false,
|
||||
`${GEMINI_AUTO_HOOKS} is auto-loaded by Gemini CLI; keep Claude/Codex hooks on manifest paths`,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -11,11 +11,7 @@ const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const root = path.join(__dirname, '..');
|
||||
const HOOKS_JSON = 'hooks/claude-codex-hooks.json';
|
||||
const HOST_PLUGIN_MANIFESTS = [
|
||||
'.claude-plugin/plugin.json',
|
||||
'.codex-plugin/plugin.json',
|
||||
];
|
||||
const HOOKS_JSON = 'hooks/hooks.json';
|
||||
// cmd.exe variable syntax (%FOO%); PowerShell leaves it literal, breaking the path.
|
||||
const CMD_VAR_SYNTAX = /%[A-Za-z_][A-Za-z0-9_]*%/;
|
||||
// Pull the hooks/<script> a command launches, so we can check it exists.
|
||||
@@ -50,10 +46,3 @@ test('every hook command points at a script that ships in hooks/', () => {
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('Claude and Codex manifests point at the shared host-specific hook config', () => {
|
||||
for (const rel of HOST_PLUGIN_MANIFESTS) {
|
||||
const manifest = JSON.parse(fs.readFileSync(path.join(root, rel), 'utf8'));
|
||||
assert.equal(manifest.hooks, `./${HOOKS_JSON}`, `${rel} must not rely on root hooks auto-discovery`);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -64,23 +64,6 @@ assert.equal(fs.existsSync(codexState), false);
|
||||
output = JSON.parse(result.stdout);
|
||||
assert.equal(output.systemMessage, 'PONYTAIL:OFF');
|
||||
|
||||
// A request that merely mentions "normal mode" must not deactivate ponytail.
|
||||
result = run('ponytail-mode-tracker.js', codexEnv, JSON.stringify({ prompt: '@ponytail lite' }));
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
assert.equal(fs.readFileSync(codexState, 'utf8'), 'lite');
|
||||
|
||||
result = run(
|
||||
'ponytail-mode-tracker.js',
|
||||
codexEnv,
|
||||
JSON.stringify({ prompt: 'add a normal mode toggle next to dark mode' }),
|
||||
);
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
assert.equal(
|
||||
fs.readFileSync(codexState, 'utf8'),
|
||||
'lite',
|
||||
'incidental "normal mode" in a request must not turn ponytail off',
|
||||
);
|
||||
|
||||
const claudeEnv = {
|
||||
HOME: home,
|
||||
USERPROFILE: home,
|
||||
|
||||
Reference in New Issue
Block a user