Fix/examples issue 127 (#131)

* docs: correct cost claim to 42-75% from 30-rep re-verification Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus): ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%. The direction holds, both ends came in a few points lower. Updates the README headline and body, the benchmark chart subtitle, and the benchmarks/README cost table, and adds a dated results doc with full method. Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On OpenAI reasoning models ponytail costs more, not less, so the claim stays Claude-scoped. Gemini run pending a fresh-quota day. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scope the body claim to Claude models "on every model" read as cross-provider, but the 30-rep verification shows the cost win reverses on OpenAI reasoning models. Match the caption and benchmarks/README, which already say Claude. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reframe the pitch as the discipline, not token savings The cost/code/latency numbers vary by model and on some (terse reasoning models like GPT-5.5) ponytail costs more, so leading with them as a universal win was misleading. Adds model-variance to the headline caption and a paragraph making the stated point the mental model: write only what the task needs, safety kept, maintainable code. Savings are a model-dependent side effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: name the ladder's reasoning cost The ladder is a deliberation step: on reasoning models the agent spends thinking tokens working through the rungs before it saves any output, which together with the always-on ruleset can outweigh the shorter code. Makes the GPT-5.5 cost increase legible rather than just stating it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: state the single-shot limitation honestly The benchmark is single-shot (one prompt, one completion); it does not measure a real multi-turn agent session, where the ruleset re-injects and the ladder deliberates every turn. Adds that caveat to the README, and corrects the benchmarks/README note that claimed caching widens the gap "in ponytail's favor" (unverified, and a measured agentic A/B in #121 found the opposite can happen). Per-session cost can land either way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix run count in caption (cost is 30 runs, not 10) Cost was re-verified at 30 reps; code and latency are still the original 10. The headline caption said "10 runs" across the board, which undersold the cost verification. Now states the split, matching benchmarks/README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(examples): replace hand-written examples with real benchmark output The examples/ before/after blocks were authored by hand, not produced by a model. Issue #127 correctly noted that nobody hand-rolls quicksort for "sort this array" - every model just calls .sort(). Regenerate all examples verbatim from a real benchmark run (Claude Haiku 4.5, no-skill arm vs ponytail arm, benchmarks/output.json) so the before/after is reproducible, not authored: email 75->3, debounce 116->10, csv 20->3, countdown 267->9, rate-limit 128->10 LOC - Delete sorting.md (pure strawman) plus the other hand-written caricatures (api-endpoint, caching, date-picker) - Add benchmarks/generate-examples.mjs to regenerate examples from any run - examples/README.md indexes the set and documents how to reproduce Closes #127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-17 04:35:22 +02:00
parent 1b4914159e
commit 45f7d2f83f
11 changed files with 1162 additions and 265 deletions
@@ -0,0 +1,211 @@
+# Debounce
+
+**Task:** "Add debounce to a search input in vanilla JavaScript. It currently fires an API call on every keystroke."
+
+Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
+
+## Without Ponytail — 116 lines of code
+
+# Debounce Search Input
+
+Here's a practical implementation to prevent excessive API calls:
+
+## Basic Debounce Function
+
+```javascript
+function debounce(func, delay) {
+  let timeoutId;
+  return function (...args) {
+    clearTimeout(timeoutId);
+    timeoutId = setTimeout(() => func(...args), delay);
+  };
+}
+
+// Usage
+const searchInput = document.getElementById('search');
+const resultsContainer = document.getElementById('results');
+
+const handleSearch = async (query) => {
+  if (!query.trim()) {
+    resultsContainer.innerHTML = '';
+    return;
+  }
+  
+  try {
+    const response = await fetch(`/api/search?q=${encodeURIComponent(query)}`);
+    const data = await response.json();
+    displayResults(data);
+  } catch (error) {
+    console.error('Search failed:', error);
+  }
+};
+
+// Debounce the search function (300ms delay)
+const debouncedSearch = debounce(handleSearch, 300);
+
+searchInput.addEventListener('input', (e) => {
+  debouncedSearch(e.target.value);
+});
+```
+
+## Enhanced Version with Loading State
+
+```javascript
+const searchInput = document.getElementById('search');
+const resultsContainer = document.getElementById('results');
+const loadingIndicator = document.getElementById('loading');
+
+const handleSearch = async (query) => {
+  if (!query.trim()) {
+    resultsContainer.innerHTML = '';
+    loadingIndicator.style.display = 'none';
+    return;
+  }
+  
+  loadingIndicator.style.display = 'block';
+  
+  try {
+    const response = await fetch(`/api/search?q=${encodeURIComponent(query)}`);
+    if (!response.ok) throw new Error('Search failed');
+    
+    const data = await response.json();
+    displayResults(data);
+  } catch (error) {
+    resultsContainer.innerHTML = '<p class="error">Search failed. Try again.</p>';
+    console.error('Search error:', error);
+  } finally {
+    loadingIndicator.style.display = 'none';
+  }
+};
+
+const debouncedSearch = debounce(handleSearch, 300);
+
+searchInput.addEventListener('input', (e) => {
+  debouncedSearch(e.target.value);
+});
+
+function displayResults(data) {
+  resultsContainer.innerHTML = data
+    .map(item => `<div class="result">${item.title}</div>`)
+    .join('');
+}
+```
+
+## Advanced: Debounce with Cancel & Immediate Options
+
+```javascript
+function debounce(func, delay, options = {}) {
+  let timeoutId;
+  let lastArgs;
+  
+  const debounced = function (...args) {
+    lastArgs = args;
+    clearTimeout(timeoutId);
+    
+    if (options.immediate && !timeoutId) {
+      func(...args);
+    }
+    
+    timeoutId = setTimeout(() => {
+      if (!options.immediate) {
+        func(...lastArgs);
+      }
+      timeoutId = null;
+    }, delay);
+  };
+  
+  // Allow manual cancellation
+  debounced.cancel = () => clearTimeout(timeoutId);
+  
+  return debounced;
+}
+
+// Usage with options
+const debouncedSearch = debounce(handleSearch, 300, { immediate: false });
+
+// Cancel pending search if needed
+searchInput.addEventListener('blur', () => {
+  debouncedSearch.cancel();
+});
+```
+
+## HTML Example
+
+```html
+<div class="search-container">
+  <input 
+    type="text" 
+    id="search" 
+    placeholder="Search..."
+    autocomplete="off"
+  >
+  <div id="loading" style="display: none;">Loading...</div>
+  <div id="results"></div>
+</div>
+
+<style>
+  .search-container {
+    max-width: 500px;
+    margin: 20px auto;
+  }
+  
+  #search {
+    width: 100%;
+    padding: 10px;
+    font-size: 16px;
+    border: 1px solid #ddd;
+    border-radius: 4px;
+  }
+  
+  #results {
+    margin-top: 10px;
+  }
+  
+  .result {
+    padding: 10px;
+    border-bottom: 1px solid #eee;
+    cursor: pointer;
+  }
+  
+  .result:hover {
+    background-color: #f5f5f5;
+  }
+  
+  .error {
+    color: red;
+    padding: 10px;
+  }
+</style>
+```
+
+## Key Benefits
+
+| Feature | Benefit |
+|---------|---------|
+| **Debounce** | Reduces API calls from 100+ to 1-2 per search |
+| **300ms delay** | Balances responsiveness with performance |
+| **Loading state** | Provides user feedback |
+| **Error handling** | Graceful failure management |
+| **Cancel method** | Cleanup when needed |
+
+**Typical keystroke reduction:** 10 characters = 10 API calls → 1-2 API calls with debounce ✅
+
+## With Ponytail — 10 lines of code
+
+```javascript
+const searchInput = document.querySelector('input[type="search"]');
+let debounceTimer;
+
+searchInput.addEventListener('input', (e) => {
+  clearTimeout(debounceTimer);
+  debounceTimer = setTimeout(() => {
+    fetch(`/api/search?q=${encodeURIComponent(e.target.value)}`)
+      .then(r => r.json())
+      .then(data => console.log(data)); // replace with your render
+  }, 300);
+});
+```
+
+Skipped: debounce utility function, class wrapper, config object — `setTimeout` + `clearTimeout` is the debounce. Add a utility when you need it on 3+ inputs.
+
+**116 → 10 lines of code** — same model, same prompt.