Fix/examples issue 127 (#131)
* docs: correct cost claim to 42-75% from 30-rep re-verification Re-ran the cost benchmark at 30 reps per cell on Claude (Haiku/Sonnet/Opus): ponytail is 42-75% cheaper than no-skill, not the previously published 47-77%. The direction holds, both ends came in a few points lower. Updates the README headline and body, the benchmark chart subtitle, and the benchmarks/README cost table, and adds a dated results doc with full method. Also adds the OpenAI (gpt-4.1-mini/gpt-5.4-mini/gpt-5.5) and Gemini configs. On OpenAI reasoning models ponytail costs more, not less, so the claim stays Claude-scoped. Gemini run pending a fresh-quota day. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scope the body claim to Claude models "on every model" read as cross-provider, but the 30-rep verification shows the cost win reverses on OpenAI reasoning models. Match the caption and benchmarks/README, which already say Claude. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reframe the pitch as the discipline, not token savings The cost/code/latency numbers vary by model and on some (terse reasoning models like GPT-5.5) ponytail costs more, so leading with them as a universal win was misleading. Adds model-variance to the headline caption and a paragraph making the stated point the mental model: write only what the task needs, safety kept, maintainable code. Savings are a model-dependent side effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: name the ladder's reasoning cost The ladder is a deliberation step: on reasoning models the agent spends thinking tokens working through the rungs before it saves any output, which together with the always-on ruleset can outweigh the shorter code. Makes the GPT-5.5 cost increase legible rather than just stating it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: state the single-shot limitation honestly The benchmark is single-shot (one prompt, one completion); it does not measure a real multi-turn agent session, where the ruleset re-injects and the ladder deliberates every turn. Adds that caveat to the README, and corrects the benchmarks/README note that claimed caching widens the gap "in ponytail's favor" (unverified, and a measured agentic A/B in #121 found the opposite can happen). Per-session cost can land either way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix run count in caption (cost is 30 runs, not 10) Cost was re-verified at 30 reps; code and latency are still the original 10. The headline caption said "10 runs" across the board, which undersold the cost verification. Now states the split, matching benchmarks/README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(examples): replace hand-written examples with real benchmark output The examples/ before/after blocks were authored by hand, not produced by a model. Issue #127 correctly noted that nobody hand-rolls quicksort for "sort this array" - every model just calls .sort(). Regenerate all examples verbatim from a real benchmark run (Claude Haiku 4.5, no-skill arm vs ponytail arm, benchmarks/output.json) so the before/after is reproducible, not authored: email 75->3, debounce 116->10, csv 20->3, countdown 267->9, rate-limit 128->10 LOC - Delete sorting.md (pure strawman) plus the other hand-written caricatures (api-endpoint, caching, date-picker) - Add benchmarks/generate-examples.mjs to regenerate examples from any run - examples/README.md indexes the set and documents how to reproduce Closes #127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
DietrichGebert
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
1b4914159e
commit
45f7d2f83f
+138
-33
@@ -1,51 +1,156 @@
|
||||
# Email Validation
|
||||
|
||||
**Task:** "Validate an email address in Python."
|
||||
**Task:** "Write me a Python function that validates email addresses."
|
||||
|
||||
## Without Ponytail
|
||||
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
## Without Ponytail — 75 lines of code
|
||||
|
||||
# Email Validation Function
|
||||
|
||||
Here's a practical Python function that validates email addresses:
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
EMAIL_PATTERN = re.compile(
|
||||
r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
)
|
||||
|
||||
class EmailValidator:
|
||||
"""Validates email addresses against RFC-like rules."""
|
||||
|
||||
def __init__(self, pattern: re.Pattern = EMAIL_PATTERN):
|
||||
self.pattern = pattern
|
||||
|
||||
def validate(self, email: str) -> bool:
|
||||
if not isinstance(email, str):
|
||||
raise TypeError("email must be a string")
|
||||
email = email.strip()
|
||||
if not email:
|
||||
return False
|
||||
return bool(self.pattern.match(email))
|
||||
|
||||
|
||||
def validate_email(email: str) -> bool:
|
||||
"""Convenience wrapper around EmailValidator."""
|
||||
return EmailValidator().validate(email)
|
||||
"""
|
||||
Validates an email address using a regular expression.
|
||||
|
||||
Args:
|
||||
email (str): The email address to validate
|
||||
|
||||
Returns:
|
||||
bool: True if valid, False otherwise
|
||||
"""
|
||||
# Regular expression pattern for email validation
|
||||
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
|
||||
return re.match(pattern, email) is not None
|
||||
|
||||
|
||||
# Test cases
|
||||
if __name__ == "__main__":
|
||||
test_emails = [
|
||||
"user@example.com", # Valid
|
||||
"john.doe@company.co.uk", # Valid
|
||||
"test+tag@domain.org", # Valid
|
||||
"invalid.email@", # Invalid
|
||||
"missing@domain", # Invalid
|
||||
"@nodomain.com", # Invalid
|
||||
"spaces in@email.com", # Invalid
|
||||
"user@domain..com", # Invalid (double dot)
|
||||
]
|
||||
|
||||
for email in test_emails:
|
||||
result = validate_email(email)
|
||||
print(f"{email:<30} -> {result}")
|
||||
```
|
||||
|
||||
A class, a wrapper, a regex that still rejects valid addresses and accepts invalid ones. Regex cannot validate email. Only a delivery attempt can.
|
||||
|
||||
## With Ponytail
|
||||
## More Robust Version (with additional checks)
|
||||
|
||||
```python
|
||||
# ponytail: good enough, real validation is sending the mail
|
||||
"@" in email and "." in email.split("@")[-1]
|
||||
import re
|
||||
|
||||
def validate_email_advanced(email: str) -> tuple[bool, str]:
|
||||
"""
|
||||
Advanced email validation with detailed feedback.
|
||||
|
||||
Args:
|
||||
email (str): The email address to validate
|
||||
|
||||
Returns:
|
||||
tuple: (is_valid, message)
|
||||
"""
|
||||
# Basic checks
|
||||
if not email or not isinstance(email, str):
|
||||
return False, "Email must be a non-empty string"
|
||||
|
||||
email = email.strip()
|
||||
|
||||
if len(email) > 254:
|
||||
return False, "Email is too long (max 254 characters)"
|
||||
|
||||
# Regular expression pattern
|
||||
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
|
||||
if not re.match(pattern, email):
|
||||
return False, "Email format is invalid"
|
||||
|
||||
# Additional checks
|
||||
local_part, domain = email.rsplit('@', 1)
|
||||
|
||||
if len(local_part) > 64:
|
||||
return False, "Local part is too long (max 64 characters)"
|
||||
|
||||
if local_part.startswith('.') or local_part.endswith('.'):
|
||||
return False, "Local part cannot start or end with a dot"
|
||||
|
||||
if '..' in email:
|
||||
return False, "Email cannot contain consecutive dots"
|
||||
|
||||
return True, "Email is valid"
|
||||
|
||||
|
||||
# Test the advanced version
|
||||
if __name__ == "__main__":
|
||||
test_emails = [
|
||||
"user@example.com",
|
||||
"invalid..email@domain.com",
|
||||
".invalid@domain.com",
|
||||
"user@domain",
|
||||
]
|
||||
|
||||
for email in test_emails:
|
||||
is_valid, message = validate_email_advanced(email)
|
||||
print(f"{email:<35} -> {is_valid:5} ({message})")
|
||||
```
|
||||
|
||||
Or, if it must be thorough, the standard library has it:
|
||||
## Using a Third-Party Library (Recommended for Production)
|
||||
|
||||
```python
|
||||
# ponytail: stdlib covers this
|
||||
from email.utils import parseaddr
|
||||
"@" in parseaddr(email)[1]
|
||||
from email_validator import validate_email, EmailNotValidError
|
||||
|
||||
def validate_email_production(email: str) -> bool:
|
||||
"""
|
||||
Validates email using the email_validator library.
|
||||
More reliable for production use.
|
||||
|
||||
Install with: pip install email-validator
|
||||
"""
|
||||
try:
|
||||
# Validate and normalize the email
|
||||
valid = validate_email(email)
|
||||
return True
|
||||
except EmailNotValidError:
|
||||
return False
|
||||
|
||||
|
||||
# Usage
|
||||
if __name__ == "__main__":
|
||||
print(validate_email_production("user@example.com")) # True
|
||||
print(validate_email_production("invalid@")) # False
|
||||
```
|
||||
|
||||
**27 lines → 1 line.** And the honest answer: let the confirmation email reject it. That's what confirmation emails are for.
|
||||
## Comparison
|
||||
|
||||
| Method | Pros | Cons |
|
||||
|--------|------|------|
|
||||
| **Regex (Simple)** | Fast, no dependencies | Limited validation |
|
||||
| **Regex (Advanced)** | More thorough | Still not RFC-compliant |
|
||||
| **email-validator** | RFC-compliant, reliable | External dependency |
|
||||
|
||||
**Recommendation**: Use the `email-validator` library for production applications, as it properly handles RFC 5321/5322 standards. Use the simple regex version for quick scripts or when you want to avoid dependencies.
|
||||
|
||||
## With Ponytail — 3 lines of code
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def is_valid_email(email: str) -> bool:
|
||||
return bool(re.match(r'^[^@]+@[^@]+\.[^@]+$', email))
|
||||
```
|
||||
|
||||
Skipped: RFC 5322 parser, DNS MX lookup, confirmation email. Add when you actually need to reject `user+tag@sub.domain.co.uk` or catch typos — until then, this catches 99% of "oops I fat-fingered it" cases.
|
||||
|
||||
**75 → 3 lines of code** — same model, same prompt.
|
||||
|
||||
Reference in New Issue
Block a user