fix(examples): replace hand-written examples with real benchmark output
The examples/ before/after blocks were authored by hand, not produced by a model. Issue #127 correctly noted that nobody hand-rolls quicksort for "sort this array" - every model just calls .sort(). Regenerate all examples verbatim from a real benchmark run (Claude Haiku 4.5, no-skill arm vs ponytail arm, benchmarks/output.json) so the before/after is reproducible, not authored: email 75->3, debounce 116->10, csv 20->3, countdown 267->9, rate-limit 128->10 LOC - Delete sorting.md (pure strawman) plus the other hand-written caricatures (api-endpoint, caching, date-picker) - Add benchmarks/generate-examples.mjs to regenerate examples from any run - examples/README.md indexes the set and documents how to reproduce Closes #127 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Emeriko
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
f72c1459dd
commit
08440027f2
+138
-33
@@ -1,51 +1,156 @@
|
||||
# Email Validation
|
||||
|
||||
**Task:** "Validate an email address in Python."
|
||||
**Task:** "Write me a Python function that validates email addresses."
|
||||
|
||||
## Without Ponytail
|
||||
Verbatim model output from a benchmark run — Claude Haiku 4.5, no-skill arm vs ponytail arm, temperature 1, source `benchmarks/output.json`. Reproduce: `npx promptfoo@latest eval -c benchmarks/promptfooconfig.yaml`.
|
||||
|
||||
## Without Ponytail — 75 lines of code
|
||||
|
||||
# Email Validation Function
|
||||
|
||||
Here's a practical Python function that validates email addresses:
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
EMAIL_PATTERN = re.compile(
|
||||
r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
)
|
||||
|
||||
class EmailValidator:
|
||||
"""Validates email addresses against RFC-like rules."""
|
||||
|
||||
def __init__(self, pattern: re.Pattern = EMAIL_PATTERN):
|
||||
self.pattern = pattern
|
||||
|
||||
def validate(self, email: str) -> bool:
|
||||
if not isinstance(email, str):
|
||||
raise TypeError("email must be a string")
|
||||
email = email.strip()
|
||||
if not email:
|
||||
return False
|
||||
return bool(self.pattern.match(email))
|
||||
|
||||
|
||||
def validate_email(email: str) -> bool:
|
||||
"""Convenience wrapper around EmailValidator."""
|
||||
return EmailValidator().validate(email)
|
||||
"""
|
||||
Validates an email address using a regular expression.
|
||||
|
||||
Args:
|
||||
email (str): The email address to validate
|
||||
|
||||
Returns:
|
||||
bool: True if valid, False otherwise
|
||||
"""
|
||||
# Regular expression pattern for email validation
|
||||
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
|
||||
return re.match(pattern, email) is not None
|
||||
|
||||
|
||||
# Test cases
|
||||
if __name__ == "__main__":
|
||||
test_emails = [
|
||||
"user@example.com", # Valid
|
||||
"john.doe@company.co.uk", # Valid
|
||||
"test+tag@domain.org", # Valid
|
||||
"invalid.email@", # Invalid
|
||||
"missing@domain", # Invalid
|
||||
"@nodomain.com", # Invalid
|
||||
"spaces in@email.com", # Invalid
|
||||
"user@domain..com", # Invalid (double dot)
|
||||
]
|
||||
|
||||
for email in test_emails:
|
||||
result = validate_email(email)
|
||||
print(f"{email:<30} -> {result}")
|
||||
```
|
||||
|
||||
A class, a wrapper, a regex that still rejects valid addresses and accepts invalid ones. Regex cannot validate email. Only a delivery attempt can.
|
||||
|
||||
## With Ponytail
|
||||
## More Robust Version (with additional checks)
|
||||
|
||||
```python
|
||||
# ponytail: good enough, real validation is sending the mail
|
||||
"@" in email and "." in email.split("@")[-1]
|
||||
import re
|
||||
|
||||
def validate_email_advanced(email: str) -> tuple[bool, str]:
|
||||
"""
|
||||
Advanced email validation with detailed feedback.
|
||||
|
||||
Args:
|
||||
email (str): The email address to validate
|
||||
|
||||
Returns:
|
||||
tuple: (is_valid, message)
|
||||
"""
|
||||
# Basic checks
|
||||
if not email or not isinstance(email, str):
|
||||
return False, "Email must be a non-empty string"
|
||||
|
||||
email = email.strip()
|
||||
|
||||
if len(email) > 254:
|
||||
return False, "Email is too long (max 254 characters)"
|
||||
|
||||
# Regular expression pattern
|
||||
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
|
||||
|
||||
if not re.match(pattern, email):
|
||||
return False, "Email format is invalid"
|
||||
|
||||
# Additional checks
|
||||
local_part, domain = email.rsplit('@', 1)
|
||||
|
||||
if len(local_part) > 64:
|
||||
return False, "Local part is too long (max 64 characters)"
|
||||
|
||||
if local_part.startswith('.') or local_part.endswith('.'):
|
||||
return False, "Local part cannot start or end with a dot"
|
||||
|
||||
if '..' in email:
|
||||
return False, "Email cannot contain consecutive dots"
|
||||
|
||||
return True, "Email is valid"
|
||||
|
||||
|
||||
# Test the advanced version
|
||||
if __name__ == "__main__":
|
||||
test_emails = [
|
||||
"user@example.com",
|
||||
"invalid..email@domain.com",
|
||||
".invalid@domain.com",
|
||||
"user@domain",
|
||||
]
|
||||
|
||||
for email in test_emails:
|
||||
is_valid, message = validate_email_advanced(email)
|
||||
print(f"{email:<35} -> {is_valid:5} ({message})")
|
||||
```
|
||||
|
||||
Or, if it must be thorough, the standard library has it:
|
||||
## Using a Third-Party Library (Recommended for Production)
|
||||
|
||||
```python
|
||||
# ponytail: stdlib covers this
|
||||
from email.utils import parseaddr
|
||||
"@" in parseaddr(email)[1]
|
||||
from email_validator import validate_email, EmailNotValidError
|
||||
|
||||
def validate_email_production(email: str) -> bool:
|
||||
"""
|
||||
Validates email using the email_validator library.
|
||||
More reliable for production use.
|
||||
|
||||
Install with: pip install email-validator
|
||||
"""
|
||||
try:
|
||||
# Validate and normalize the email
|
||||
valid = validate_email(email)
|
||||
return True
|
||||
except EmailNotValidError:
|
||||
return False
|
||||
|
||||
|
||||
# Usage
|
||||
if __name__ == "__main__":
|
||||
print(validate_email_production("user@example.com")) # True
|
||||
print(validate_email_production("invalid@")) # False
|
||||
```
|
||||
|
||||
**27 lines → 1 line.** And the honest answer: let the confirmation email reject it. That's what confirmation emails are for.
|
||||
## Comparison
|
||||
|
||||
| Method | Pros | Cons |
|
||||
|--------|------|------|
|
||||
| **Regex (Simple)** | Fast, no dependencies | Limited validation |
|
||||
| **Regex (Advanced)** | More thorough | Still not RFC-compliant |
|
||||
| **email-validator** | RFC-compliant, reliable | External dependency |
|
||||
|
||||
**Recommendation**: Use the `email-validator` library for production applications, as it properly handles RFC 5321/5322 standards. Use the simple regex version for quick scripts or when you want to avoid dependencies.
|
||||
|
||||
## With Ponytail — 3 lines of code
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def is_valid_email(email: str) -> bool:
|
||||
return bool(re.match(r'^[^@]+@[^@]+\.[^@]+$', email))
|
||||
```
|
||||
|
||||
Skipped: RFC 5322 parser, DNS MX lookup, confirmation email. Add when you actually need to reject `user+tag@sub.domain.co.uk` or catch typos — until then, this catches 99% of "oops I fat-fingered it" cases.
|
||||
|
||||
**75 → 3 lines of code** — same model, same prompt.
|
||||
|
||||
Reference in New Issue
Block a user