Regex Vs Llm Structured Text

1---2name: regex-vs-llm-structured-text3description: Decision framework for choosing between regex and LLM when parsing structured text — start with regex, add LLM only for low-confidence edge cases.4origin: ECC5---6 7# Regex vs LLM for Structured Text Parsing8 9A practical decision framework for parsing structured text (quizzes, forms, invoices, documents). The key insight: regex handles 95-98% of cases cheaply and deterministically. Reserve expensive LLM calls for the remaining edge cases.10 11## When to Activate12 13- Parsing structured text with repeating patterns (questions, forms, tables)14- Deciding between regex and LLM for text extraction15- Building hybrid pipelines that combine both approaches16- Optimizing cost/accuracy tradeoffs in text processing17 18## Decision Framework19 20```21Is the text format consistent and repeating?22├── Yes (>90% follows a pattern) → Start with Regex23│   ├── Regex handles 95%+ → Done, no LLM needed24│   └── Regex handles <95% → Add LLM for edge cases only25└── No (free-form, highly variable) → Use LLM directly26```27 28## Architecture Pattern29 30```31Source Text32    │33    ▼34[Regex Parser] ─── Extracts structure (95-98% accuracy)35    │36    ▼37[Text Cleaner] ─── Removes noise (markers, page numbers, artifacts)38    │39    ▼40[Confidence Scorer] ─── Flags low-confidence extractions41    │42    ├── High confidence (≥0.95) → Direct output43    │44    └── Low confidence (<0.95) → [LLM Validator] → Output45```46 47## Implementation48 49### 1. Regex Parser (Handles the Majority)50 51```python52import re53from dataclasses import dataclass54 55@dataclass(frozen=True)56class ParsedItem:57    id: str58    text: str59    choices: tuple[str, ...]60    answer: str61    confidence: float = 1.062 63def parse_structured_text(content: str) -> list[ParsedItem]:64    """Parse structured text using regex patterns."""65    pattern = re.compile(66        r"(?P<id>\d+)\.\s*(?P<text>.+?)\n"67        r"(?P<choices>(?:[A-D]\..+?\n)+)"68        r"Answer:\s*(?P<answer>[A-D])",69        re.MULTILINE | re.DOTALL,70    )71    items = []72    for match in pattern.finditer(content):73        choices = tuple(74            c.strip() for c in re.findall(r"[A-D]\.\s*(.+)", match.group("choices"))75        )76        items.append(ParsedItem(77            id=match.group("id"),78            text=match.group("text").strip(),79            choices=choices,80            answer=match.group("answer"),81        ))82    return items83```84 85### 2. Confidence Scoring86 87Flag items that may need LLM review:88 89```python90@dataclass(frozen=True)91class ConfidenceFlag:92    item_id: str93    score: float94    reasons: tuple[str, ...]95 96def score_confidence(item: ParsedItem) -> ConfidenceFlag:97    """Score extraction confidence and flag issues."""98    reasons = []99    score = 1.0100 101    if len(item.choices) < 3:102        reasons.append("few_choices")103        score -= 0.3104 105    if not item.answer:106        reasons.append("missing_answer")107        score -= 0.5108 109    if len(item.text) < 10:110        reasons.append("short_text")111        score -= 0.2112 113    return ConfidenceFlag(114        item_id=item.id,115        score=max(0.0, score),116        reasons=tuple(reasons),117    )118 119def identify_low_confidence(120    items: list[ParsedItem],121    threshold: float = 0.95,122) -> list[ConfidenceFlag]:123    """Return items below confidence threshold."""124    flags = [score_confidence(item) for item in items]125    return [f for f in flags if f.score < threshold]126```127 128### 3. LLM Validator (Edge Cases Only)129 130```python131def validate_with_llm(132    item: ParsedItem,133    original_text: str,134    client,135) -> ParsedItem:136    """Use LLM to fix low-confidence extractions."""137    response = client.messages.create(138        model="claude-haiku-4-5-20251001",  # Cheapest model for validation139        max_tokens=500,140        messages=[{141            "role": "user",142            "content": (143                f"Extract the question, choices, and answer from this text.\n\n"144                f"Text: {original_text}\n\n"145                f"Current extraction: {item}\n\n"146                f"Return corrected JSON if needed, or 'CORRECT' if accurate."147            ),148        }],149    )150    # Parse LLM response and return corrected item...151    return corrected_item152```153 154### 4. Hybrid Pipeline155 156```python157def process_document(158    content: str,159    *,160    llm_client=None,161    confidence_threshold: float = 0.95,162) -> list[ParsedItem]:163    """Full pipeline: regex -> confidence check -> LLM for edge cases."""164    # Step 1: Regex extraction (handles 95-98%)165    items = parse_structured_text(content)166 167    # Step 2: Confidence scoring168    low_confidence = identify_low_confidence(items, confidence_threshold)169 170    if not low_confidence or llm_client is None:171        return items172 173    # Step 3: LLM validation (only for flagged items)174    low_conf_ids = {f.item_id for f in low_confidence}175    result = []176    for item in items:177        if item.id in low_conf_ids:178            result.append(validate_with_llm(item, content, llm_client))179        else:180            result.append(item)181 182    return result183```184 185## Real-World Metrics186 187From a production quiz parsing pipeline (410 items):188 189| Metric | Value |190|--------|-------|191| Regex success rate | 98.0% |192| Low confidence items | 8 (2.0%) |193| LLM calls needed | ~5 |194| Cost savings vs all-LLM | ~95% |195| Test coverage | 93% |196 197## Best Practices198 199- **Start with regex** — even imperfect regex gives you a baseline to improve200- **Use confidence scoring** to programmatically identify what needs LLM help201- **Use the cheapest LLM** for validation (Haiku-class models are sufficient)202- **Never mutate** parsed items — return new instances from cleaning/validation steps203- **TDD works well** for parsers — write tests for known patterns first, then edge cases204- **Log metrics** (regex success rate, LLM call count) to track pipeline health205 206## Anti-Patterns to Avoid207 208- Sending all text to an LLM when regex handles 95%+ of cases (expensive and slow)209- Using regex for free-form, highly variable text (LLM is better here)210- Skipping confidence scoring and hoping regex "just works"211- Mutating parsed objects during cleaning/validation steps212- Not testing edge cases (malformed input, missing fields, encoding issues)213 214## When to Use215 216- Quiz/exam question parsing217- Form data extraction218- Invoice/receipt processing219- Document structure parsing (headers, sections, tables)220- Any structured text with repeating patterns where cost matters