Santa Method

1---2name: santa-method3description: "Multi-agent adversarial verification with convergence loop. Two independent review agents must both pass before output ships."4origin: "Ronald Skelton - Founder, RapportScore.ai"5---6 7# Santa Method8 9Multi-agent adversarial verification framework. Make a list, check it twice. If it's naughty, fix it until it's nice.10 11The core insight: a single agent reviewing its own output shares the same biases, knowledge gaps, and systematic errors that produced the output. Two independent reviewers with no shared context break this failure mode.12 13## When to Activate14 15Invoke this skill when:16- Output will be published, deployed, or consumed by end users17- Compliance, regulatory, or brand constraints must be enforced18- Code ships to production without human review19- Content accuracy matters (technical docs, educational material, customer-facing copy)20- Batch generation at scale where spot-checking misses systemic patterns21- Hallucination risk is elevated (claims, statistics, API references, legal language)22 23Do NOT use for internal drafts, exploratory research, or tasks with deterministic verification (use build/test/lint pipelines for those).24 25## Architecture26 27```28┌─────────────┐29│  GENERATOR   │  Phase 1: Make a List30│  (Agent A)   │  Produce the deliverable31└──────┬───────┘32       │ output33       ▼34┌──────────────────────────────┐35│     DUAL INDEPENDENT REVIEW   │  Phase 2: Check It Twice36│                                │37│  ┌───────────┐ ┌───────────┐  │  Two agents, same rubric,38│  │ Reviewer B │ │ Reviewer C │  │  no shared context39│  └─────┬─────┘ └─────┬─────┘  │40│        │              │        │41└────────┼──────────────┼────────┘42         │              │43         ▼              ▼44┌──────────────────────────────┐45│        VERDICT GATE           │  Phase 3: Naughty or Nice46│                                │47│  B passes AND C passes → NICE  │  Both must pass.48│  Otherwise → NAUGHTY           │  No exceptions.49└──────┬──────────────┬─────────┘50       │              │51    NICE           NAUGHTY52       │              │53       ▼              ▼54   [ SHIP ]    ┌─────────────┐55               │  FIX CYCLE   │  Phase 4: Fix Until Nice56               │              │57               │ iteration++  │  Collect all flags.58               │ if i > MAX:  │  Fix all issues.59               │   escalate   │  Re-run both reviewers.60               │ else:        │  Loop until convergence.61               │   goto Ph.2  │62               └──────────────┘63```64 65## Phase Details66 67### Phase 1: Make a List (Generate)68 69Execute the primary task. No changes to your normal generation workflow. Santa Method is a post-generation verification layer, not a generation strategy.70 71```python72# The generator runs as normal73output = generate(task_spec)74```75 76### Phase 2: Check It Twice (Independent Dual Review)77 78Spawn two review agents in parallel. Critical invariants:79 801. **Context isolation** — neither reviewer sees the other's assessment812. **Identical rubric** — both receive the same evaluation criteria823. **Same inputs** — both receive the original spec AND the generated output834. **Structured output** — each returns a typed verdict, not prose84 85```python86REVIEWER_PROMPT = """87You are an independent quality reviewer. You have NOT seen any other review of this output.88 89## Task Specification90{task_spec}91 92## Output Under Review93{output}94 95## Evaluation Rubric96{rubric}97 98## Instructions99Evaluate the output against EACH rubric criterion. For each:100- PASS: criterion fully met, no issues101- FAIL: specific issue found (cite the exact problem)102 103Return your assessment as structured JSON:104{105  "verdict": "PASS" | "FAIL",106  "checks": [107    {"criterion": "...", "result": "PASS|FAIL", "detail": "..."}108  ],109  "critical_issues": ["..."],   // blockers that must be fixed110  "suggestions": ["..."]         // non-blocking improvements111}112 113Be rigorous. Your job is to find problems, not to approve.114"""115```116 117```python118# Spawn reviewers in parallel (Claude Code subagents)119review_b = Agent(prompt=REVIEWER_PROMPT.format(...), description="Santa Reviewer B")120review_c = Agent(prompt=REVIEWER_PROMPT.format(...), description="Santa Reviewer C")121 122# Both run concurrently — neither sees the other123```124 125### Rubric Design126 127The rubric is the most important input. Vague rubrics produce vague reviews. Every criterion must have an objective pass/fail condition.128 129| Criterion | Pass Condition | Failure Signal |130|-----------|---------------|----------------|131| Factual accuracy | All claims verifiable against source material or common knowledge | Invented statistics, wrong version numbers, nonexistent APIs |132| Hallucination-free | No fabricated entities, quotes, URLs, or references | Links to pages that don't exist, attributed quotes with no source |133| Completeness | Every requirement in the spec is addressed | Missing sections, skipped edge cases, incomplete coverage |134| Compliance | Passes all project-specific constraints | Banned terms used, tone violations, regulatory non-compliance |135| Internal consistency | No contradictions within the output | Section A says X, section B says not-X |136| Technical correctness | Code compiles/runs, algorithms are sound | Syntax errors, logic bugs, wrong complexity claims |137 138#### Domain-Specific Rubric Extensions139 140**Content/Marketing:**141- Brand voice adherence142- SEO requirements met (keyword density, meta tags, structure)143- No competitor trademark misuse144- CTA present and correctly linked145 146**Code:**147- Type safety (no `any` leaks, proper null handling)148- Error handling coverage149- Security (no secrets in code, input validation, injection prevention)150- Test coverage for new paths151 152**Compliance-Sensitive (regulated, legal, financial):**153- No outcome guarantees or unsubstantiated claims154- Required disclaimers present155- Approved terminology only156- Jurisdiction-appropriate language157 158### Phase 3: Naughty or Nice (Verdict Gate)159 160```python161def santa_verdict(review_b, review_c):162    """Both reviewers must pass. No partial credit."""163    if review_b.verdict == "PASS" and review_c.verdict == "PASS":164        return "NICE"  # Ship it165 166    # Merge flags from both reviewers, deduplicate167    all_issues = dedupe(review_b.critical_issues + review_c.critical_issues)168    all_suggestions = dedupe(review_b.suggestions + review_c.suggestions)169 170    return "NAUGHTY", all_issues, all_suggestions171```172 173Why both must pass: if only one reviewer catches an issue, that issue is real. The other reviewer's blind spot is exactly the failure mode Santa Method exists to eliminate.174 175### Phase 4: Fix Until Nice (Convergence Loop)176 177```python178MAX_ITERATIONS = 3179 180for iteration in range(MAX_ITERATIONS):181    verdict, issues, suggestions = santa_verdict(review_b, review_c)182 183    if verdict == "NICE":184        log_santa_result(output, iteration, "passed")185        return ship(output)186 187    # Fix all critical issues (suggestions are optional)188    output = fix_agent.execute(189        output=output,190        issues=issues,191        instruction="Fix ONLY the flagged issues. Do not refactor or add unrequested changes."192    )193 194    # Re-run BOTH reviewers on fixed output (fresh agents, no memory of previous round)195    review_b = Agent(prompt=REVIEWER_PROMPT.format(output=output, ...))196    review_c = Agent(prompt=REVIEWER_PROMPT.format(output=output, ...))197 198# Exhausted iterations — escalate199log_santa_result(output, MAX_ITERATIONS, "escalated")200escalate_to_human(output, issues)201```202 203Critical: each review round uses **fresh agents**. Reviewers must not carry memory from previous rounds, as prior context creates anchoring bias.204 205## Implementation Patterns206 207### Pattern A: Claude Code Subagents (Recommended)208 209Subagents provide true context isolation. Each reviewer is a separate process with no shared state.210 211```bash212# In a Claude Code session, use the Agent tool to spawn reviewers213# Both agents run in parallel for speed214```215 216```python217# Pseudocode for Agent tool invocation218reviewer_b = Agent(219    description="Santa Review B",220    prompt=f"Review this output for quality...\n\nRUBRIC:\n{rubric}\n\nOUTPUT:\n{output}"221)222reviewer_c = Agent(223    description="Santa Review C",224    prompt=f"Review this output for quality...\n\nRUBRIC:\n{rubric}\n\nOUTPUT:\n{output}"225)226```227 228### Pattern B: Sequential Inline (Fallback)229 230When subagents aren't available, simulate isolation with explicit context resets:231 2321. Generate output2332. New context: "You are Reviewer 1. Evaluate ONLY against this rubric. Find problems."2343. Record findings verbatim2354. Clear context completely2365. New context: "You are Reviewer 2. Evaluate ONLY against this rubric. Find problems."2376. Compare both reviews, fix, repeat238 239The subagent pattern is strictly superior — inline simulation risks context bleed between reviewers.240 241### Pattern C: Batch Sampling242 243For large batches (100+ items), full Santa on every item is cost-prohibitive. Use stratified sampling:244 2451. Run Santa on a random sample (10-15% of batch, minimum 5 items)2462. Categorize failures by type (hallucination, compliance, completeness, etc.)2473. If systematic patterns emerge, apply targeted fixes to the entire batch2484. Re-sample and re-verify the fixed batch2495. Continue until a clean sample passes250 251```python252import random253 254def santa_batch(items, rubric, sample_rate=0.15):255    sample = random.sample(items, max(5, int(len(items) * sample_rate)))256 257    for item in sample:258        result = santa_full(item, rubric)259        if result.verdict == "NAUGHTY":260            pattern = classify_failure(result.issues)261            items = batch_fix(items, pattern)  # Fix all items matching pattern262            return santa_batch(items, rubric)   # Re-sample263 264    return items  # Clean sample → ship batch265```266 267## Failure Modes and Mitigations268 269| Failure Mode | Symptom | Mitigation |270|-------------|---------|------------|271| Infinite loop | Reviewers keep finding new issues after fixes | Max iteration cap (3). Escalate. |272| Rubber stamping | Both reviewers pass everything | Adversarial prompt: "Your job is to find problems, not approve." |273| Subjective drift | Reviewers flag style preferences, not errors | Tight rubric with objective pass/fail criteria only |274| Fix regression | Fixing issue A introduces issue B | Fresh reviewers each round catch regressions |275| Reviewer agreement bias | Both reviewers miss the same thing | Mitigated by independence, not eliminated. For critical output, add a third reviewer or human spot-check. |276| Cost explosion | Too many iterations on large outputs | Batch sampling pattern. Budget caps per verification cycle. |277 278## Integration with Other Skills279 280| Skill | Relationship |281|-------|-------------|282| Verification Loop | Use for deterministic checks (build, lint, test). Santa for semantic checks (accuracy, hallucinations). Run verification-loop first, Santa second. |283| Eval Harness | Santa Method results feed eval metrics. Track pass@k across Santa runs to measure generator quality over time. |284| Continuous Learning v2 | Santa findings become instincts. Repeated failures on the same criterion → learned behavior to avoid the pattern. |285| Strategic Compact | Run Santa BEFORE compacting. Don't lose review context mid-verification. |286 287## Metrics288 289Track these to measure Santa Method effectiveness:290 291- **First-pass rate**: % of outputs that pass Santa on round 1 (target: >70%)292- **Mean iterations to convergence**: average rounds to NICE (target: <1.5)293- **Issue taxonomy**: distribution of failure types (hallucination vs. completeness vs. compliance)294- **Reviewer agreement**: % of issues flagged by both reviewers vs. only one (low agreement = rubric needs tightening)295- **Escape rate**: issues found post-ship that Santa should have caught (target: 0)296 297## Cost Analysis298 299Santa Method costs approximately 2-3x the token cost of generation alone per verification cycle. For most high-stakes output, this is a bargain:300 301```302Cost of Santa = (generation tokens) + 2×(review tokens per round) × (avg rounds)303Cost of NOT Santa = (reputation damage) + (correction effort) + (trust erosion)304```305 306For batch operations, the sampling pattern reduces cost to ~15-20% of full verification while catching >90% of systematic issues.