Iterative Retrieval

1---2name: iterative-retrieval3description: Pattern for progressively refining context retrieval to solve the subagent context problem4origin: ECC5---6 7# Iterative Retrieval Pattern8 9Solves the "context problem" in multi-agent workflows where subagents don't know what context they need until they start working.10 11## When to Activate12 13- Spawning subagents that need codebase context they cannot predict upfront14- Building multi-agent workflows where context is progressively refined15- Encountering "context too large" or "missing context" failures in agent tasks16- Designing RAG-like retrieval pipelines for code exploration17- Optimizing token usage in agent orchestration18 19## The Problem20 21Subagents are spawned with limited context. They don't know:22- Which files contain relevant code23- What patterns exist in the codebase24- What terminology the project uses25 26Standard approaches fail:27- **Send everything**: Exceeds context limits28- **Send nothing**: Agent lacks critical information29- **Guess what's needed**: Often wrong30 31## The Solution: Iterative Retrieval32 33A 4-phase loop that progressively refines context:34 35```36┌─────────────────────────────────────────────┐37│                                             │38│   ┌──────────┐      ┌──────────┐            │39│   │ DISPATCH │─────│ EVALUATE │            │40│   └──────────┘      └──────────┘            │41│        ▲                  │                 │42│        │                  ▼                 │43│   ┌──────────┐      ┌──────────┐            │44│   │   LOOP   │─────│  REFINE  │            │45│   └──────────┘      └──────────┘            │46│                                             │47│        Max 3 cycles, then proceed           │48└─────────────────────────────────────────────┘49```50 51### Phase 1: DISPATCH52 53Initial broad query to gather candidate files:54 55```javascript56// Start with high-level intent57const initialQuery = {58  patterns: ['src/**/*.ts', 'lib/**/*.ts'],59  keywords: ['authentication', 'user', 'session'],60  excludes: ['*.test.ts', '*.spec.ts']61};62 63// Dispatch to retrieval agent64const candidates = await retrieveFiles(initialQuery);65```66 67### Phase 2: EVALUATE68 69Assess retrieved content for relevance:70 71```javascript72function evaluateRelevance(files, task) {73  return files.map(file => ({74    path: file.path,75    relevance: scoreRelevance(file.content, task),76    reason: explainRelevance(file.content, task),77    missingContext: identifyGaps(file.content, task)78  }));79}80```81 82Scoring criteria:83- **High (0.8-1.0)**: Directly implements target functionality84- **Medium (0.5-0.7)**: Contains related patterns or types85- **Low (0.2-0.4)**: Tangentially related86- **None (0-0.2)**: Not relevant, exclude87 88### Phase 3: REFINE89 90Update search criteria based on evaluation:91 92```javascript93function refineQuery(evaluation, previousQuery) {94  return {95    // Add new patterns discovered in high-relevance files96    patterns: [...previousQuery.patterns, ...extractPatterns(evaluation)],97 98    // Add terminology found in codebase99    keywords: [...previousQuery.keywords, ...extractKeywords(evaluation)],100 101    // Exclude confirmed irrelevant paths102    excludes: [...previousQuery.excludes, ...evaluation103      .filter(e => e.relevance < 0.2)104      .map(e => e.path)105    ],106 107    // Target specific gaps108    focusAreas: evaluation109      .flatMap(e => e.missingContext)110      .filter(unique)111  };112}113```114 115### Phase 4: LOOP116 117Repeat with refined criteria (max 3 cycles):118 119```javascript120async function iterativeRetrieve(task, maxCycles = 3) {121  let query = createInitialQuery(task);122  let bestContext = [];123 124  for (let cycle = 0; cycle < maxCycles; cycle++) {125    const candidates = await retrieveFiles(query);126    const evaluation = evaluateRelevance(candidates, task);127 128    // Check if we have sufficient context129    const highRelevance = evaluation.filter(e => e.relevance >= 0.7);130    if (highRelevance.length >= 3 && !hasCriticalGaps(evaluation)) {131      return highRelevance;132    }133 134    // Refine and continue135    query = refineQuery(evaluation, query);136    bestContext = mergeContext(bestContext, highRelevance);137  }138 139  return bestContext;140}141```142 143## Practical Examples144 145### Example 1: Bug Fix Context146 147```148Task: "Fix the authentication token expiry bug"149 150Cycle 1:151  DISPATCH: Search for "token", "auth", "expiry" in src/**152  EVALUATE: Found auth.ts (0.9), tokens.ts (0.8), user.ts (0.3)153  REFINE: Add "refresh", "jwt" keywords; exclude user.ts154 155Cycle 2:156  DISPATCH: Search refined terms157  EVALUATE: Found session-manager.ts (0.95), jwt-utils.ts (0.85)158  REFINE: Sufficient context (2 high-relevance files)159 160Result: auth.ts, tokens.ts, session-manager.ts, jwt-utils.ts161```162 163### Example 2: Feature Implementation164 165```166Task: "Add rate limiting to API endpoints"167 168Cycle 1:169  DISPATCH: Search "rate", "limit", "api" in routes/**170  EVALUATE: No matches - codebase uses "throttle" terminology171  REFINE: Add "throttle", "middleware" keywords172 173Cycle 2:174  DISPATCH: Search refined terms175  EVALUATE: Found throttle.ts (0.9), middleware/index.ts (0.7)176  REFINE: Need router patterns177 178Cycle 3:179  DISPATCH: Search "router", "express" patterns180  EVALUATE: Found router-setup.ts (0.8)181  REFINE: Sufficient context182 183Result: throttle.ts, middleware/index.ts, router-setup.ts184```185 186## Integration with Agents187 188Use in agent prompts:189 190```markdown191When retrieving context for this task:1921. Start with broad keyword search1932. Evaluate each file's relevance (0-1 scale)1943. Identify what context is still missing1954. Refine search criteria and repeat (max 3 cycles)1965. Return files with relevance >= 0.7197```198 199## Best Practices200 2011. **Start broad, narrow progressively** - Don't over-specify initial queries2022. **Learn codebase terminology** - First cycle often reveals naming conventions2033. **Track what's missing** - Explicit gap identification drives refinement2044. **Stop at "good enough"** - 3 high-relevance files beats 10 mediocre ones2055. **Exclude confidently** - Low-relevance files won't become relevant206 207## Related208 209- [The Longform Guide](https://x.com/affaanmustafa/status/2014040193557471352) - Subagent orchestration section210- `continuous-learning` skill - For patterns that improve over time211- Agent definitions bundled with ECC (manual install path: `agents/`)