Content Hash Cache Pattern

1---2name: content-hash-cache-pattern3description: Cache expensive file processing results using SHA-256 content hashes — path-independent, auto-invalidating, with service layer separation.4origin: ECC5---6 7# Content-Hash File Cache Pattern8 9Cache expensive file processing results (PDF parsing, text extraction, image analysis) using SHA-256 content hashes as cache keys. Unlike path-based caching, this approach survives file moves/renames and auto-invalidates when content changes.10 11## When to Activate12 13- Building file processing pipelines (PDF, images, text extraction)14- Processing cost is high and same files are processed repeatedly15- Need a `--cache/--no-cache` CLI option16- Want to add caching to existing pure functions without modifying them17 18## Core Pattern19 20### 1. Content-Hash Based Cache Key21 22Use file content (not path) as the cache key:23 24```python25import hashlib26from pathlib import Path27 28_HASH_CHUNK_SIZE = 65536  # 64KB chunks for large files29 30def compute_file_hash(path: Path) -> str:31    """SHA-256 of file contents (chunked for large files)."""32    if not path.is_file():33        raise FileNotFoundError(f"File not found: {path}")34    sha256 = hashlib.sha256()35    with open(path, "rb") as f:36        while True:37            chunk = f.read(_HASH_CHUNK_SIZE)38            if not chunk:39                break40            sha256.update(chunk)41    return sha256.hexdigest()42```43 44**Why content hash?** File rename/move = cache hit. Content change = automatic invalidation. No index file needed.45 46### 2. Frozen Dataclass for Cache Entry47 48```python49from dataclasses import dataclass50 51@dataclass(frozen=True, slots=True)52class CacheEntry:53    file_hash: str54    source_path: str55    document: ExtractedDocument  # The cached result56```57 58### 3. File-Based Cache Storage59 60Each cache entry is stored as `{hash}.json` — O(1) lookup by hash, no index file required.61 62```python63import json64from typing import Any65 66def write_cache(cache_dir: Path, entry: CacheEntry) -> None:67    cache_dir.mkdir(parents=True, exist_ok=True)68    cache_file = cache_dir / f"{entry.file_hash}.json"69    data = serialize_entry(entry)70    cache_file.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8")71 72def read_cache(cache_dir: Path, file_hash: str) -> CacheEntry | None:73    cache_file = cache_dir / f"{file_hash}.json"74    if not cache_file.is_file():75        return None76    try:77        raw = cache_file.read_text(encoding="utf-8")78        data = json.loads(raw)79        return deserialize_entry(data)80    except (json.JSONDecodeError, ValueError, KeyError):81        return None  # Treat corruption as cache miss82```83 84### 4. Service Layer Wrapper (SRP)85 86Keep the processing function pure. Add caching as a separate service layer.87 88```python89def extract_with_cache(90    file_path: Path,91    *,92    cache_enabled: bool = True,93    cache_dir: Path = Path(".cache"),94) -> ExtractedDocument:95    """Service layer: cache check -> extraction -> cache write."""96    if not cache_enabled:97        return extract_text(file_path)  # Pure function, no cache knowledge98 99    file_hash = compute_file_hash(file_path)100 101    # Check cache102    cached = read_cache(cache_dir, file_hash)103    if cached is not None:104        logger.info("Cache hit: %s (hash=%s)", file_path.name, file_hash[:12])105        return cached.document106 107    # Cache miss -> extract -> store108    logger.info("Cache miss: %s (hash=%s)", file_path.name, file_hash[:12])109    doc = extract_text(file_path)110    entry = CacheEntry(file_hash=file_hash, source_path=str(file_path), document=doc)111    write_cache(cache_dir, entry)112    return doc113```114 115## Key Design Decisions116 117| Decision | Rationale |118|----------|-----------|119| SHA-256 content hash | Path-independent, auto-invalidates on content change |120| `{hash}.json` file naming | O(1) lookup, no index file needed |121| Service layer wrapper | SRP: extraction stays pure, cache is a separate concern |122| Manual JSON serialization | Full control over frozen dataclass serialization |123| Corruption returns `None` | Graceful degradation, re-processes on next run |124| `cache_dir.mkdir(parents=True)` | Lazy directory creation on first write |125 126## Best Practices127 128- **Hash content, not paths** — paths change, content identity doesn't129- **Chunk large files** when hashing — avoid loading entire files into memory130- **Keep processing functions pure** — they should know nothing about caching131- **Log cache hit/miss** with truncated hashes for debugging132- **Handle corruption gracefully** — treat invalid cache entries as misses, never crash133 134## Anti-Patterns to Avoid135 136```python137# BAD: Path-based caching (breaks on file move/rename)138cache = {"/path/to/file.pdf": result}139 140# BAD: Adding cache logic inside the processing function (SRP violation)141def extract_text(path, *, cache_enabled=False, cache_dir=None):142    if cache_enabled:  # Now this function has two responsibilities143        ...144 145# BAD: Using dataclasses.asdict() with nested frozen dataclasses146# (can cause issues with complex nested types)147data = dataclasses.asdict(entry)  # Use manual serialization instead148```149 150## When to Use151 152- File processing pipelines (PDF parsing, OCR, text extraction, image analysis)153- CLI tools that benefit from `--cache/--no-cache` options154- Batch processing where the same files appear across runs155- Adding caching to existing pure functions without modifying them156 157## When NOT to Use158 159- Data that must always be fresh (real-time feeds)160- Cache entries that would be extremely large (consider streaming instead)161- Results that depend on parameters beyond file content (e.g., different extraction configs)