Memory Lancedb Pro

1---2name: memory-lancedb-pro3description: "Comprehensive guide for maintaining, debugging, and upgrading the memory-lancedb-pro OpenClaw plugin — an enhanced LanceDB-backed long-term memory system with hybrid retrieval (Vector + BM25), cross-encoder reranking, multi-scope isolation, noise filtering, adaptive retrieval, and a management CLI. Use this skill when: (1) developing new features or fixing bugs in memory-lancedb-pro, (2) modifying the retrieval pipeline (vector search, BM25, RRF fusion, reranking, scoring stages), (3) adding or changing embedding providers, (4) updating scope/access control logic, (5) modifying agent tools or CLI commands, (6) troubleshooting memory quality issues (noise, duplicates, low recall), (7) working on the JSONL session distillation pipeline, (8) migrating data between memory backends, or (9) understanding the plugin's architecture to plan enhancements."4---5 6# memory-lancedb-pro Plugin Maintenance Guide7 8## Overview9 10**memory-lancedb-pro** is an enhanced long-term memory plugin for [OpenClaw](https://github.com/openclaw/openclaw). It replaces the built-in `memory-lancedb` plugin with advanced retrieval capabilities, multi-scope memory isolation, and a management CLI.11 12**Repository**: https://github.com/win4r/memory-lancedb-pro13**License**: MIT | **Language**: TypeScript (ESM) | **Runtime**: Node.js via OpenClaw Gateway14 15## Architecture16 17```18┌─────────────────────────────────────────────────────────┐19│                   index.ts (Entry Point)                │20│  Plugin Registration · Config Parsing · Lifecycle Hooks │21└────────┬──────────┬──────────┬──────────┬───────────────┘22         │          │          │          │23    ┌────▼───┐ ┌────▼───┐ ┌───▼────┐ ┌──▼──────────┐24    │ store  │ │embedder│ │retriever│ │   scopes    │25    │ .ts    │ │ .ts    │ │ .ts    │ │    .ts      │26    └────────┘ └────────┘ └────────┘ └─────────────┘27         │                     │28    ┌────▼───┐           ┌─────▼──────────┐29    │migrate │           │noise-filter.ts │30    │ .ts    │           │adaptive-       │31    └────────┘           │retrieval.ts    │32                         └────────────────┘33    ┌─────────────┐   ┌──────────┐34    │  tools.ts   │   │  cli.ts  │35    │ (Agent API) │   │ (CLI)    │36    └─────────────┘   └──────────┘37```38 39## File Reference (Quick Navigation)40 41| File | Purpose | Key Exports |42|------|---------|-------------|43| `index.ts` | Plugin entry point. Registers with OpenClaw Plugin API, parses config, mounts lifecycle hooks | `memoryLanceDBProPlugin` (default), `shouldCapture`, `detectCategory` |44| `openclaw.plugin.json` | Plugin metadata + full JSON Schema config with `uiHints` | — |45| `package.json` | NPM package. Deps: `@lancedb/lancedb`, `openai`, `@sinclair/typebox` | — |46| `cli.ts` | CLI: `memory-pro list/search/stats/delete/delete-bulk/export/import/reembed/migrate` | `createMemoryCLI`, `registerMemoryCLI` |47| `src/store.ts` | LanceDB storage layer. Table creation, FTS indexing, CRUD, vector/BM25 search | `MemoryStore`, `MemoryEntry`, `loadLanceDB` |48| `src/embedder.ts` | Embedding abstraction. OpenAI-compatible API, task-aware, LRU cache | `Embedder`, `createEmbedder`, `getVectorDimensions` |49| `src/retriever.ts` | Hybrid retrieval engine. Full scoring pipeline | `MemoryRetriever`, `createRetriever`, `DEFAULT_RETRIEVAL_CONFIG` |50| `src/scopes.ts` | Multi-scope access control | `MemoryScopeManager`, `createScopeManager` |51| `src/tools.ts` | Agent tool definitions: `memory_recall/store/forget/update/stats/list` | `registerAllMemoryTools` |52| `src/noise-filter.ts` | Noise filter for low-quality content | `isNoise`, `filterNoise` |53| `src/adaptive-retrieval.ts` | Skip retrieval for greetings, commands, emoji | `shouldSkipRetrieval` |54| `src/migrate.ts` | Migration from legacy `memory-lancedb` | `MemoryMigrator`, `createMigrator` |55| `scripts/jsonl_distill.py` | JSONL session distillation script (Python) | — |56 57## Core Subsystem Reference58 59For detailed deep-dives into each subsystem, read the appropriate reference file:60 61- **Retrieval Pipeline** (scoring math, RRF fusion, reranking, all scoring stages): See [references/retrieval_pipeline.md](references/retrieval_pipeline.md)62- **Storage & Data Model** (LanceDB schema, FTS indexing, CRUD, vector dim): See [references/storage_and_schema.md](references/storage_and_schema.md)63- **Embedding System** (providers, task-aware API, caching, dimensions): See [references/embedding_system.md](references/embedding_system.md)64- **Plugin Lifecycle & Config** (hooks, registration, config parsing): See [references/plugin_lifecycle.md](references/plugin_lifecycle.md)65- **Scope System** (multi-scope isolation, agent access, patterns): See [references/scope_system.md](references/scope_system.md)66- **Tools & CLI** (agent tools, CLI commands, parameters): See [references/tools_and_cli.md](references/tools_and_cli.md)67- **Common Gotchas & Troubleshooting**: See [references/troubleshooting.md](references/troubleshooting.md)68 69## Development Workflows70 71### Adding a New Embedding Provider72 731. Check if it's OpenAI-compatible (most are). If so, no code change needed — just config742. If the model is not in `EMBEDDING_DIMENSIONS` map in `src/embedder.ts`, add it753. If the provider needs special request fields beyond `task` and `normalized`, extend `buildPayload()` in `src/embedder.ts`764. Test with `embedder.test()` method775. Document the provider in README.md table78 79### Adding a New Rerank Provider80 811. Add provider name to `RerankProvider` type in `src/retriever.ts`822. Add case in `buildRerankRequest()` for request format (headers + body)833. Add case in `parseRerankResponse()` for response parsing844. Add to `rerankProvider` enum in `openclaw.plugin.json`855. Test with actual API calls — reranker has 5s timeout protection86 87### Adding a New Scoring Stage88 891. Create a `private apply<StageName>(results: RetrievalResult[]): RetrievalResult[]` method in `MemoryRetriever`902. Add corresponding config fields to `RetrievalConfig` interface913. Insert the stage in the pipeline sequence in both `hybridRetrieval()` and `vectorOnlyRetrieval()`924. Add defaults to `DEFAULT_RETRIEVAL_CONFIG`935. Add JSON Schema fields to `openclaw.plugin.json`946. Pipeline order: Fusion → Rerank → Recency → Importance → LengthNorm → TimeDecay → HardMin → Noise → MMR95 96### Adding a New Agent Tool97 981. Create `registerMemory<ToolName>Tool()` in `src/tools.ts`992. Define parameters with `Type.Object()` from `@sinclair/typebox`1003. Use `stringEnum()` from `openclaw/plugin-sdk` for enum params1014. Always validate scope access via `context.scopeManager`1025. Register in `registerAllMemoryTools()` — decide if core (always) or management (optional)1036. Return `{ content: [{ type: "text", text }], details: {...} }`104 105### Adding a New CLI Command106 1071. Add command in `registerMemoryCLI()` in `cli.ts`1082. Pattern: `memory.command("name <args>").description("...").option("--flag", "...").action(async (args, opts) => { ... })`1093. Support `--json` flag for machine-readable output1104. Use `process.exit(1)` for error cases1115. CLI is registered via `api.registerCli()` in `index.ts`112 113### Modifying Auto-Capture Logic114 1151. `shouldCapture(text)` in `index.ts` controls what gets auto-captured1162. `MEMORY_TRIGGERS` regex array defines trigger patterns (supports EN/CJK)1173. `detectCategory(text)` classifies captures as preference/fact/decision/entity/other1184. Auto-capture runs in `agent_end` hook, limited to 3 per turn1195. Duplicate detection threshold: cosine similarity > 0.95120 121### Modifying Auto-Recall Logic122 1231. Auto-recall uses `before_agent_start` hook (OFF by default)1242. `shouldSkipRetrieval()` from `src/adaptive-retrieval.ts` gates retrieval1253. Injected as `<relevant-memories>` XML block with UNTRUSTED DATA warning1264. `sanitizeForContext()` strips HTML, newlines, limits to 300 chars per memory1275. Max 3 memories injected per turn128 129## Key Design Decisions130 131- **autoRecall defaults to OFF** — prevents model from echoing injected memory context132- **autoCapture defaults to ON** — transparent memory accumulation133- **sessionMemory defaults to OFF** — raw session summaries degrade retrieval quality; use JSONL distillation instead134- **LanceDB dynamic import** — loaded asynchronously to avoid blocking; cached in singleton promise135- **Startup checks are fire-and-forget** — gateway binds HTTP port immediately; embedding/retrieval tests run in background with 8s timeout136- **Daily JSONL backup** — 24h interval, keeps last 7 files, runs 1 min after start137- **BM25 score normalization** — raw BM25 scores are unbounded, normalized with sigmoid: `1 / (1 + exp(-score/5))`138- **Update = delete + re-add** — LanceDB doesn't support in-place updates139- **ID prefix matching** — 8+ hex char prefix resolves to full UUID for user convenience140- **CJK-aware thresholds** — shorter minimum lengths for Chinese/Japanese/Korean text (4–6 chars vs 10–15 for English)141- **Env var resolution** — `${VAR}` syntax resolved at config parse time; gateway service may not inherit shell env142 143## Testing144 145- Smoke test: `node test/cli-smoke.mjs`146- Manual verification: `openclaw plugins doctor`, `openclaw memory-pro stats`147- Embedding test: `embedder.test()` returns `{ success, dimensions, error? }`148- Retrieval test: `retriever.test()` returns `{ success, mode, hasFtsSupport, error? }`