# Auto-Categorization & Auto-Save Plan (Roadmap 411final)

**Goal:** Enhance `MemoryService` with local LLM-driven auto-categorization — every document is automatically classified into a hierarchical topic/tag structure stored as plain Markdown, searchable via hybrid BM25+semantic retrieval, with in-memory dedup and incremental watch-mode processing.

**Current Gap:** Documents go into LanceDB vectors but never get classified into hierarchical topics/tags. Metadata dict gets a `"topic"` field but it's always `None`.

---

## Design Decisions

### 1. Storage Format — Plain Markdown + hierarchy.md

**NOT** LanceDB vectors alone. Following ByteRover, NeuRAG, Memex, RAGraph research:

```
knowledge_docs/
├── _index.md                  # root overview (generated)
├── ai/                        # type category
│   ├── security/              # subtopic by domain
│   │   ├── audit_chain.md
│   │   └── policy.md
│   └── architecture/
│       └── design.md
├── research/                  # type category
├── notes/                     # free-form notes
└── reports/                   # formal documents
```

Each Markdown file has frontmatter with: `topic`, `subtopic`, `tags[]`, `summary`, `confidence` (0-1), `entities[]`, `intent` (`actionable`|`reference`|`idea`), `created`, `updated`.

**Why:** Zero external DB dependency (per ByteRover paper), version-controllable (git), portable, MCP-compatible. SQLite stays for document catalog/index only.

### 2. Classification Pipeline — LLM Classify Step

Before storing any document, run LLM via Ollama to extract structured metadata:

```python
def classify_document(text: str) -> ClassificationResult:
    """Run local LLM to classify document into hierarchy."""
    payload = {
        "model": config.model,
        "prompt": CLASSIFY_PROMPT,  # strict JSON schema
        "options": {"temperature": 0, "num_ctx": 8192},
    }
    response = ollama_client._client.post("/api/chat", json=payload)
    classifications = parse_json(response.json())
    return ClassificationResult(
        topic=classifications["topic"],
        subtopic=classifications.get("subtopic", "general"),
        tags=classifications["tags"],
        summary=classifications["summary"],
        entities=classifications["entities"],
        confidence=float(classifications["confidence"]),
        intent=classifications["intent"],
    )
```

Prompt enforces JSON output with strict schema (temperature=0, per Memex research). Output fields are all mandatory.

### 3. Directory Layout — 4-Level Hierarchy

```
knowledge_docs/
├── _index.md                  # root overview tree
├── ai/                        # type: AI/agent decisions
│   ├── security/              # subtopic: security domain
│   │   └── audit_protocols.md
│   └── architecture/          # subtopic: architecture
│       └── knowledge_plane.md
├── research/                  # type: external research
├── notes/                     # free-form notes (uncategorized)
└── reports/                   # type: formal reports
```

Four levels: Root > Type > Domain(Subtopic) > Document. Topics map to folder paths. `hierarchy.md` maintains the tree as plain text the agent can navigate (per secondmem and RAGraph research).

### 4. Watch Mode — Incremental Processing

Following Grimore-MD and RAGraph research: SHA-256 hash gate. Only process changed/new files. Each file processed independently — a broken PDF never blocks the queue.

```python
KNOWN_HASHES = set()

def should_process(filepath: Path) -> bool:
    h = sha256(filepath.read_bytes())
    if h in KNOWN_HASHES:
        return False  # unchanged
    KNOWN_HASHES.add(h)
    return True
```

### 5. Deduplication Check Before Write

```python
def check_duplicate(tags: list[str]) -> bool:
    """Check if document is duplicate via tag overlap."""
    for doc in SQLiteKnowledgeRepository.get_all_tags():
        if len(set(tags) & set(doc.tags)) >= 3:  # 3+ shared tags = duplicate
            return True
    return False
```

### 6. Search Hybrid — BM25 + Semantic + RRF

Following RAGraph research: fuse three retrieval methods:

1. **BM25** — SQLite FTS5 full-text index
2. **Cosine similarity** — vector search in LanceDB
3. **Reciprocal Rank Fusion (RRF)** — merge and re-rank results

```python
def hybrid_search(query: str, top_k: int = 10) -> list[SearchResult]:
    bm25 = fts5_search(query, top_k=top_k*2)
    vec = embedding_service.embed_text(query)
    semantic = vector_store.search(vec, limit=top_k*2)
    fused = rrf_fuse(bm25, semantic, k=64)
    return sorted(fused, key=lambda r: r.rrf_score)[:top_k]
```

### 7. Integration into MemoryService

Modify `MemoryService.register_file()` to call classification pipeline before registration:

1. If `auto_classify=True` (config option), call `classifier.classify_document()`
2. Write classification frontmatter to `~/knowledge_docs/{topic}/{subtopic}.md`
3. Update `hierarchy.md` tree
4. Store in SQLite + LanceDB as before (idempotent — skip if already registered)
5. CLI: `cyberbrain knowledge ingest /path --recursive`
6. CLI: `cyberbrain knowledge search "dotaz"`

### 8. Search Enhancement — FTS5 + Cross-Encoder

1. Add FTS5 full-text index to SQLite migration
2. Hybrid search: BM25 + cosine via RRF fusion
3. Cross-encoder reranking optional for precision
4. Source citation in results (`[[Title#p.42]]`)
5. Fallback to lexical search when semantic unavailable

---

## Implementation Phases

### Phase 1: Classification Service (Week 1-2)
1. New module `src/cyberbrain26/knowledge_plane/classifier.py` — `LLMClassifier` class
2. `CLASSIFY_PROMPT` — strict JSON schema, temperature=0, confidence field
3. `classify_document()` returns `ClassificationResult` dataclass
4. Config option `auto_classify: bool = True` in `MemoryService.__init__`
5. Gate: `pytest -q`, `ruff check`, `mypy src/cyberbrain26/knowledge_plane` green

### Phase 2: Markdown Store Layer (Week 2-3)
1. New module `src/cyberbrain26/knowledge_plane/markdown_store.py` — `MarkdownStore` class
2. Write classification result to `~/knowledge_docs/{topic}/{subtopic}.md`
3. Maintain `hierarchy.md` tree of all topics/tags
4. `list_documents(topic)` → glob walk over filesystem (no DB needed)
5. De-duplicate via SHA-256 + tag overlap
6. Archive old documents (keep in SQLite as quick lookup)

### Phase 3: Integration into MemoryService (Week 3-4)
1. Modify `MemoryService.register_file()` to call `classifier.classify_document()` before registration
2. If already registered, skip reprocessing (idempotent)
3. Store classification in `document.metadata` AND write markdown file
4. Add CLI command: `cyberbrain knowledge ingest /path --recursive`
5. Add CLI command: `cyberbrain knowledge search "dotaz"`
6. Update `docs/CAPABILITIES.md` with `WORKING` status

### Phase 4: Search Enhancement (Week 4-5)
1. Add FTS5 full-text index to SQLite (migration SQL)
2. Hybrid search: BM25 + cosine via RRF fusion
3. Cross-encoder reranking optional
4. Source citation in results
5. Fallback to lexical search when semantic unavailable

### Phase 5: Self-Organization / Rebalance (Week 5-6)
1. Background consolidator: merge fragmented topics, split oversized folders
2. Dead link detection and cleanup
3. Merge candidates based on ≥70% tag overlap (Neurag research)
4. Hierarchy rebuild script `rebalance_knowledge.sh`

---

## Validation Gates

| Gate | Command | Requirement |
|------|---------|------------|
| Gate 1 | `pytest -q` | All knowledge plane tests pass (≥747) |
| Gate 2 | `pytest tests/ -q --ignore=test_knowledge_integration.py` | No regressions; integration tests skipped when Ollama absent |
| Gate 3 | `mypy src/cyberbrain26/knowledge_plane` | Type clean — zero errors |

### Gate 1 Success Criteria
- `pytest -q` passes in knowledge_plane directory
- ruff check green
- mypy green (zero new errors)

### Gate 2 Success Criteria
- Full suite passes (except integration tests that require Ollama)
- `--ignore=tests/test_knowledge_integration.py` skips embedding-dependent tests

### Gate 3 Success Criteria
- mypy reports zero errors in `src/cyberbrain26/knowledge_plane/`
- All new modules have type annotations

---

## Open Questions (Decision Points)

1. **Which model for classification?** Currently only `qwen3-embedding:0.6b` is configured. Recommendation: Add chat endpoint support to `EmbeddingService` — use same model for classification (it's an embedding model but supports `/api/chat` too), or add `classification_model` config field. **Decision:** Extend `EmbeddingService` with `chat()` method using same Ollama endpoint; default to `qwen3-embedding:0.6b` for both embed and classify.

2. **Confidence threshold for auto-approval?** Recommend 0.7+. Below that, stage through review workflow. **Decision:** `AUTO_CLASSIFY_THRESHOLD = 0.7` in config; below threshold, document goes to `notes/` with `confidence: null` and requires manual tagging.

3. **What happens when LLM classification fails?** Fail closed — log warning, keep document unclassified in `notes/` directory with `confidence: null`. **Decision:** Never silently skip classification; always produce some output (even if `confidence: 0`).

4. **How to handle contradictions across documents?** Memex has conflict detection; recommend flagging cross-references for later resolution. **Decision:** Track `source_conflicts` in SQLite metadata; auto-merge when tag overlap ≥70%, else surface for review.

5. **Should hierarchy.md be auto-generated or hand-maintained?** Research suggests auto-generated with human review option. **Decision:** Auto-generate from filesystem tree; `hierarchy.md` is always in sync with actual files.

---

## References (Deep Web Research)

- **ByteRover** (arXiv:2604.01599) — agent-native memory, hierarchical Context Tree, all storage as human-readable markdown, zero external infrastructure
- **Grimore-MD** (kahz12/Grimore-MD) — hybrid BM25+semantic, two-tier change detection (SHA-256 gates expensive extraction), RRF fusion
- **secondmem** (Rinil-Parmar/secondmem) — AI classification into topic markdown files, confidence-scored extraction, 7-step rebalance
- **RAGraph** (ADVASYS/ragraph) — graph-based organization, automatic ingestion, entity/topic resolution, hybrid BM25+vector retrieval
- **NeuRAG** (recla93/neurag) — trigger-navigable hierarchy, AST-aware chunking, entity merging, two-stage retrieval
- **Memex** (jagdeepvirdi/memex) — Ollama auto-classify, fuzzy canonical taxonomy, confidence scores, category review staging queue
- **ByteRover paper key insight:** "Memory operations are tools in the agent's toolkit, not API calls to an external service" — enables stateful feedback loop, real-time error recovery

---

## Migration from Current State

**Current:** Documents → SQLite metadata + LanceDB vectors only; no topics/tags in filesystem.

**Migration Script** (`migrate_knowledge.py`):
1. Read all existing SQLite `knowledge_documents` rows
2. For each, extract existing `metadata.json` → map to `topic`, `tags` if present
3. Create folder structure `~/knowledge_docs/{topic}/` 
4. Move existing markdown content (if any) into appropriate subfolder
5. Update `hierarchy.md` with existing topics
6. Flag documents with no classification → `notes/` with `confidence: null`
7. Keep SQLite + LanceDB as-is for backward compatibility

Migration is idempotent — running it twice produces same state.

---

## CLI Commands (to be added)

| Command | Description | Status |
|---------|-------------|--------|
| `cyberbrain knowledge ingest /path --recursive` | Ingest directory with auto-classification | Phase 3 |
| `cyberbrain knowledge search "dotaz"` | Hybrid BM25+semantic search with citations | Phase 4 |
| `cyberbrain knowledge rebalance` | Run hierarchy consolidation/dead link cleanup | Phase 5 |
| `cyberbrain knowledge list` | List all topics/subdocuments | Phase 5 |

---

## Plan File

Saved at: `.kilo/plans/1793777392110-auto-categorize-plan.md`

---

## How to Run Validation

```bash
# Phase 1 validation
python -m pytest tests/test_independent_memory.py -q  # should pass

# Phase 2-3 after integration
python -m pytest tests/ -q --ignore=tests/test_knowledge_integration.py

# Type check
mypy src/cyberbrain26/knowledge_plane -q

# Run gate 1
pytest -q && ruff check && mypy src/cyberbrain26/knowledge_plane
```