Chroma Hybrid Search Skill

This skill provides local RAG retrieval by combining vector search (semantic) with BM25 keyword search (exact match), followed by Cross-Encoder re-ranking using the BGE-Reranker model. It searches both the hot store (knowledge-base/.md, experience/.md) and the cold store (cold-notes/raw.jsonl) — the document set it reads must match what update_db.py indexed, or vectorized cold-store entries would be dropped from results.

Cross-platform command convention — in every command below, <PY> is the virtual-env Python:

- Windows (PowerShell): & "$HOME\.deep-memory\.venv\Scripts\python"
- Linux / macOS: ~/.deep-memory/.venv/bin/python

The venv lives at a fixed absolute path inside the global workspace (~/.deep-memory/.venv), next to the data — commands work from any project directory; no project-local .venv is assumed. On PowerShell keep the & call operator and $HOME (PowerShell does not expand ~ in arguments to native commands); in Git Bash on Windows the same venv is ~/.deep-memory/.venv/Scripts/python.

> All skills/... paths assume the skill pack lives inside your current project (project-local install).

Workspace Storage Path Resolution:
By default, deep-memory uses the user's global directory ~/.deep-memory (which resolves to C:\Users\<username>\.deep-memory on Windows) to store all knowledge bases, cold notes, and database files. This unifies memories across all your project workspaces.
- If you want to use a specific directory, set the DEEPMEMORYWORKSPACE environment variable (e.g., DEEPMEMORYWORKSPACE="." or DEEPMEMORYWORKSPACE="D:\my-memories").
- You can also pass --workspace <path> to any script to override the workspace path for that specific command.

⚙️ First-Time Initialization (Bootstrap)

When this skill is first invoked by the Agent or user, confirm that the .venv virtual environment and the chromahybriddb vector database have been created and all required packages are installed. If not, execute the following steps:

# 1. Create virtual environment (if it doesn't exist) — fixed location in the global workspace, shared by all projects
#    Windows (PowerShell): python -m venv "$HOME\.deep-memory\.venv"
#    Linux / macOS:        python3 -m venv ~/.deep-memory/.venv

# 2. Install required packages (no activation needed — call the venv Python directly)
<PY> -m pip install -r skills/chroma-hybrid-search/requirements.txt

# 3. Initialize and build the local vector index database
<PY> skills/chroma-hybrid-search/scripts/update_db.py

Notes:
1. Do NOT commit .venv or chromahybriddb/ to GitHub — both live under ~/.deep-memory (outside any repo) and must be generated locally using the steps above.
2. Step 3 is only needed once, to create the index. After that search.py tops it up automatically before every query — see Index Maintenance below.

🔄 Index Maintenance

search.py tops up the index before every query, so update_db.py is no longer part of the daily loop. An indexing failure prints a warning and never blocks the search; --no-auto-index disables the behaviour.

Run updatedb.py explicitly only to index new entries immediately rather than at the next search — e.g. right after a refinement pass. updatedb.py --rebuild forces every entry to be re-embedded (~6 minutes for a 1,265-entry store); it is only for making vectors correspond exactly to the latest text format, never required for normal operation.

Content-addressed IDs

Cold-store entries are keyed by a semantic hash — cold-notes#<hash12> — not by line number. The hash covers topic / content / tags / skill / memory_type and deliberately excludes date / time / project / quality and every template string. So:

• Deleting or inserting a line anywhere in raw.jsonl does not re-embed the entries after it. Measured on a 928-line store: removing the middle line re-embedded 0 entries — the 463 entries after it only had their line metadata refreshed, at zero embedding cost.
• Marking an entry quality: reviewed during refinement does not trigger a re-embed.
• Changing how the indexed text is assembled does not invalidate the store.

The line number survives in metadata as line and is returned with every cold-store hit, so results stay traceable to their source line.

Run log

Each indexing run appends one JSON line to ~/.deep-memory/logs/index-runs.jsonl with per-stage timings and why entries were re-embedded:

A run that re-embeds more than 20% of the store prints a warning with the reason breakdown. When the reason is id-missing, the usual cause is that the repo copy and the ~/.claude/skills installed copy of kb_reader.py are on different versions — they assemble the indexed text differently, so each sees the other's entries as changed.

🔍 Usage

The Agent can execute scripts/search.py directly from the terminal to perform precise retrieval over the knowledge base and experience store.

1. Hybrid Search + Semantic Re-ranking (Default — Recommended)

Best for complex questions requiring deep semantic understanding:

<PY> skills/chroma-hybrid-search/scripts/search.py --query "spring animation tuning guidelines" --min-score 0.35

Always pass --min-score 0.35 in this mode — besides filtering noise, it lets the project-first pass fall back to the full cross-project store when the current project only has low-relevance hits (without it, a near-zero-score project hit still counts as "found something" and blocks the fallback).

2. Vector Similarity Search Only

Best for cross-language or fuzzy semantic search:

<PY> skills/chroma-hybrid-search/scripts/search.py --query "Spring Boot 404" --mode vector

3. BM25 Keyword Search Only

Best for finding exact proper nouns, variable names, or code snippets:

<PY> skills/chroma-hybrid-search/scripts/search.py --query "pxmin pxmax" --mode bm25

4. Scoped to One Skill, Tag, or Memory Type

Cold-store entries carry skill, tags, and memory_type metadata; experience/skill-[skill-id].md files carry skill derived from their filename, and hot-store entries are tagged as knowledge or experience by source folder. Use --skill for an exact skill-id match, --tag to check the tags array (Chroma's native $contains, requires chromadb ≥1.5.0 — already pinned in requirements.txt), and --memory-type to separate general knowledge from skill/tool experience. These filters narrow the BM25 corpus and the vector where clause identically, so every retrieval method sees the same filtered candidate set:

<PY> skills/chroma-hybrid-search/scripts/search.py --query "session timeout" --skill backend-dev
<PY> skills/chroma-hybrid-search/scripts/search.py --query "config drift"     --tag redis
<PY> skills/chroma-hybrid-search/scripts/search.py --query "agent browser fill fallback" --memory-type experience

Plain knowledge-base/*.md category files have no single skill/tag (they mix many topics), so they're never included by --skill or --tag filters unless the indexed entry explicitly has that metadata; they do participate in --memory-type knowledge.

📋 Output Format

The script outputs a standard JSON array. path for hot-store hits is file.md#entry-slug — one ## 🔧 entry, not the whole file (see "Entry-Level Chunking" below). Use the text field directly as context — do not separately open the file at path with the Read tool. Stripping the #entry-slug and reading the whole source file re-introduces exactly the whole-file dilution chunking exists to avoid; path is for citation (telling the user where a fact came from), not an instruction to read further.

[
  {
    "path": "experience/skill-remotion-best-practices.md#fps-30-causes-av-desync",
    "rerank_score": 0.9402,
    "text": "## 🔧 FPS 30 causes A/V desync\n**Date:** ...\n(just this one entry, not the rest of the file)"
  }
]

✂️ Entry-Level Chunking

knowledge-base/.md and experience/.md are indexed per ## 🔧 entry, not per file — kbreader.py's splitentries() splits on that heading and gives each entry a stable slug (from its title, de-duplicated within the file) so results point to the specific matching entry. A file with no ## 🔧 headings at all falls back to whole-file indexing for backward compatibility. updatedb.py and search.py both import this splitting logic from the same kbreader.py — if you ever need to change how entries are parsed, change it there once, not in both scripts.

🛡️ Anti-Hallucination RAG Routing Guide

• Filename-first matching: If the user's question explicitly mentions a filename, read that file's full content directly via Python glob or fast text search first.
• Tiered RAG: Prioritize filename matching; fall back to search.py hybrid search + rerank only if no match is found.
• Score threshold filtering: Do not pass any chunks or files with a Rerank Score below 0.35 to the model — this prevents noise from degrading response quality. Apply it search-side with --min-score 0.35 rather than filtering the output afterwards: the search-side filter is also what allows the project-first pass to fall back to the cross-project store when project hits are all low-relevance.