04 · RAG & Memory System

Fourth installment of the architecture deep-dive: memory layering (scope × category), RAG scoring formula, dialogue trees, MemoryWriter / MemoryReader / Consolidator, and DeciderService eviction strategy. This is the "long-term memory" core of Familiars. Corresponds to ARCHITECTURE.md §5.

§5.1 Why Not Use a Vector Database

Approach	Reason for rejection
Pinecone / Weaviate / Qdrant	Data scale (single user, a few thousand to tens of thousands of embeddings) is well served by in-memory + SQLite; introducing extra services and ops overhead is unjustified
Plain files + no index	Retrieval too slow; cannot do dense recall
Familiars approach	In-memory + SQLite + file-tree hybrid; vectors stored under `workspace/` and `familiars.db`, privacy stays on-device; abstraction at the `memory_store` layer — swapping in an external vector store later is straightforward

§5.2 Memory Scope × Category Matrix

5.2.1 Scope

Defined in src/utils/memory_store.py:SCOPES:

python

SCOPES: tuple[str, ...] = ("user", "identity", "dialogue", "ephemeral")

Scope	Persistence path	Meaning	TTL semantics
`user`	`workspace/user/<category>.md`	User profile (shared across agents)	Long-term, cross-session
`identity`	`workspace/agents/<agent>/identity/<category>.md`	Agent self-awareness	Long-term, follows the agent
`dialogue`	`workspace/agents/<agent>/<dialogue>/dialogue/<category>.md`	Dialogue context	Follows the dialogue
`ephemeral`	`workspace/agents/<agent>/<dialogue>/ephemeral/<category>.md`	Per-turn temporary	Cleared at turn end

5.2.2 Category

Defined in src/utils/memory_store.py:CATEGORIES:

python

CATEGORIES: tuple[str, ...] = (
    "profile",      # User / Agent profile
    "preference",   # Preferences
    "relation",     # Relationships
    "knowledge",    # Knowledge
    "event",        # Events
    "state",        # State
    "procedure",    # Procedures
    "directive",    # Directives
)

5.2.3 Valid (Scope, Category) Combinations

Not a full Cartesian product — only the following combinations are valid (VALID_CELLS):

python

VALID_CELLS: frozenset[tuple[str, str]] = frozenset({
    ("user", "profile"),       # User profile
    ("user", "preference"),    # User preferences
    ("user", "relation"),      # User relationships
    ("user", "knowledge"),     # User knowledge
    ("user", "event"),         # User events
    
    ("identity", "preference"),    # Agent preferences
    ("identity", "procedure"),     # Agent behavioral procedures
    ("identity", "directive"),     # Agent directives
    
    ("dialogue", "preference"),    # In-dialogue preferences
    ("dialogue", "relation"),      # In-dialogue relationships
    ("dialogue", "knowledge"),     # In-dialogue knowledge
    ("dialogue", "event"),         # In-dialogue events
    ("dialogue", "state"),         # In-dialogue state
    ("dialogue", "directive"),     # In-dialogue directives
    
    ("ephemeral", "state"),        # Ephemeral state
    ("ephemeral", "directive"),    # Ephemeral directives
})

5.2.4 Path Resolution

python

def resolve_path(*, workspace, scope, category, agent_id="agent_0",
                 dialogue_id=None) -> Path:
    fname = f"{category}.md"
    if scope == "user":
        return ws / "user" / fname
    if scope == "identity":
        return ws / "agents" / agent_id / "identity" / fname
    if scope == "dialogue":
        return ws / "agents" / agent_id / dialogue_id / "dialogue" / fname
    if scope == "ephemeral":
        return ws / "agents" / agent_id / dialogue_id / "ephemeral" / fname

§5.3 Memory Entry Format

5.3.1 On-Disk Format

Each memory entry is a logical record formatted as an HTML comment wrapping JSON metadata plus markdown body:

<!-- {"id":"m_a1b2c3d4","scope":"user","category":"profile","ts":"2026-08-29T12:00:00+00:00","tags":["name"],"salience":0.8,"ttl":null,"supersedes":null,"deprecated":false,"schema_version":2} -->
The user's name is Alice and they like cats.

5.3.2 MemoryRecord Data Structure

python

@dataclass
class MemoryRecord:
    id: str                                    # Stable str; applier generates "m_" + token_hex(4)
    scope: str                                 # "user" | "identity" | "dialogue" | "ephemeral"
    category: str = "knowledge"                # One of 8 categories
    ts: str = ""                               # ISO 8601 timestamp
    content: str = ""                          # Markdown body
    tags: list[str] = []                       # Tags
    salience: float = 0.5                      # Salience [0, 1]
    ttl: Optional[dict] = None                 # TTL policy
    supersedes: Optional[str] = None           # ID of the entry being superseded
    deprecated: bool = False                   # Deprecated → filtered by reader
    schema_version: int = 2                    # Format version
    evidence: Optional[dict] = None            # Provenance {"dialogue_id", "turn_id"}
    epistemic_status: str = "asserted"         # "asserted" | "inferred" | "speculative"
    last_accessed_ts: Optional[str] = None     # Last access time (reader batch-flushes)
    access_count: int = 0                      # Access frequency signal
    pinned: bool = False                       # Pinned → never auto-consolidated
    derived_from: list[str] = []               # Source IDs (consolidator output)
    source_quote: list[str] = []               # Writer source quotes

5.3.3 TTL Policies

python

def is_expired(rec, *, now=None, turn_counter=None,
               session_active=True, dialogue_active=True) -> bool:
    if rec.ttl is None:
        return False
    kind = rec.ttl.get("kind")
    if kind == "absolute":
        # {"kind": "absolute", "expires_at": "2026-09-01T00:00:00Z"}
        return now >= datetime.fromisoformat(rec.ttl["expires_at"])
    if kind == "turns":
        # {"kind": "turns", "turns": 5, "created_turn": 10}
        return (turn_counter - created_turn) >= n
    if kind == "scope":
        # {"kind": "scope", "scope": "session" | "dialogue"}
        scope = rec.ttl.get("scope")
        if scope == "session":
            return not session_active
        if scope == "dialogue":
            return not dialogue_active

§5.4 RAG Scoring Formula

5.4.1 Dialogue Tree Attachment (Construction)

Scoring when a new Q node selects a parent:

score = α·cos_sim + β·recency + δ·is_immediate_predecessor

Default parameters (DialogTree.__init__):

Parameter	Default	Meaning
`alpha`	0.3	Semantic similarity weight
`beta`	0.3	Time decay weight
`delta`	0.35	Immediate predecessor weight
`tau_attach`	0.35	Attachment threshold (below this → new root)
`recency_tau`	600.0	Time decay rate (seconds), `exp(-Δt / 600)`
`tree_tau`	3.0	Tree distance decay rate (hops)

python

def _choose_parent(self, node: Node) -> Optional[int]:
    best_score = -math.inf
    for cand in self.node_list:
        sim = self._cosine(node.embedding, cand.embedding)
        dt = max(0.0, node.timestamp - cand.timestamp)
        recency = math.exp(-dt / self.recency_tau)
        is_pred = 1.0 if cand.id == self._last_id else 0.0
        score = self.alpha * sim + self.beta * recency + self.delta * is_pred
        if score > best_score:
            best_score = score
            best_cand = cand
    if best_score < self.tau_attach:
        return None  # New root (topic shift → forest)
    # If the best candidate is a Q node with a parent A, climb to A (subtopic attaches under the shared answer)
    if best_cand.node_type == "Q" and best_cand.parent is not None:
        return best_cand.parent
    return best_cand.id

5.4.2 Query

DialogTree.query(text, k) reuses embeddings + topology:

final = α·cos_sim + β·recency + γ·exp(-tree_distance / τ_d)

Query parameters:

Parameter	Default	Meaning
`alpha`	0.3	Semantic similarity
`beta`	0.3	Time decay
`gamma`	0.15	Tree distance weight
`tree_tau`	3.0	Hop-count decay scale

5.4.3 Embedding Model

Model: BAAI/bge-m3 (default EMBEDDING_MODEL env var)
Shared by all components: DialogTree, AssistantRouter, RAG dense recall, rewriter data pipeline → comparable vectors across the entire codebase
Model instances are not cached on self (property routes to a singleton getter), allowing VRAM eviction to reclaim ~700MB

§5.5 Node Data Structure

python

@dataclass
class Node:
    title: str                          # Title (≤40 chars + "…")
    content: str = ""                   # Body
    id: int = -1                        # List index (assigned at create_node)
    level: int = 0                      # Tree depth
    parent: Optional[int] = None        # Parent node id
    node_type: str = "Q"                # "Q" (user) / "A" (assistant)
    embedding: list[float] = []         # bge-m3 vector
    children: list[int] = []            # Child node id list
    timestamp: float = time.time()      # Unix timestamp
    origin: str = "response"            # "user" / "response" / "proactive"
    metadata: dict = {}                 # Metadata (redactions, decision_id, ...)

5.5.1 Origin Semantics

origin	Meaning
`"user"`	User-input Q node
`"response"`	Model-reply A node (in response to a user Q)
`"proactive"`	Agent-initiated A node (decider / reminder triggered, no preceding Q)

5.5.2 Privacy Redaction Replay

python

def node_content_for_llm(node: Node) -> str:
    """Return node.content with stored privacy decisions replayed.
    
    Q nodes carry metadata['redactions'] with per-discovery decisions:
    - span=[start, end]
    - decision: "ignore" | "mask" | "record"
    - token: replacement string
    
    Replacements applied in reverse span order. If redactions_content_hash
    does not match, falls back to raw content.
    """

§5.6 DialogTree Storage Format

json

{
  "nodes": [
    {"id": 0, "title": "...", "content": "...", "node_type": "Q", 
     "parent": null, "children": [1], "embedding": [0.1, ...],
     "timestamp": 1724860800.0, "origin": "user", "metadata": {}},
    {"id": 1, "title": "...", "content": "...", "node_type": "A",
     "parent": 0, "children": [], "embedding": [0.2, ...],
     "timestamp": 1724860801.0, "origin": "response", "metadata": {}}
  ],
  "roots": [0],
  "_last_id": 1
}

Storage path: workspace/agents/agent_<id>/dialogue_<n>/dialogue.json

§5.7 MemoryWriter (Write Chain)

src/models/memorizer/writer.py decides what is worth remembering.

5.7.1 Write Pipeline

User message + dialogue history → MemorizerWriter (local small model)
    → Determine whether worth remembering
    → Classify (scope, category)
    → apply_writes(writer_output_json, ...)
    → MemoryStore.append(scope, category, content, ...)
    → Write to workspace/user/<category>.md (or identity / dialogue / ephemeral paths)

5.7.2 WriterOutput JSON

json

{
  "writes": [
    {
      "op": "add",           // "add" | "update" | "skip"
      "scope": "user",
      "category": "profile",
      "entry": {
        "content": "The user's name is Alice",
        "tags": ["name"],
        "salience": 0.8,
        "ttl": null,
        "supersedes_id": null,
        "epistemic_status": "asserted",
        "pinned": false,
        "source_quote": ["User message: 'My name is Alice'"]
      }
    }
  ]
}

5.7.3 apply_writes Budget

python

def apply_writes(writer_output_json, *, workspace, agent_id="agent_0",
                 dialogue_id=None, turn_counter=None, turn_id=None,
                 max_chars_total=1500) -> list[MemoryRecord]:
    """Apply WriterOutput JSON to disk. Returns new records committed.
    
    Greedy budget: accept writes in order until cumulative content exceeds
    max_chars_total. The first write is always allowed. Turns that exceed
    the budget retain the highest-priority facts.
    """

§5.8 MemoryReader (Read Chain)

src/models/memorizer/reader.py assembles RAG context.

5.8.1 Read Pipeline

Every NL query → DialogueManager.build_rag_context(query)
    → Dense recall (bge-m3 cosine)
    → Time decay
    → Tree distance
    → Sort by combined score
    → Inject top_k into context

5.8.2 is_live Filtering

python

def is_live(rec, **ctx) -> bool:
    return not rec.deprecated and not is_expired(rec, **ctx)

§5.9 Consolidator

src/models/memorizer/consolidator.py periodically consolidates duplicate / similar memories.

python

# Consolidator pipeline:
# 1. Group by (scope, category)
# 2. Compute embedding similarity
# 3. High-similarity clusters → generate consolidated entry
# 4. Mark original entries deprecated=True
# 5. New entry has derived_from=[...source_ids]

§5.10 DeciderService Eviction Strategy

5.10.1 VRAM Gate Integration

The Decider model is included in VRAMGate's evictable list (priority 3 — NICE):

Priority	Model	Category	Description
1 — CRITICAL	TTS / whisper / embedder (user-active requests)	sync	Never evicted
2 — IMPORTANT	Selected agent's decider, bge embedding, reranker	sync	During user interaction
3 — NICE	Memorizer writer, mini-model, EasyOCR, emotion tagger	queue	Can be interrupted + checkpointed
4 — BACKGROUND	Indexing / training	queue	Never user-visible

5.10.2 Eviction Categories

python

@dataclass
class _Entry:
    model_id: str
    priority: int
    estimated_mb: float
    on_evict: Optional[Callable[[], None]] = None
    category: Literal["sync", "queue"] = "sync"
    is_working: bool = False
    on_checkpoint: Optional[Callable[[], Any]] = None
    last_active_at: float = field(default_factory=time.monotonic)
    idle_ttl_sec: Optional[float] = None

sync: user is blocked. A new sync acquire can evict any idle resident or queue resident (with checkpoint).
queue: interruptible. On eviction, on_checkpoint() runs to snapshot state for later resumption.

5.10.3 Idle Reclamation

python

# Gate compares time.monotonic() - last_active_at against VRAM_GATE_IDLE_EVICT_SEC
# Residents exceeding the threshold are reclaimed; VRAM returned to the pool

5.10.4 Thread Priority Boosting

python

@contextmanager
def boost_thread_priority(priority: int) -> Iterator[None]:
    """Temporarily boost priority for a user-blocking thread.
    
    Nesting takes the min — inner CRITICAL + outer NICE = inner CRITICAL.
    """

§5.11 Observability

Interface	Purpose
`memory writer-status` (CLI)	View writer status
`GET /api/memory_status`	Memory size / hit statistics
`POST /api/agents/{id}/release-memory`	Manually release resident memory
`GET /api/memory`	Memory operations

Next: 05 - Persistence Layer →

04 · RAG & Memory System ​

§5.1 Why Not Use a Vector Database ​

§5.2 Memory Scope × Category Matrix ​

5.2.1 Scope ​

5.2.2 Category ​

5.2.3 Valid (Scope, Category) Combinations ​

5.2.4 Path Resolution ​

§5.3 Memory Entry Format ​

5.3.1 On-Disk Format ​

5.3.2 MemoryRecord Data Structure ​

5.3.3 TTL Policies ​

§5.4 RAG Scoring Formula ​

5.4.1 Dialogue Tree Attachment (Construction) ​

5.4.2 Query ​

5.4.3 Embedding Model ​

§5.5 Node Data Structure ​

5.5.1 Origin Semantics ​

5.5.2 Privacy Redaction Replay ​

§5.6 DialogTree Storage Format ​

§5.7 MemoryWriter (Write Chain) ​

5.7.1 Write Pipeline ​

5.7.2 WriterOutput JSON ​

5.7.3 apply_writes Budget ​

§5.8 MemoryReader (Read Chain) ​

5.8.1 Read Pipeline ​

5.8.2 is_live Filtering ​

§5.9 Consolidator ​

§5.10 DeciderService Eviction Strategy ​

5.10.1 VRAM Gate Integration ​

5.10.2 Eviction Categories ​

5.10.3 Idle Reclamation ​

5.10.4 Thread Priority Boosting ​

§5.11 Observability ​

04 · RAG & Memory System

§5.1 Why Not Use a Vector Database

§5.2 Memory Scope × Category Matrix

5.2.1 Scope

5.2.2 Category

5.2.3 Valid (Scope, Category) Combinations

5.2.4 Path Resolution

§5.3 Memory Entry Format

5.3.1 On-Disk Format

5.3.2 MemoryRecord Data Structure

5.3.3 TTL Policies

§5.4 RAG Scoring Formula

5.4.1 Dialogue Tree Attachment (Construction)

5.4.2 Query

5.4.3 Embedding Model

§5.5 Node Data Structure

5.5.1 Origin Semantics

5.5.2 Privacy Redaction Replay

§5.6 DialogTree Storage Format

§5.7 MemoryWriter (Write Chain)

5.7.1 Write Pipeline

5.7.2 WriterOutput JSON

5.7.3 apply_writes Budget

§5.8 MemoryReader (Read Chain)

5.8.1 Read Pipeline

5.8.2 is_live Filtering

§5.9 Consolidator

§5.10 DeciderService Eviction Strategy

5.10.1 VRAM Gate Integration

5.10.2 Eviction Categories

5.10.3 Idle Reclamation

5.10.4 Thread Priority Boosting

§5.11 Observability