Internals Development Manual

For developers doing secondary development on the source code. Not a user manual — this covers data flow, startup order, memory model, tool system, and "pitfalls to avoid."
Source: CLAUDE.md + ARCHITECTURE.md + src/bridge/app.py + empirically accumulated knowledge.

0. One-liner

Familiars = Electron (Vue 3) frontend → local FastAPI bridge (127.0.0.1:18711, HTTPS/HTTP2) → OpenAI-compatible LLM + local QLoRA small models + tool system. Single-process backend, Windows-first, local-first.

Renderer ──window.familiars API──▶ main.js (preload / IPC)
 │ spawn
 ▼
 src.bridge.run (hypercorn HTTPS/2)
 │ X-Bridge-Token
 ┌────────────┴──────────────┐
 ▼                           ▼
 DialogueManager             ToolDispatcher
 / ConversationSession       (LLM tool-loop, MAX_TURNS=50)
 │                           │
 ▼                           ├─ security gate (risk score/approval)
 MemoryWriter                ├─ browser/shared-Chrome
 (Scoping:                   ├─ multimodal/vision
  dialogue/agent/user)       ├─ memory/tools
 │                           └─ broker / skill……
 ▼                           ▼
 SQLite (familiars.db, WAL)  workspace/ file tree

1. Three-tier Agent Architecture

1.1 AssistantManager (`src/models/assistant.py`)

Agent 0 (the manager). Holds N Assistant instances.
Runs a separate MANAGER_MODEL (default gpt-4o) for executive summaries and persona evolution.
Owns the bridge_call tool (unique), can call all bridge REST endpoints.
System prompt is injected with framework doc index + Bridge API surface index.

1.2 Assistant

Single specialized agent. Owns its DialogueManager, MemoryWriter, vision pipeline, voice/digital human pipeline.
Sends notifications: Feishu webhook / Windows toast / macOS osascript / Telegram.
Writes reports: daily/weekly/monthly/annual stored under workspace/agents/agent_<id>/reports/.

1.3 ConversationSession + DialogueManager (`src/models/conversation.py`)

ConversationSession: Streams from OpenAI-compatible endpoint, drives multi-turn tool-call loop (up to MAX_TURNS=50).
DialogueManager: Owns dialogue trees (DialogTree), builds RAG history for each turn (α·cos + e^{-β·Δt} + γ·tree_distance).
MAX_TURNS is no longer a hard cap — when budget is exhausted, ConversationSession asks _turn_budget_escalator (manager LLM) whether to extend or abort.

2. Bridge Startup Side-effect Order (Critical!)

src/bridge/app.py executes side-effects at import time in a precise order. Changing the order causes problems:

#	Step	Code Location	Why This Order
1	Pin `WORKSPACE_ROUTE`	`app.py:11-14`	Inject repo root into env var before any other module import
2	Main thread pin CUDA	`app.py:28-44`	`torch.cuda.init()` + `zeros(1, device="cuda")` + `synchronize()`. Windows WDDM does not allow worker threads to first-touch CUDA → access violation
3	`align_all_paths()`	`app.py:46-80`	Auto-align path drift. Must run before DialogueManager reads dialogues, otherwise LLM will echo old paths as context
4	Import 46+ route modules	`app.py:82-86`	One-shot import of all routes + comm_router
5	CORS + BridgeTokenMiddleware	`app.py:108/123/235/381`	Only allows `localhost:5173` / `127.0.0.1:5173` / `null` origin
6	`include_router` mount routes	`app.py:383-443`	Order is code order. First-registered prefix wins
7	`_bootstrap_broker_workshop`	`on_event("startup")`	Sync built-in brokers to workspace, idempotent
8	`_bump_anyio_thread_limiter`	Same	Thread pool limit 40 → 128
9	`_start_pool_saturation_watchdog`	Same	Sample every 30s, consecutive ≥90% writes stack dump
10	`_install_browser_lifecycle_hooks`	Same	`atexit` reclaims Playwright driver, idle reaper guard
11	`_kill_orphan_gpu_subprocesses_on_boot`	Module level (`app.py:863`, call site `1058`)	Kill orphan GPU processes via `nvidia-smi` + `.bridges.json`
12	Model prewarm is on-demand JIT (no `_prewarm_models`)	Module level	embedder / rembg / rewriter lazy-load when admitted by vram_gate; digital-human sessions prewarm on demand via routes (`routes/digital_human.py:336/381`)
13	autorun drift check	Same	Auto-reinstall OS autorun when lost during background running
14	`_rehydrate_monitors`	Same	Restore monitor watchers from DB
15	`_rehydrate_cron_jobs`	Same	Force DailyTools instantiation + catch-up fires missed within 12h
16	`_register_manager_jobs`	Same	Register manager daily/weekly/monthly/annual report aggregation crons
17	`_rehydrate_skill_scheduler`	Same	Restore skill scheduler periodic tasks (must be after #16)

3. Path Alignment Mechanism (Don't Panic When Paths Change)

Two mechanisms cooperate:

3.1 Repo-root Drift (Auto-detected)

At startup, compares all dialogue.path prefixes against _repo_root(). On drift, auto-rewrites:

dialogue.json (node content + metadata.tool_calls args/results + attachments)
turn_*/plan.json
step_*.json

Recorded to workspace/.paths_aliases.json for audit.

3.2 External Directory Aliases (Manual Entry)

bash

# Preview
python -m src.tools.paths_migrate <old_path> <new_path> --dry-run

# Execute (each modified file backed up as .bak_<ts>)
python -m src.tools.paths_migrate <old_path> <new_path> --record

Replayed on every subsequent boot.

3.3 What Is NOT Changed

Log files (.bridge.log, audit.log) are never rewritten — they preserve the authentic history.

4. Memory / VRAM Model

4.1 Single VRAM Admission Gate

src/base/vram_gate.py: 4-level priority, 3-level eviction. All VRAM-needing components (embedder / rembg / rewriter / DH / ASR) go through the same gate.

4.2 Dialogue JIT Loading

DialogueManager.release_memory + ensure_dialogue_loaded: MRU callback evicts inactive dialogues, new requests JIT-load.
HTTP endpoints: POST /api/agents/{id}/release-memory; GET /api/memory for observation.

4.3 on_evict Convention

Any object holding VRAM/memory must implement an on_evict callback for cooperative release during eviction. This is a hard convention — not implementing it = VRAM leak.

5. Tool System

5.1 ToolDispatcher Registry

src/tools/tool_dispatcher.py registers 15 default tool instances:

#	Instance	Source
1	`DailyTools`	`src/utils/daily_tools.py`
2	`DailyMonitorTools`	`src/utils/daily_monitor/`
3	`BaseTools`	`src/utils/base_tools.py`
4	`BaseTools.InputController`	`src/utils/base_tools.py`
5	`MultimodalTools`	`src/utils/multimodal_tools.py`
6	`BrowserTools`	`src/utils/browser_tools.py`
7	`DocTools`	`src/utils/doc/tools.py`
8	`MemoryTools`	`src/utils/memory_tools.py`
9	`SystemTools`	`src/utils/system_tools.py`
10	`UserTools`	`src/utils/user_tools.py`
11	`UserSecretTools`	`src/utils/user_secret_tools.py`
12	`PhoneTools`	`src/utils/phone_tools.py`
13	`SkillTools`	`src/tools/skill_tools.py`
14	`SkillWatcherTools`	`src/utils/skill_watcher_tools.py`
15	`BrokerTools`	`src/utils/broker_tools.py`

Registered on demand:

run_call_conversation (when phone call session is importable)
ask_user_questions (when src/models/processor/ask_user.py is importable)
bridge_call (manager only, src/tools/bridge_call.py)

5.2 Schema Distribution

Tool schemas stored in workspace/tools.json.
Merged per-client: src/models/clients/{base,browser,multimodal,input,doc,daily,system,mini_model}_client.py each filter independently.
_CONSULT_TARGETS still routes consult_doc to doc_tools — do not delete the mapping in base_client.py.
Recipe SKILLs reach flow agents via execute_skill pseudo-tool + tools.whitelist.

5.3 Security Gate

src/tools/security.py:

Each tool annotated with risk level
allow_tool_for_turn decides whether to allow based on agent mode (intense/turbo) + current turn context
High-risk operations pause and wait for user confirmation
FAMILIARS_APPROVAL_DISABLED=1 disables the approval gate (for development)

5.4 Turn Secrets

src/tools/turn_secrets.py: Replaces ENC:: encrypted tokens for the current turn, destroyed at turn end.

6. Agent Pipeline (Generation.py)

User message → Rewriter → Planner → Executor(+Transferer) → Verifier → Synthesizer → Final reply
                                                              ↓ on failure
                                                         Replanner → back to Planner
                                                                         ↓
                                                                  MemoryWriter (background)

Phase	File	Responsibility
Rewriter	`src/models/rewriter.py`	Raw prompt → structured query
Planner	`src/models/processor/planner.py`	Outputs Step list (tool call plan)
Executor	`src/models/processor/executor.py`	Dispatches each Step via Transferer
Transferer	`src/models/transferer.py`	6-priority cross-agent/client/model scheduler
Verifier	`src/models/processor/verifier.py`	RuleVerifier (deterministic) + ModelVerifier (LLM semantic)
Replanner	`src/models/processor/replanner.py`	Re-plans when verification fails
Synthesizer	`src/models/processor/synthesizer.py`	Execution results → natural language reply
MemoryWriter	`src/models/memorizer/writer.py`	Background structured memory extraction (does not block SSE)

6.1 Model Routing

ModelRouter (src/models/router/model_router.py): Selects DEFAULT_MODEL or stronger model based on task characteristics.
MetaRouter (src/models/router/meta_router.py): Manager meta-role unified call entry point.
All model calls go through ConversationSession uniformly.

6.2 Client Abstraction

ToolClient base class (src/models/clients/base_client.py):

chat() implements bounded tool loop (MAX_TURNS)
Auto-handles consult_* cross-client calls
end_step synthesis validation + death loop detection
_guard_native_calls prevents cross-client phantom calls

7. Memory Architecture

7.1 Two Axes

Axis	Persistence Location
`dialogue`	`workspace/agents/agent_<id>/dialogue_<n>/`
`agent`	`workspace/agents/agent_<id>/`
`user` / `identity`	`workspace/` (USER.md / IDENTITY.md / memory layers)
`ephemeral`	Per-turn, evicted at turn end

7.2 Five Categories

event, state, profile, procedure, directive. Each category has independent decay constants.

7.3 DialogTree

Node carries embedding (BAAI/bge-m3)
Embedding model is shared by DialogTree / AssistantRouter / RAG dense recall / rewriter data generation pipeline
All components produce comparable vectors

8. Configuration System

src/base/config.py uses pydantic-settings, validate_config() returns Settings singleton.

8.1 Key Environment Variables

See Setup → Environment Variables. Key ones listed here:

BASE_URL / API_KEY / DEFAULT_MODEL: Required, missing throws ValueError
FAMILIARS_DB_PATH: SQLite path, use :memory: for tests
FAMILIARS_HTTP_INSECURE=1: Fall back to HTTP + uvicorn
FAMILIARS_APPROVAL_DISABLED=1: Disable approval gate
FAMILIARS_TIER*_MAX_MB: GPU tier VRAM thresholds

8.2 HKDF Key Derivation

src/base/crypto.py: info string b"familiars-secrets-v1"
Rotation invalidates all workspace/SECRET.json, must accompany a re-encryption migration

8.3 SQLite

File: workspace/familiars.db (WAL mode)
src/storage/db.py:get_default_db_path
Migrations: src/storage/migrations/ v1 ~ v19 (includes Python migration v16)

9. Test Red Lines

#	Red Line	Why
1	Must use real SQLite (`FAMILIARS_DB_PATH=:memory:`)	Mock repos cannot cover WAL concurrency / migrations / transaction boundaries
2	Bridge route tests: restore `sys.modules` captures	Do not `importlib.reload` route modules — import-time side-effects (CUDA pin / path alignment) cannot be replayed
3	Scratch agent directories go in `runs/`	Bridge scans `workspace/agents/` at startup and will create real agents from what it finds
4	Browser automation goes through `familiars_cdp_port()`	Single shared Chrome instance (bridge + Playwright + see-through rendering all share)
5	New logs must enter `KNOWN_LOGS`	The list in `src/utils/framework_tools.py` controls manager visibility
6	Never use `git stash`	CLAUDE.md hard rule. stash pop has repeatedly corrupted the working tree (path.txt / source code reverts)

10. Common Pitfalls (Hard-won)

Pitfall	Description	Correct Approach
`uvicorn src.bridge.app:app --reload`	Loses HTTP/2 + worker thread race on CUDA init → crash	Use `python -m src.bridge.run`
Two routers same prefix	First-registered wins. `broker_user` vs `user` once collided on `/api/user/profile`	Think about conflicts before adding prefix routes
HKDF info string	`b"familiars-secrets-v1"` — rotation invalidates all SECRET.json	Must accompany re-encryption migration
`workspace/tools.json` empty parameters	LLM cannot see that tool	Add `reason` + `placeholder` to every empty schema
Windows toast swallowed by Focus Assist	WinRT Show() no exception but fullscreen game triggers Focus Assist silently dropping Normal level	`build_toast_script` sets `ToastNotificationPriority.High` to pierce through
Bridge port is not fixed 18711	Bridge reads `workspace/.bridge.port` at runtime	Probe the port file before determining bridge is down
Copilot LLM cannot see custom tools	`filtered_schemas` skips falsy `parameters`	Every empty schema must have reason + placeholder
VueFlow node-types must be markRaw	Inline `{ graph: GraphNode }` → Vue wraps in Proxy	`markRaw(types)`
VueFlow custom edge/node single root	Two-root template → warning spam → main thread saturation	Wrap in single `<g>` root
Deep watcher on store.graph	`watch(store.graph, ..., {deep:true})` → render loop	Use fingerprint string watcher
contextvars cross-request leak	BaseHTTPMiddleware reuses worker thread	`_fresh_request_context()` resets vars

11. Manager's Three Wiring Points

Manager (agent 0) has three always-on wiring points:

11.1 Framework Doc Index

src/models/RAG/framework_docs.py:render_framework_index injects every turn:

One-line summary of OPERATION.md / ARCHITECTURE.md / SECURITY.md / REPOSITORY.md / CLAUDE.md / CLI.md
Log registry

11.2 Bridge API Surface Index

src/models/RAG/bridge_api_surface.py hand-curated REST endpoint catalog.

11.3 Four Dedicated Tools

src/utils/framework_tools.py filtered_schemas injects extra tools for manager:

list_framework_docs — list framework doc sections
read_framework_doc — read document content
list_logs — list known log files
read_log — read log content

KNOWN_LOGS list controls visibility. New noteworthy logs must be registered here.

12. Broker

12.1 Four-layer Maturity

Layer	Status	Description
A Intra-cluster communication	✅ Complete	Local broker hub: identity + transport + match engine + rate limiter
B Application container	Partial	No sandbox, plugins run in bridge process
C Marketplace distribution	✅ MVP	Pack + sign + verify + install + relay publish
D Cross-cluster federation	Hooks only	Service basic addressing, no complete protocol

12.2 Key Files

Subsystem	Code
Hub	`src/broker/hub.py`
Workshop	`src/broker/workshop/__init__.py`
Routes	`src/bridge/routes/broker.py` / `broker_workshop.py` / `broker_app.py`
Author keys	`~/.familiars/authors/<name>.key` (base64 Ed25519 seed)
Reference template	`workspace/manager/brokers/starter_broker/`

12.3 Signing/Install Flow

bash

# Author
python main.py broker keygen --name mykey
python main.py broker pack workspace/manager/brokers/my_broker
python main.py broker sign my_broker.broker.zip --author-key mykey
python main.py broker publish ... --author-key mykey --relay https://api.familiars.cn

# User
python main.py broker install --url https://... --sha256 <hex> --pubkey <b64>

Install enforces sha256 + Ed25519 signature + author pubkey match verification.

13. Recipe / Flow Ecosystem

13.1 Recipe

Folder protocol: manifest.json + knowledge/ + utils/ + docs/ + renderers/
Compile → dry-run → e2e test (POST /api/recipe_dev/*)
Bind to agent → runtime maybe_override replaces pipeline chain segments
Author tools: python main.py recipe pack/sign/verify/publish
Capability approval: capabilities.extra_modules + per-(recipe,agent) approval hash

13.2 Flow

V2 track-based flow editor
Templates stored at workspace/manager/flows/<template_id>/
Instance lifecycle: /api/flows (list / get / trace / cancel)
Template CRUD: /api/flow-templates
Built-in templates: src/flow/_builtin/<id>/, install_built_ins() read-only
Engine: src/flow/engine.py (owner-thread + per-role worker-thread concurrency)
Resume: src/flow/resume.py (mutation replay + hash verification)

14. Security Model

14.1 Data Flow

User data defaults to landing in workspace/
Only sends context to cloud LLM when necessary
src/base/crypto.py encrypts sensitive values → workspace/SECRET.json

14.2 Approval Gate

src/tools/security.py: Risk levels + approval gate
High-risk operations pause and wait for user confirmation
FAMILIARS_APPROVAL_DISABLED=1 skips (for development)

14.3 Sensitive Data Detection

src/tools/sensitive_detector.py: gitleaks → Presidio → mini-LLM three-channel detector.

14.4 Token Authentication

X-Bridge-Token (header or ?token= query)
BridgeTokenMiddleware global validation
Exemptions: /health, /api/auth/*, /api/broker_app/*, /api/demo/*, /api/chat/*

15. Frontend Architecture Notes

15.1 Two Manager Homepages

New Desktop.vue: Desktop-style shell (icons + floating windows + wallpaper), built-in capabilities + broker iframe
Legacy ManagerLanding.vue: Five-card layout, switched via localStorage.familiars:legacyLanding

15.2 Broker iframe Sandbox

BrokerAppWindow mounts iframe to load broker UI
Static assets served via /api/broker_app/{broker_id}/ui/{rel} (token-exempt)
Sensitive operations go through postMessage → renderer → bridge (with token)

15.3 API SDK

electron/renderer/src/api/index.ts centralizes fetch wrapper, auto-attaches X-Bridge-Token + HTTPS/2 base URL.

Internals Development Manual ​

0. One-liner ​

1. Three-tier Agent Architecture ​

1.1 AssistantManager (src/models/assistant.py) ​

1.2 Assistant ​

1.3 ConversationSession + DialogueManager (src/models/conversation.py) ​

2. Bridge Startup Side-effect Order (Critical!) ​

3. Path Alignment Mechanism (Don't Panic When Paths Change) ​

3.1 Repo-root Drift (Auto-detected) ​

3.2 External Directory Aliases (Manual Entry) ​

3.3 What Is NOT Changed ​

4. Memory / VRAM Model ​

4.1 Single VRAM Admission Gate ​

4.2 Dialogue JIT Loading ​

4.3 on_evict Convention ​

5. Tool System ​

5.1 ToolDispatcher Registry ​

5.2 Schema Distribution ​

5.3 Security Gate ​

5.4 Turn Secrets ​

6. Agent Pipeline (Generation.py) ​

6.1 Model Routing ​

6.2 Client Abstraction ​

7. Memory Architecture ​

7.1 Two Axes ​

7.2 Five Categories ​

7.3 DialogTree ​

8. Configuration System ​

8.1 Key Environment Variables ​

8.2 HKDF Key Derivation ​

8.3 SQLite ​

9. Test Red Lines ​

10. Common Pitfalls (Hard-won) ​

11. Manager's Three Wiring Points ​

11.1 Framework Doc Index ​

11.2 Bridge API Surface Index ​

11.3 Four Dedicated Tools ​

12. Broker ​

12.1 Four-layer Maturity ​

12.2 Key Files ​

12.3 Signing/Install Flow ​

13. Recipe / Flow Ecosystem ​

13.1 Recipe ​

13.2 Flow ​

14. Security Model ​

14.1 Data Flow ​

14.2 Approval Gate ​

14.3 Sensitive Data Detection ​

14.4 Token Authentication ​

15. Frontend Architecture Notes ​

15.1 Two Manager Homepages ​

15.2 Broker iframe Sandbox ​

15.3 API SDK ​

16. Next Steps ​