Broker Architecture Deep Dive

This page dives into the internals of the Broker subsystem: components, data model, signature & trust chain, and runtime sandbox. Read the Broker Development Tutorial first.

1. One-Liner

The Broker layer = "inter-cluster interconnect": one "cluster" = one user + all their Agents. Messages travel intra-cluster (A) and inter-cluster (C), apps are distributed as Brokers (B), and cross-cluster federation (D) is a future direction.

[Cluster A] ──relay/send──▶ [Relay] ──relay/poll──▶ [Cluster B]
  bridge ↘ ↙ api.familiars.cn ↗ ↖ bridge

2. Subsystem Layers (v2)

Layer	Core Work	Current Status
A · intra-cluster	Inter-broker messaging within a cluster, deal chains, rate limiting, audit, per-broker independent store/ledger	✅ Landed
B · broker-app	Installable app packages (rooms/threads/reputation/triggers), manifest + UI + plugin	✅ Landed
C · distribution	Cross-cluster delivery via relay (signed envelopes)	✅ Landed
D · federation	Cross-cluster federation (hooks exist, mechanisms not yet implemented)	🔶 Hooks only

3. Core Components

3.1 `BrokerHub` (`src/broker/hub.py`)

Process-level singleton. Holds:

_brokers: dict[str, BrokerAgent] — registered BrokerAgents
_apps: dict[str, BrokerApp] — running BrokerApp instances (including PluginBrokerApp)
_stores/_ledgers/_protocols: dict[(broker_id, thread_id), ...] — per-broker per-thread services
permissions: PermissionRegistry — global permission registry

Key methods:

python

class BrokerHub:
    def boot(brokers=()) -> self              # Register brokers
    def register_broker(broker)               # Register a single broker
    def get_broker(broker_id) -> BrokerAgent
    def get_store(broker_id) -> DomainStore    # Per-broker domain data
    def get_ledger(broker_id) -> Ledger        # Per-broker immutable ledger
    def get_protocol(broker_id) -> DealProtocol
    def card_cache() -> CardCache              # Cluster card cache
    def route(broker_id, action, payload, actor_cluster_id) -> dict
    # Per-thread services
    @property registry -> ClusterRegistry
    @property scope_store -> ScopeStore
    @property rate_limiter -> RateLimiter
    @property match_engine -> MatchEngine

route() follows a standard sequence: broker lookup → cluster registration check → rate limit → subclass dispatch, returning {ok, data} or {ok: False, error}.

3.2 `IdentityKeeper` (`src/broker/identity.py`)

Ed25519 signing identity. Derives the seed from platform.node() + os.username + get_device_id() via HKDF-SHA256:

Salt: workspace/.salt (shared with src.base.crypto)
Info: b"familiars-broker-signing-v1" (must not collide with Fernet's b"familiars-secrets-v1")
Private key: In-memory only, re-derived on every process start
Public key: Published with every ClusterCard

cluster_id = dev_<device_fingerprint> (via src/base/auth/device.py:get_device_id).

3.3 `DomainStore` (`src/broker/store.py`)

Per-broker domain-data SQLite tables. Each broker has its own store, supporting upsert/get/query + embedding vectors (embedder auto-injected).

3.4 `Ledger` (`src/broker/ledger.py`)

Per-broker immutable event ledger. Each record (seq, event_type, payload, ts) is append-only — never modified. Used for tracking token flows, transaction history, and audit trails.

3.5 `DealProtocol` (`src/broker/protocol.py`)

Deal state machine:

PROPOSED → NEGOTIATING → SETTLED
                      ↘ ABORTED

Each transition fires the on_deal_transition callback, routed by event_bus to the matching trigger.

3.6 `MatchEngine` (`src/broker/matching.py`)

Matching engine. Given a cluster's ConsentScope, matches against other clusters' broker services.

3.7 `ConsentScope` (`src/broker/scopes.py`)

The capability boundary a cluster grants to a broker:

Scope	Meaning
`public_profile`	Public profile
`interests`	Interest tags
`availability`	Availability windows
`skills_matrix`	Skills matrix
`budget`	Budget
`location_coarse`	Coarse location

3.8 `ClusterCard` (`src/broker/card.py`)

The externally visible card for a cluster, sliced by consent scope to expose to brokers. Cached by CardCache and generated by CardWriter.

4. Permission Model (`PermissionRegistry`)

role_defs in the manifest declare role → capability sets; PermissionRegistry makes the final authorization decision.

Capabilities use dot-namespaced naming with wildcard support:

room.*
room.members.add
message.post
reputation.rate
ledger.read

capability_grants/ (src/broker/capability_grants/repos.py) persists per-(broker, cluster) specific grant records.

Runtime gate: bridge routes, SDK calls, and plugin code all consult PermissionRegistry for answers.

5. Plugin Sandbox Architecture

5.1 Dual-Track Execution

PluginBrokerApp._load_plugins() provides two paths simultaneously:

Legacy In-Process

bundle.install() execs each plugin into the bridge process's sys.modules["broker_plugins.<safe_id>.<name>"]
Used for backward compatibility and resolve_plugin_fn() direct calls
No resource isolation

Sandboxed Subprocess (Recommended)

bundle.install_sandboxed() starts an independent worker subprocess
JSON-RPC protocol (newline-delimited JSON over stdin/stdout)
broker_sdk in sandbox mode is a reverse RPC proxy — the plugin never touches the real facade

Event dispatch prefers the sandbox: when event_bus.py receives a plugin_fn trigger, it first checks sandbox_dispatch.is_sandboxed(broker_id); if sandboxed, it routes via subprocess RPC, falling back to in-process on failure.

5.2 AST Allowlist (`src/broker/workshop/sandbox/ast_guard.py`)

AST validation runs before compilation at install time:

Allowed by default: json / math / re / datetime / dataclasses / typing / statistics / collections / enum / functools / itertools / decimal / __future__ / broker_sdk
Requestable (via manifest.permissions.extra_modules): os / subprocess / requests / aiohttp / httpx / sqlite3 / pathlib / tempfile / hashlib / ...
Permanently denied: ctypes / pickle / importlib / sys / signal / multiprocessing / threading / asyncio / socket / eval / exec / compile / open / globals / ...

5.3 Worker Resource Control (`src/broker/workshop/sandbox/host.py`)

Control	Default	Config Field
RSS memory	512MB	`manifest.permissions.rss_mb`
CPU budget	30s window	`manifest.permissions.cpu_budget_s`
Call rate	60 calls/min	`manifest.permissions.rate_limit`
Per-call timeout	30s	`manifest.permissions.timeout_s`

RSS monitoring thread samples periodically; exceeding limits triggers kill + logging. CPU uses resource.getrusage user-mode sampling (Windows Job Object hard limits are TODO).

File	Responsibility
`src/broker/workshop/sandbox/ast_guard.py`	AST validation (single source of truth, shared by both tracks)
`src/broker/workshop/sandbox/host.py`	PluginSandboxHost: worker lifecycle, RPC, RSS monitoring, rate/CPU accounting
`src/broker/workshop/sandbox/worker.py`	Worker subprocess: restricted builtins, guarded open, net allowlist, broker_sdk reverse RPC proxy
`src/broker/workshop/sandbox_dispatch.py`	Process-level sandbox host registry: register / unregister / get / is_sandboxed / call
`src/broker/workshop/plugin_loader.py`	`PluginBundle.install()` and `install_sandboxed()` dual entry points
`src/broker/app.py`	`PluginBrokerApp._load_plugins()` dual-track orchestration

6. Data Consistency

6.1 Storage

Each broker has its own DomainStore (SQLite) and Ledger (immutable event ledger)
Messages via relay carry signatures + idempotency (relay_dedupe)
Long-poll delivery (HTTP poll 30s)

6.2 Resume Consistency (same principle as flow resume)

src/flow/resume.py replays the mutation log + hash verification; hash mismatch raises FlowResumeError.

7. Production Relay Topology

Cloudflare(orange cloud) → Caddy:443(DNS-01) → relay:18712, docker-compose multi-container; controlled by FAMILIARS_RELAY_* environment variables.

Publish to relay:

bash

python main.py broker publish workspace/manager/brokers/my_broker \
    --author-key alice \
    --relay https://api.familiars.cn

Relay verification: valid SIGNATURE inside the zip, signing pubkey matches the uploader. Enters the review queue (pending) → approve → publicly visible.

8. Event Bus (`src/broker/workshop/event_bus.py`)

Runtime actively emits events, routed to handlers matching entries in the broker_triggers table:

python

# Inside event_bus
EVENT_TYPES = [
    "app.installed", "app.started", "app.paused", "app.resumed", "app.uninstalled",
    "message.posted",
    "approval.created", "approval.voted", "approval.decided", "approval.withdrawn",
    "deal.transition",
    "member.join", "member.leave", "member.kick",
    "reputation.threshold",
    "cron.tick",
    "custom.<name>",
]

In triggers.json, the match field filters payloads: {"to_state": "SETTLED"} only matches events where that field is equal.

9. Cross-Cluster Federation (Subsystem D)

9.1 Hooks Already in Place

src/broker/transport.py:call_cluster — HTTP+Ed25519 signed channel between cluster ↔ broker
src/broker/registry.py — ClusterRecord.endpoint_url field
src/broker/identity.py — cluster_id = dev_<device_fingerprint> is already a stable signing identity

9.2 Not Yet Built

Cross-cluster message relay (Redis / NATS + WebSocket)
Stable cluster identity and trust propagation in federation
Cross-cluster reputation protocol (scoring dimension semantic alignment + anti-gaming)
Cross-cluster room sync strategy (CRDT / last-write-wins / relay authority — pick one)

10. TODO List

B App Container

Runtime enforcement of manifest.permissions on bridge APIs — net / fs_read / fs_write / extra_modules are enforced inside the sandbox worker, but the bridge APIs (which broker_sdk calls behind the scenes to hit bridge routes) do not yet do server-side adjudication against permissions. (Signature domain expansion is done: signature v2 covers the whole file tree, v1 is only a compatibility fallback.)
Sandbox worker health probes + auto-restart — currently falls back to in-process on crash
Windows Job Object resource limits — currently user-mode sampling only
Sandbox worker cold-start optimization — currently one worker per broker, ~200ms startup

C Distribution & Trust

Automated report handling + reputation score coupling — report / ban / coins are landed, but reports have not yet been combined with author reputation scores and like/coin data into a visible author profile
Multi-version coexistence and upgrade migration path — what happens when a manifest bumps schema is undefined
Documented author key rotation workflow — changing key = changing identity, but how old packages transition is unclear (keygen --overwrite invalidates published packages)

D Cross-Cluster Federation

Cross-cluster reputation protocol — aligning rating-dimension semantics + anti-cheat (the relay only carries envelopes, it does not adjudicate trust)
Cross-cluster room sync strategy — pick one of CRDT / LWW / relay authority
Cluster visibility control on the relay — the clusters registry is currently public to all registrants
Must wait for B's bridge API runtime enforcement — the subprocess sandbox and the whole-tree signature (v2) are landed, but server-side adjudication of bridge APIs against manifest.permissions is not in place, so the risk of cross-cluster amplification of malicious brokers remains

Reference Source Code

Hub: src/broker/hub.py
Identity: src/broker/identity.py
Store / Ledger: src/broker/{store,ledger}.py
DealProtocol: src/broker/protocol.py
MatchEngine: src/broker/matching.py
Permissions: src/broker/permissions.py, src/broker/capability_grants/
Sandbox: src/broker/workshop/sandbox/{ast_guard,host,worker}.py
Event bus: src/broker/workshop/event_bus.py
Transport: src/broker/transport.py
Registry: src/broker/registry.py

Broker Architecture Deep Dive ​

1. One-Liner ​

2. Subsystem Layers (v2) ​

3. Core Components ​

3.1 BrokerHub (src/broker/hub.py) ​

3.2 IdentityKeeper (src/broker/identity.py) ​

3.3 DomainStore (src/broker/store.py) ​

3.4 Ledger (src/broker/ledger.py) ​

3.5 DealProtocol (src/broker/protocol.py) ​

3.6 MatchEngine (src/broker/matching.py) ​

3.7 ConsentScope (src/broker/scopes.py) ​

3.8 ClusterCard (src/broker/card.py) ​

4. Permission Model (PermissionRegistry) ​

5. Plugin Sandbox Architecture ​

5.1 Dual-Track Execution ​

Legacy In-Process ​

Sandboxed Subprocess (Recommended) ​

5.2 AST Allowlist (src/broker/workshop/sandbox/ast_guard.py) ​

5.3 Worker Resource Control (src/broker/workshop/sandbox/host.py) ​

5.4 Key Source Navigation ​

6. Data Consistency ​

6.1 Storage ​

6.2 Resume Consistency (same principle as flow resume) ​

7. Production Relay Topology ​

8. Event Bus (src/broker/workshop/event_bus.py) ​

9. Cross-Cluster Federation (Subsystem D) ​

9.1 Hooks Already in Place ​

9.2 Not Yet Built ​

10. TODO List ​

B App Container ​

C Distribution & Trust ​

D Cross-Cluster Federation ​

Reference Source Code ​