07 · Digital Human
Seventh installment of the architecture deep-dive: the Digital Human rendering pipeline, speech synthesis (GPT-SoVITS), backend abstractions, and VRAM gating. Corresponds to
ARCHITECTURE.md§8.
§8.1 System Overview
Digital Human = Voice Cloning (GPT-SoVITS) + Animation Driving (anime backend) + Streaming Multiplexing
text gen ──► tts_worker ──► audio_q ──┬─► backend_worker ──► frame_q ──┐
│ │
└─► mux_worker ◄─────────────────┘
│
▼
StreamChunkThree asyncio tasks communicate via bounded asyncio.Queues. Cancellation: closing the consumer iterator → finally blocks cancel all workers.
§8.2 DigitalHumanSession (src/digital_human/session.py)
8.2.1 Construction
python
class DigitalHumanSession:
def __init__(
self,
config: Optional[DigitalHumanConfig] = None,
library: Optional[CharacterLibrary] = None,
):
self.config = config or DigitalHumanConfig()
self.tts = StreamingTTS(self.config)
self.library = library or CharacterLibrary()
self._backend: Optional[Backend] = None
self._source: Optional[Source] = None8.2.2 Lifecycle
python
async def prepare(
self,
*,
voice_ref: str, # Path to voice.wav
voice_ref_text: str = "", # Reference text
character_id: Optional[str] = None,
backend: Optional[str] = None,
) -> None:
"""Load TTS + select backend.
When backend is not specified, select_backend() auto-selects based on inputs.
"""
async def stream(
self,
text_source: AsyncIterator[str],
) -> AsyncIterator[StreamChunk]:
"""Streaming generation. text_source is an async token iterator.
Produces StreamChunk = (pcm_bytes, video_frame).
"""8.2.3 TTS Liveness Check
python
@property
def _tts_loaded(self) -> bool:
"""True when TTS subprocess is admitted AND actually alive.
The old version only checked self.tts._runner is not None → marked as
"loaded" even after the subprocess died → the user's first _synth call
hit a 5-15s cold start (exactly the dead silence that eager load was
meant to prevent).
Now polls the subprocess; if dead → treated as not-loaded → next load()
re-spawns it.
"""§8.3 DigitalHumanConfig (src/digital_human/config.py)
python
@dataclass
class DigitalHumanConfig:
# Paths
gpt_sovits_dir: str = _default_gpt_sovits_dir() # GPT-SoVITS bundle root
gpt_sovits_version: str = "v2" # v1/v2/v3
gpt_sovits_checkpoint: Optional[str] = None # .ckpt/.pth weights
tha3_dir: str = _default_tha3_dir() # THA3 demo checkout
tha3_variant: str = "separable_float" # THA3 variant
# Streaming parameters
audio_chunk_ms: int = 200 # Target PCM chunk size in ms
# Image processing
anime_auto_bust_up: bool = True # rembg + bust-up reconstructionEnvironment Variables:
GPT_SOVITS_DIR— Override GPT-SoVITS pathTHA3_DIR— Override THA3 path
§8.4 Backend Abstraction
8.4.1 Backend Catalog
| Backend | Directory | Description |
|---|---|---|
anime_morph | backends/anime_morph/ | Primary backend. Paired lip-sync atlas + explicit mask blend |
anime_dynamic | backends/anime_dynamic/ | THA3 single-image driven (fallback) |
anime_pretrained | backends/anime_pretrained/ | Live2D pretrained characters |
anime_seethrough | backends/anime_seethrough/ | See-through mode |
realphoto_dynamic | backends/realphoto_dynamic/ | Real photo driven (deprecated) |
8.4.2 Backend Base Class
python
# src/digital_human/backends/base.py
class Backend:
"""Abstract base class. Each backend implements:
- prepare(spec: PrepareSpec) → Source
- drive(audio_chunk) → list[MotionFrame]
- render(frame) → np.ndarray
"""
@dataclass
class PrepareSpec:
voice_ref: str
voice_ref_text: str
character_id: Optional[str]
source_image: Optional[str]
@dataclass
class Source:
"""Backend resource handle."""8.4.3 Backend Selection
python
# src/digital_human/backends/registry.py
def select_backend(hint: Optional[str] = None, **kwargs) -> Backend:
"""Select a backend based on the hint and available resources.
hint=None → defaults to anime_morph.
"""8.4.4 Project Intent
The project is intentionally anime-first. Older LivePortrait / JoyVASA real-photo experiments are deprecated because they did not produce reliable Chinese lip-sync and required far more weights/runtime complexity than the current mouth-atlas pipeline.
§8.5 Streaming Parameters
8.5.1 audio_chunk_ms = 200
Why 200ms (old value was 500ms):
The mux worker pairs each audio chunk with
audio_chunk_ms × fps / 1000 rendered frames to produce a StreamChunk.
500ms × 25fps = 12 frames/chunk
→ seethrough GPU render ~30-50ms/frame
→ 360-600ms wall-clock wait per chunk
→ first chunk arrives ~500ms later than TTS completion
200ms × 25fps = 5 frames/chunk
→ ~150-250ms render budget
→ first chunk arrives at the renderer almost as soon as TTS completes
Cost: more WS messages per second. Benefit: first-chunk latency drops dramatically.8.5.2 StreamChunk Structure
python
# src/digital_human/types.py
@dataclass
class StreamChunk:
audio_pcm: bytes # PCM audio data
video_frame: np.ndarray # Video frame
sample_rate: int # Sample rate
pts_ms: float # Presentation timestamp§8.6 Pipeline Submodules
src/digital_human/pipeline/
├── tts.py # StreamingTTS (GPT-SoVITS subprocess communication)
├── motion.py # Motion generation (audio → driving signals)
├── render.py # Rendering (driving signals → video frames)
├── emotion.py # Emotion classifier
├── theme.py # Theme/style
└── voice_ref_gen.py # Voice reference generation§8.7 VRAM Gating
8.7.1 GPU Tiers & Feature Matrix
| Tier | VRAM | Digital Human Support | Description |
|---|---|---|---|
mode0 | No GPU | ❌ | All API-only |
tier4 | ~4GB | ⚠️ Limited | Audio only, no video/emotion classification |
tier6 | ~6GB | ⚠️ Audio | Audio + basic video |
tier8 | ≥8GB | ✅ Full | Full features |
8.7.2 VRAM Priority
| Priority | Digital Human Component | Description |
|---|---|---|
| 1 — CRITICAL | TTS (while speaking), whisper (while listening) | Active user requests |
| 2 — IMPORTANT | bge embedding | Steady-state interaction |
| 3 — NICE | emotion tagger, avatar poser | Useful but can wait |
| 4 — BACKGROUND | training | Never user-visible |
8.7.3 Prewarm Strategy
The bridge prewarms the "most recent agent's DH" at startup:
python
# src/bridge/app.py at module level
# Prewarm embedder / rembg / rewriter / most-recent-agent's digital humanInteractive within moments of startup.
§8.8 Memory Integration
- DH sessions use the same
DialogueManager/ memory - Digital Human is just another "speaking shell"
- Memory is not isolated per DH
Assistant.send_message()is the entry point:
python
async def send_message(self, text_stream, *, ref_text="",
character_id=None, backend=None):
"""Stream reply as text+audio+video.
text_stream: async token iterator
voice_ref: {agent_path}/voice.wav
yield: (text_chunk, StreamChunk)
"""
session = DigitalHumanSession(DigitalHumanConfig())
await asyncio.to_thread(session.prepare,
voice_ref=voice_ref, voice_ref_text=ref_text,
character_id=character_id, backend=backend)
async for chunk in session.stream(_text_gen()):
yield chunk§8.9 CharacterLibrary
python
# src/digital_human/library.py
class CharacterLibrary:
"""Character library management.
get_default() → default character entry
list() → all available characters
"""Character data is stored in workspace/agents/agent_<id>/:
voice.wav— voice referenceclient_portraits/{key}.png— client portraits (white background →strip_white_fileto make transparent)
Next: 08-Security & Monitoring →