Native-Streaming-Avatar
In-Memory Audio-Driven Conversational Avatar Pipeline with Sub-5s Time-to-First-Frame
A four-stage conversational AI avatar pipeline (Faster-Whisper ASR → Qwen2.5-7B → Edge-TTS → Wav2Lip) engineered with native in-memory chunk streaming (Stream V2) to eliminate disk I/O bottlenecks, achieving <5s Time-to-First-Frame and >15 FPS video generation.
The Technical Bottleneck
End-to-end talking-head avatar pipelines traditionally serialize intermediate audio and video segments to disk between ASR, LLM generation, TTS synthesis, and lip-sync rendering, causing high Time-to-First-Frame (TTFF) latency and severe I/O fragmentation.
Why Existing Baselines Fail
For interactive virtual assistants and clinical/educational training avatars, users perceive delays above 5 seconds as broken conversation. Eliminating file-system roundtrips via sentence-level in-memory streaming enables overlapped execution across all four neural stages.
Core Formulation & Insight
We architected Stream V2, an in-memory streaming pipeline that transcribes user speech with Faster-Whisper, streams token chunks from Qwen2.5-7B segmented at sentence boundaries, synthesizes PCM audio buffers in memory via Edge-TTS, and feeds contiguous audio-visual tensors directly into Wav2Lip for real-time lip-synced frame streaming.
Architectural Design
Four-stage overlapped streaming pipeline with zero-disk-I/O in-memory buffer passing between ASR, LLM, TTS, and Wav2Lip rendering.
1. Low-Latency Speech Recognition (Faster-Whisper)
Transcribes incoming user audio streams with CTranslate2-accelerated Faster-Whisper.
2. Sentence-Chunked LLM Generation (Qwen2.5-7B)
Streams response tokens from Qwen2.5-7B and flushes complete clause/sentence chunks immediately to the TTS worker.
3. In-Memory Audio Synthesis (Edge-TTS)
Generates speech audio waveforms directly into memory buffers without writing temporary MP3/WAV files to disk.
4. Native Stream V2 Lip-Sync Synthesis (Wav2Lip + SyncNet Eval)
Renders lip-synced facial video chunks in memory at >15 FPS and evaluates audio-visual alignment using SyncNet metrics.
Technical Stack & Systems Engineering
Implementation Highlights
- →Eliminated intermediate disk I/O and file fragmentation between TTS and Wav2Lip via native in-memory tensor streaming.
- →Integrated SyncNet lip-sync confidence and offset benchmarking alongside latency profiling.
Datasets & Protocol
Datasets & Corpora
- Conversational Avatar Prompt & Audio-Visual Synchronization Benchmark
Evaluation Protocol
Benchmarked Stream V1 (file-backed sequential pipeline) against Stream V2 (in-memory chunked pipeline) measuring Time-to-First-Frame (TTFF), rendering throughput (FPS), and SyncNet synchronization scores.
Tracked Evaluation Metrics
Reported sub-5-second Time-to-First-Frame (<5s TTFF) and >15 FPS real-time generation throughput via native in-memory chunk streaming.
Reported by Stream V2 native in-memory pipeline
Sustained audio-driven Wav2Lip generation rate
Faster-Whisper → Qwen2.5-7B → Edge-TTS → Wav2Lip
Empirical Analysis & Interpretation
Overlapping sentence-level LLM generation with in-memory TTS and Wav2Lip inference removes the dominant serialization bottleneck in conversational avatar systems.
What This System Contributes
- 01Designed and implemented the Stream V2 in-memory streaming architecture and SyncNet evaluation harness.
Boundary Conditions & Future Work
- ·Throughput and TTFF depend on GPU VRAM availability for co-locating Qwen2.5-7B and Wav2Lip.