Hallo-Live: Real-Time Text-Driven Audiovisual Digital Humans

·Toolin Editorial Team

An open-source real-time digital human generation system: text input synchronously generates talking video and speech, with 20.38 FPS throughput and 0.94-second end-to-end latency

Hallo-Live: Real-Time Text-Driven Audiovisual Digital Humans

Text-driven audiovisual digital humans are moving from "can generate" to "can interact in real time." Hallo-Live, a project from Shanghai Innovation Institute, Fudan University, and other institutions, achieves 20.38 FPS throughput and 0.94-second end-to-end latency on two NVIDIA H200 GPUs, while keeping visual quality and audio-visual sync close to the teacher model's level.

Paper: arxiv.org/abs/2604.23632 Code: github.com/fudan-generative-vision/Hallo-Live

What Is Hallo-Live

Traditional audio-driven digital humans only need lip sync, but a text-driven approach has to do two things at once: first "understand" the character, scene, and tone in the text, then synchronously generate the matching talking video and speech. Lip shapes, pronunciation, expressions, and even upper-body movements all have to land on the same timeline.

Against the teacher model Ovi, Hallo-Live's key metrics:

MetricHallo-LiveImprovement
Throughput20.38 FPS16.0x increase
End-to-end latency0.94 secondsdown 99.3%

It supports anime styles, photorealistic characters, and multi-person scenes, and is already open-sourced on GitHub.

Core Technology

Asynchronous Dual-Stream Diffusion

Hallo-Live's training has two stages:

  • Stage 1 -- Dual-Stream ODE Init: the model takes in audio-video blocks at different noise levels simultaneously, training a dual-stream DiT with unimodal and cross-modal Block-Causal Masks
  • Stage 2 -- Self-Rollout + Dual-Stream DMD: the student model autoregressively generates full audio-video from the audio-video KV Cache, then audio-video-sync-related reward distillation is introduced to distill it into a few-step model

Overall architecture

Causal Fusion Block

This is the core unit of the dual-stream DiT. The video stream and audio stream each do unimodal Block-Causal Self-Attention first, then text conditioning is injected, and they subsequently exchange information through cross-modal Block-Causal Cross-Attention — completing audio-video fusion under streaming generation.

Causal Fusion Block architecture

Future-Expanding Attention

In real speech, lip movements often arrive ahead of the sound (the coarticulation phenomenon). Strictly causal block-level attention can't see the "near-future" speech information, which leads to unnatural lip shapes.

Hallo-Live makes the video-to-audio cross-modal attention "asymmetric": video focuses on the current block, but the audio key-value range extends a small look-ahead window further forward — effectively giving the video stream a short-term "preview area."

Future-Expanding Attention mechanism

This future audio block isn't a final output but a temporary transition block, so audio quality isn't degraded.

Human-Preference-Guided Distillation

Few-step distillation speeds things up but tends to bring "averaged" degradation -- blurrier video texture, more mechanical speech, drifting audio-visual alignment. Hallo-Live introduces audio, video, and audio-video-sync-related rewards to weight the dual-stream DMD loss, maintaining generation quality while accelerating.

Application Scenarios

  • Virtual streamers / digital human livestreams: real-time text-driven, ideal for interactive livestream scenarios
  • Customer service digital humans: generate voice and video replies in real time from text input
  • Content creation: batch-generate digital human videos with speech
  • Education/training: generate digital human instructors to deliver course explanations

Technical Requirements

  • Hardware: two NVIDIA H200 GPUs (the paper's experimental environment)
  • Input: text + a reference character image
  • Output: synchronized talking video + speech
  • Latency: 0.94 seconds end to end

The project is open-sourced on GitHub, well suited for researchers and developers working on digital human interaction and real-time streaming generation.

References:

Related articles

A Field-Tested Method for Managing a 700,000-Line Codebase with Claude Code
AI Tutorials

A Field-Tested Method for Managing a 700,000-Line Codebase with Claude Code

Build a three-layer context system with CLAUDE.md, Skills, and MCP so the AI truly understands your codebase — with a comparison against OpenAI's Symphony parallel-orchestration approach

Toolin Editorial Team
Neat.Skill: An Open-Source Tool That Makes AI Agents Smarter the More You Use Them
AI Tutorials

Neat.Skill: An Open-Source Tool That Makes AI Agents Smarter the More You Use Them

An open-source Agent documentation maintenance tool that automatically reviews and iteratively updates a project's documentation system, supporting four platforms: Claude Code, Codex, OpenCode, and OpenClaw.

Toolin Editorial Team
Build a Claude AI Desktop Pet in 10 Minutes: An Open-Source Hardware Tutorial
AI Tutorials

Build a Claude AI Desktop Pet in 10 Minutes: An Open-Source Hardware Tutorial

Anthropic open-sourced Claude-Desktop-Buddy: build an AI desktop pet on a Shenzhen-made M5StickC Plus dev board that shows working status, approves operations, and ships with 18 built-in animations.

Toolin Editorial Team
DeepSeek V4 Practical Guide: Costs Just One-Tenth of GPT-5.5
AI Tutorials

DeepSeek V4 Practical Guide: Costs Just One-Tenth of GPT-5.5

DeepSeek V4 has been released open-source under the MIT license, with a 1M-token context window and output at just $3.48 per million tokens. Real-world tests show coding costs dropping 83% — this is the best window to migrate.

Toolin Editorial Team
GPT Image 2: OpenAI's First Image Model That Thinks
AI Products

GPT Image 2: OpenAI's First Image Model That Thinks

GPT Image 2 topped the Arena text-to-image leaderboard on launch, leading the runner-up by 241 points, with native thinking ability, 99% text rendering accuracy, and API images at just $0.21.

Toolin Editorial Team