AI long-video generation has always faced an "impossible triangle": long duration, high consistency, and fast speed — you seemingly can't have all three. The same character looks different from one shot to the next, the speaker's voice wobbles up and down, and rendering still takes half a day to wait out. JD's newly open-sourced JoyAI-Echo is here to break these pain points one by one.

What Is JoyAI-Echo

JoyAI-Echo is JD's first open-sourced long audio-video generation framework, supporting minute-level narrative video generation with character appearance and voice tone staying consistent across multiple shots. Code and model weights are fully open, so developers can build on top of them and fine-tune.

• GitHub: https://github.com/jd-opensource/JoyAI-Echo
• Hugging Face: https://huggingface.co/jdopensource/JoyAI-Echo
• Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-LongVideo-Page/

Core Technical Breakthroughs

1. Cross-Modal Audio-Video Memory Bank: Solving the "Face-Changing" Problem

Traditional models lack memory of prior content when generating shot by shot, starting over each time as if stricken with amnesia. JoyAI-Echo ships with a dedicated memory bank that continuously stores and precisely recalls characters' visual and auditory features. Across 5 minutes of multi-shot generation, this memory bank acts like the "character file" in a director's hands, guaranteeing consistent output on every call.

2. Memory-Driven Post-Training: 7.5x Speedup

JoyAI-Echo designed a three-stage post-training pipeline: SFT -> cross-modal RLHF -> distribution matching distillation (DMD). DMD compresses the multi-step diffusion teacher-student distillation into 8-step fast inference, delivering roughly a 7.5x inference speedup and turning long videos from "waiting half a day" into "footage in seconds."

3. Director Agent: Conversational Editing

You can tell it what you want in natural language — say, "swap the cafe background in scene three for a library." It automatically breaks down the request, generates the video, and checks the results. Only the unsatisfactory parts get regenerated shot by shot; the whole video doesn't have to be redone.

4. Lightweight Real-Time Super-Resolution: From 720p to HD

The bundled real-time super-resolution module upscales the native 720p video to as high as 1472x2560 resolution with almost no added latency.

Benchmark Data

In a rigorous evaluation across 100 independent story scripts totaling 3000 storyboard shots:

Use Cases

• Virtual anime and story creation: direct AI in natural language to generate coherent anime episodes
• Digital-human livestreams and short dramas: keep voice, lip movements, and expressions consistent over long stretches
• Brand marketing content: change a line of dialogue or a few local shots to generate multiple video versions
• Film storyboard pre-visualization: quickly generate preview videos to validate the shot language
• Educational courseware and game animation: dynamically generate coherent story animations

How to Get It

Code and model weights are fully open source; head to the GitHub repo jd-opensource/JoyAI-Echo to get them.