UniRL is a reinforcement learning post-training framework for multimodal generative models, open-sourced by Tencent Hunyuan's Pang Tianyu team. It solves a long-standing pain point: image diffusion models have one training pipeline, video generation has another standard, and VLMs and LLMs run on yet another tech stack. UniRL unifies all of them in a single framework.

What Problem It Solves

RL training for multimodal generative models faces four major challenges:

1. Different generation processes: LLMs process discrete token sequences, while image/video generation corresponds to denoising trajectories in a continuous latent space. A single rollout of a unified model also mixes token generation with latent denoising.

2. Hard-to-stabilize system loop: rollout, log-prob replay, and policy updates span multiple models and backends; the training side must strictly reproduce the sampling side's conditions, noise, and timesteps, or a training-inference mismatch appears.

3. Heavier reward systems: multimodal rewards often depend on VLMs, OCR, aesthetic models, and video understanding models — not simple text rules.

4. Heavy VRAM pressure: intermediate artifacts are high-dimensional latents, noise, timesteps, and conditioning states, which scale up rapidly with resolution and frame count in video generation.

The result is an industry status quo of "one model, one set of training code", with developers wasting huge amounts of time on repeated engineering work.

UniRL's unified multimodal RL loop: rollout -> reward -> advantage -> train -> weight-sync.

Supported Models

UniRL offers the industry's broadest support for multimodal generative models:

Built-in Algorithms and Reward Systems

Algorithm Side

• Policy-gradient family: FlowGRPO, DanceGRPO, MixGRPO, LLM/VLM GRPO
• Forward-process family: DiffusionNFT (efficient training without a full SDE rollout)
• Tencent Hunyuan in-house:
  • Flow-DPPO: replaces PPO ratio clipping with a per-step KL-divergence proximal constraint, achieving more stable RL training of flow/diffusion models
  • DRPO: replaces hard clipping/masking with an advantage-weighted smooth policy-shift regularizer, providing continuous gradient correction even past the trust-region boundary

Reward Side

UniRL integrates many commonly used reward components:

• Rule/similarity: CLIPScore, GOT-OCR-2.0
• Preference/aesthetics: PickScore, HPSv2/HPSv3, ImageReward
• VLM-as-judge: UnifiedReward, GenEval2, WISE
• Video evaluation: VideoPickScore, VideoAlign

Core Design

UniRL is built around Ray worker groups, a Hydra flat recipe, composable training backends, and pluggable rollout engines as its core skeleton. It uses tracks to represent the generation trajectories carried through different stages: the AR stage is a TextSegment, the image generation stage is a LatentSegment, and different tracks connect through parent-child relationships.

This lets the chained flow of unified multimodal models like Bagel and HunyuanImage 3.0 (AR text thinking first, then DiT image generation) be represented naturally.

How to Use It

• GitHub repository: https://github.com/Tencent-Hunyuan/UniRL
• Official documentation: https://unirl-project.github.io/unirl/
• Flow-DPPO paper: https://arxiv.org/pdf/2606.11025
• DRPO paper: https://arxiv.org/pdf/2606.09821

The framework ships with comprehensive examples, making it easy to quickly launch experiments and reproduce algorithms. It is still under active iteration; rollout engine support will continue to expand and large-scale training performance will keep improving.