Episode
Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models
- Podcast
- Daily Paper Cast
- Published
- Jun 11, 2026
- Duration seconds
- 1300
- Processing state
not_requested- Canonical source
- https://share.transistor.fm/s/3c9dc9d2
Actions
POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/flow-dppo-divergence-proximal-policy-optimization-for-flow-matching-models/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/daily-paper-cast-7079649/flow-dppo-divergence-proximal-policy-optimization-for-flow-matching-models.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
🤗 Upvotes: 35 | cs.LG Authors: Bowen Ping, Xiangxin Zhou, Penghui Qi, Minnan Luo, Liefeng Bo, Tianyu Pang Title: Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models Arxiv: http://arxiv.org/abs/2606.11025v1 Abstract: Recent work has demonstrated that online reinforcement learning (RL) can substantially improve the quality and alignment of flow matching models for image and video generation. Methods such as Flow-GRPO and CPS cast the denoising process as a Markov Decision Process and apply PPO-style ratio clipping to enforce a trust region. However, we argue that ratio clipping is structurally ill-suited for flow models: the probability ratio between new and old policies is a noisy, single-sample estimate of the true policy divergence, leading to over-constraining in some regions of the trajectory and under-constraining in others. We propose Flow-DPPO (Flow Divergence Proximal Policy Optimization), which replaces ratio clipping with a divergence proximal constraint. A key observation is that the per-step policy in flow models is Gaussian, enabling exact and cheap computation of the KL divergence between old and new policies. Flow-DPPO employs an asymmetric divergence mask that blocks gradient updates only when they simultaneously move away from the trusted region and violate the divergence threshold. Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency, alleviates catastrophic forgetting, promotes balanced multi-objective optimization, and enables stable multi-epoch training where ratio clipping degrades. Code and models are available at https://github.com/Tencent-Hunyuan/UniRL/tree/main/FlowDPPO.