# Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models

Page: https://stenobird.com/podcast/daily-paper-cast-7079649/flow-dppo-divergence-proximal-policy-optimization-for-flow-matching-models
Text version: https://stenobird.com/podcast/daily-paper-cast-7079649/flow-dppo-divergence-proximal-policy-optimization-for-flow-matching-models.md
Podcast: [Daily Paper Cast](https://stenobird.com/podcast/daily-paper-cast-7079649)
Published: 2026-06-11T04:27:36+00:00
Episode link: https://share.transistor.fm/s/3c9dc9d2
Audio file: https://media.transistor.fm/3c9dc9d2/4da043db.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/flow-dppo-divergence-proximal-policy-optimization-for-flow-matching-models
Duration seconds: 1300

## Resource

🤗 Upvotes: 35 | cs.LG Authors: Bowen Ping, Xiangxin Zhou, Penghui Qi, Minnan Luo, Liefeng Bo, Tianyu Pang Title: Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models Arxiv: http://arxiv.org/abs/2606.11025v1 Abstract: Recent work has demonstrated that online reinforcement learning (RL) can substantially improve the quality and alignment of flow matching models for image and video generation. Methods such as Flow-GRPO and CPS cast the denoising process as a Markov Decision Process and apply PPO-style ratio clipping to enforce a trust region. However, we argue that ratio clipping is structurally ill-suited for flow models: the probability ratio between new and old policies is a noisy, single-sample estimate of the true policy divergence, leading to over-constraining in some regions of the trajectory and under-constraining in others. We propose Flow-DPPO (Flow Divergence Proximal Policy Optimization), which replaces ratio clipping with a divergence proximal constraint. A key observation is that the per-step policy in flow models is Gaussian, enabling exact and cheap computation of the KL divergence between old and new policies. Flow-DPPO employs an asymmetric divergence mask that blocks gradient updates only when they simultaneously move away from the trusted region and violate the divergence threshold. Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency, alleviates catastrophic forgetting, promotes balanced multi-objective optimization, and enables stable multi-epoch training where ratio clipping degrades. Code and models are available at https://github.com/Tencent-Hunyuan/UniRL/tree/main/FlowDPPO.

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/flow-dppo-divergence-proximal-policy-optimization-for-flow-matching-models/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/daily-paper-cast-7079649/flow-dppo-divergence-proximal-policy-optimization-for-flow-matching-models.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.