Episode
Weak-to-Strong Generalization via Direct On-Policy Distillation
- Podcast
- Daily Paper Cast
- Published
- Jul 15, 2026
- Duration seconds
- 1182
- Processing state
not_requested- Canonical source
- https://share.transistor.fm/s/c2959ca2
Actions
POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/weak-to-strong-generalization-via-direct-on-policy-distillation/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/daily-paper-cast-7079649/weak-to-strong-generalization-via-direct-on-policy-distillation.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
🤗 Upvotes: 100 | cs.LG, cs.AI, cs.CL Authors: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou Title: Weak-to-Strong Generalization via Direct On-Policy Distillation Arxiv: http://arxiv.org/abs/2607.05394v2 Abstract: Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results…