Episode
Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context
- Podcast
- Daily Paper Cast
- Published
- May 15, 2026
- Duration seconds
- 1385
- Processing state
not_requested- Canonical source
- https://share.transistor.fm/s/fab16fc9
Actions
POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/training-long-context-vision-language-models-effectively-with-generalization-beyond-128k-context/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/daily-paper-cast-7079649/training-long-context-vision-language-models-effectively-with-generalization-beyond-128k-context.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
🤗 Upvotes: 75 | cs.CV Authors: Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu, Sijin Wu, Ji Luo, Shen Yan, Shuai Peng, Sihang Yuan, Chaoyi Huang, Yi Lin, Yangqiu Song Title: Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context Arxiv: http://arxiv.org/abs/2605.13831v1 Abstract: Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video analysis, and multi-turn tool use in agentic workflows. Yet practical training recipes remain insufficiently explored, particularly for designing and balancing long-context data mixtures. In this work, we present a systematic study of long-context continued pre-training for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. We first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, our ablations further yield three key findings: i) for sequence-length distribution, balanced data outperforms target-length-focused data (e.g., 128K), suggesting that long-context ability requires generalizable key-information retrieval across various lengths and positions; ii) retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity; and iii) pure long-document VQA largely preserves short-context capabilities, suggesting that instruction-formatted long data reduces the need for short-data mixing. Based on these findings, we introduce MMProLong, obtained by long-context continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. MMProLong improves long-document VQA scores by 7.1% and maintains strong performa…