# Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

Page: https://stenobird.com/podcast/daily-paper-cast-7079649/training-long-context-vision-language-models-effectively-with-generalization-beyond-128k-context
Text version: https://stenobird.com/podcast/daily-paper-cast-7079649/training-long-context-vision-language-models-effectively-with-generalization-beyond-128k-context.md
Podcast: [Daily Paper Cast](https://stenobird.com/podcast/daily-paper-cast-7079649)
Published: 2026-05-15T05:01:15+00:00
Episode link: https://share.transistor.fm/s/fab16fc9
Audio file: https://media.transistor.fm/fab16fc9/fcfb5007.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/training-long-context-vision-language-models-effectively-with-generalization-beyond-128k-context
Duration seconds: 1385

## Resource

🤗 Upvotes: 75 | cs.CV Authors: Zhaowei Wang, Lishu Luo, Haodong Duan, Weiwei Liu, Sijin Wu, Ji Luo, Shen Yan, Shuai Peng, Sihang Yuan, Chaoyi Huang, Yi Lin, Yangqiu Song Title: Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context Arxiv: http://arxiv.org/abs/2605.13831v1 Abstract: Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video analysis, and multi-turn tool use in agentic workflows. Yet practical training recipes remain insufficiently explored, particularly for designing and balancing long-context data mixtures. In this work, we present a systematic study of long-context continued pre-training for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. We first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, our ablations further yield three key findings: i) for sequence-length distribution, balanced data outperforms target-length-focused data (e.g., 128K), suggesting that long-context ability requires generalizable key-information retrieval across various lengths and positions; ii) retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity; and iii) pure long-document VQA largely preserves short-context capabilities, suggesting that instruction-formatted long data reduces the need for short-data mixing. Based on these findings, we introduce MMProLong, obtained by long-context continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. MMProLong improves long-document VQA scores by 7.1% and maintains strong performa…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/training-long-context-vision-language-models-effectively-with-generalization-beyond-128k-context/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/daily-paper-cast-7079649/training-long-context-vision-language-models-effectively-with-generalization-beyond-128k-context.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.