Episode
Language Models Can Control Their Own Attention
- Podcast
- Daily Paper Cast
- Published
- Sep 3, 2026
- Duration seconds
- 1259
- Processing state
not_requested- Canonical source
- https://share.transistor.fm/s/c6f3d073
Actions
POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/language-models-can-control-their-own-attention/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/daily-paper-cast-7079649/language-models-can-control-their-own-attention.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
🤗 Upvotes: 37 | cs.CL, cs.AI, cs.LG Authors: Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos Title: Language Models Can Control Their Own Attention Arxiv: http://arxiv.org/abs/2609.02737v1 Abstract: Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: (full context), (a specific region), and (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.