# Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

Page: https://stenobird.com/podcast/daily-paper-cast-7079649/memory-decoder-at-scale-a-pretrained-parametric-long-term-memory
Text version: https://stenobird.com/podcast/daily-paper-cast-7079649/memory-decoder-at-scale-a-pretrained-parametric-long-term-memory.md
Podcast: [Daily Paper Cast](https://stenobird.com/podcast/daily-paper-cast-7079649)
Published: 2026-08-01T04:02:06+00:00
Episode link: https://share.transistor.fm/s/758c95f9
Audio file: https://media.transistor.fm/758c95f9/aa436fb6.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/memory-decoder-at-scale-a-pretrained-parametric-long-term-memory
Duration seconds: 1343

## Resource

🤗 Upvotes: 48 | cs.CL Authors: Rubin Wei, Jiaqi Cao, Jiarui Wang, Junming Zhang, Qipeng Guo, Bowen Zhou, Zhouhan Lin Title: Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory Arxiv: http://arxiv.org/abs/2607.27919v1 Abstract: Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/memory-decoder-at-scale-a-pretrained-parametric-long-term-memory/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/daily-paper-cast-7079649/memory-decoder-at-scale-a-pretrained-parametric-long-term-memory.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.