Episode
How Much Do Language Models Memorize?
- Podcast
- Best AI papers explained
- Published
- Jul 9, 2026
- Duration seconds
- 1432
- Processing state
not_requested- Canonical source
- https://podcasters.spotify.com/pod/show/ehwkang/episodes/How-Much-Do-Language-Models-Memorize-e3lrl61
Actions
POST https://stenobird.com/v1/public/podcasts/best-ai-papers-explained-7258006/episodes/how-much-do-language-models-memorize/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/best-ai-papers-explained-7258006/how-much-do-language-models-memorize.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
This research paper investigates language model capacity by introducing a new method to measure how much a model truly memorizes versus what it generalizes. The authors distinguish between unintended memorization, which is specific data storage, and generalization, which is the understanding of broader patterns. By testing the GPT family, they determine these models possess a storage capacity of approximately 3.6 bits-per-parameter. The study reveals that the double descent phenomenon occurs specifically when a dataset's size surpasses the model's total bit capacity. Furthermore, the researchers established scaling laws to predict the success of membership inference attacks, which identify if a specific datapoint was used in training. Their findings suggest that modern models are trained on so much data that standard membership inference is increasingly difficult for average samples.