Episode
Dynamic Token Merging for Efficient Byte-level Language Models with Julie Kallini - #724
- Published
- Mar 24, 2025
- Duration seconds
- 3032
- Processing state
failed
Actions
POST https://stenobird.com/v1/public/podcasts/twiml-ai-podcast/episodes/dynamic-token-merging-for-efficient-byte-level-language-models-with-julie-kallini-724/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/twiml-ai-podcast/dynamic-token-merging-for-efficient-byte-level-language-models-with-julie-kallini-724.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
Today, we're joined by Julie Kallini, PhD student at Stanford University to discuss her recent papers, “MrT5: Dynamic Token Merging for Efficient Byte-level Language Models” and “Mission: Impossible Language Models.” For the MrT5 paper, we explore the importance and failings of tokenization in large language models—including inefficient compression rates for under-resourced languages—and dig into byte-level modeling as an alternative. We discuss the architecture of MrT5, its ability to learn language-specific compression rates, its performance on multilingual benchmarks and character-level manipulation tasks, and its performance and efficiency. For the “Mission: Impossible Language Models” paper, we review the core idea behind the research, the definition and creation of impossible languages, the creation of impossible language training datasets, and explore the bias of language model architectures towards natural language. The complete show notes for this episode can be found at https://twimlai.com/go/724.