# Dynamic Token Merging for Efficient Byte-level Language Models with Julie Kallini - #724

Page: https://stenobird.com/podcast/twiml-ai-podcast/dynamic-token-merging-for-efficient-byte-level-language-models-with-julie-kallini-724
Text version: https://stenobird.com/podcast/twiml-ai-podcast/dynamic-token-merging-for-efficient-byte-level-language-models-with-julie-kallini-724.md
Podcast: [The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)](https://stenobird.com/podcast/twiml-ai-podcast)
Published: 2025-03-24T19:42:00+00:00
Episode link: https://twimlai.com/podcast/twimlai/dynamic-token-merging-for-efficient-byte-level-language-models/
Audio file: https://pscrb.fm/rss/p/traffic.megaphone.fm/MLN6993632573.mp3?updated=1742845563
Processing state: failed
JSON: https://stenobird.com/v1/public/podcasts/twiml-ai-podcast/episodes/dynamic-token-merging-for-efficient-byte-level-language-models-with-julie-kallini-724
Duration seconds: 3032

## Resource

Today, we're joined by Julie Kallini, PhD student at Stanford University to discuss her recent papers, “MrT5: Dynamic Token Merging for Efficient Byte-level Language Models” and “Mission: Impossible Language Models.” For the MrT5 paper, we explore the importance and failings of tokenization in large language models—including inefficient compression rates for under-resourced languages—and dig into byte-level modeling as an alternative. We discuss the architecture of MrT5, its ability to learn language-specific compression rates, its performance on multilingual benchmarks and character-level manipulation tasks, and its performance and efficiency. For the “Mission: Impossible Language Models” paper, we review the core idea behind the research, the definition and creation of impossible languages, the creation of impossible language training datasets, and explore the bias of language model architectures towards natural language. The complete show notes for this episode can be found at https://twimlai.com/go/724.

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/twiml-ai-podcast/episodes/dynamic-token-merging-for-efficient-byte-level-language-models-with-julie-kallini-724/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/twiml-ai-podcast/dynamic-token-merging-for-efficient-byte-level-language-models-with-julie-kallini-724.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.