# Speculative Decoding and Efficient LLM Inference with Chris Lott - #717

Page: https://stenobird.com/podcast/twiml-ai-podcast/speculative-decoding-and-efficient-llm-inference-with-chris-lott-717
Text version: https://stenobird.com/podcast/twiml-ai-podcast/speculative-decoding-and-efficient-llm-inference-with-chris-lott-717.md
Podcast: [The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)](https://stenobird.com/podcast/twiml-ai-podcast)
Published: 2025-02-04T07:23:33+00:00
Episode link: https://twimlai.com/podcast/twimlai/speculative-decoding-and-efficient-llm-inference/
Audio file: https://pscrb.fm/rss/p/traffic.megaphone.fm/MLN3928866342.mp3?updated=1738618160
Processing state: failed
JSON: https://stenobird.com/v1/public/podcasts/twiml-ai-podcast/episodes/speculative-decoding-and-efficient-llm-inference-with-chris-lott-717
Duration seconds: 4590

## Resource

Today, we're joined by Chris Lott, senior director of engineering at Qualcomm AI Research to discuss accelerating large language model inference. We explore the challenges presented by the LLM encoding and decoding (aka generation) and how these interact with various hardware constraints such as FLOPS, memory footprint and memory bandwidth to limit key inference metrics such as time-to-first-token, tokens per second, and tokens per joule. We then dig into a variety of techniques that can be used to accelerate inference such as KV compression, quantization, pruning, speculative decoding, and leveraging small language models (SLMs). We also discuss future directions for enabling on-device agentic experiences such as parallel generation and software tools like Qualcomm AI Orchestrator. The complete show notes for this episode can be found at https://twimlai.com/go/717.

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/twiml-ai-podcast/episodes/speculative-decoding-and-efficient-llm-inference-with-chris-lott-717/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/twiml-ai-podcast/speculative-decoding-and-efficient-llm-inference-with-chris-lott-717.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.