# LLM-D, with Clayton Coleman and Rob Shaw

Page: https://stenobird.com/podcast/kubernetes-podcast-from-google/llm-d-with-clayton-coleman-and-rob-shaw
Text version: https://stenobird.com/podcast/kubernetes-podcast-from-google/llm-d-with-clayton-coleman-and-rob-shaw.md
Podcast: [Kubernetes Podcast from Google](https://stenobird.com/podcast/kubernetes-podcast-from-google)
Published: 2025-08-20T13:24:00+00:00
Episode link: https://e780d51f-f115-44a6-8252-aed9216bb521.libsyn.com/llm-d-with-clayton-coleman-and-rob-shaw
Audio file: https://traffic.libsyn.com/secure/e780d51f-f115-44a6-8252-aed9216bb521/KPOD258.mp3?dest-id=3486674
Processing state: processed
JSON: https://stenobird.com/v1/public/podcasts/kubernetes-podcast-from-google/episodes/llm-d-with-clayton-coleman-and-rob-shaw
Duration seconds: 3150

## Resource

Large Language Models (LLMs) introduce unique networking and compute challenges that traditional microservice load balancing cannot handle. This discussion explores how specialized Kubernetes patterns like LLM-D and inference gateways optimize for auto-regressive token generation and massive data throughput.

## Highlights
- Main idea: LLM workloads are fundamentally different from web apps because auto-regressive generation requires iterative passes and handles highly variable request lengths
- Practical takeaway: Implementing intelligent inference scheduling and specialized gateways can optimize the expensive transition between prefill and decode phases
- Failure mode: Standard random load balancing fails to account for the massive network throughput (up to 10GB) required by long-context prompts
- Main idea: The industry is moving toward standardized, OpenAI-compatible APIs to decouple the application layer from the underlying model server
- Practical takeaway: Leveraging ecosystem-wide optimizations like vLLM and LeaderWorkerSet (LWS) creates a 'well-lit path' for scaling AI/ML on Kubernetes

## Topics

Kubernetes, Large Language Models, Inference Gateway, vLLM, Machine Learning Operations, Load Balancing, Cloud Native, GPU Orchestration

## Chapters
- 1:00 — Kubernetes 1.34 and CNCF News: Updates on the upcoming Kubernetes 1.34 release and recognition of top CNCF open-source projects.
- 5:00 — The Uniqueness of LLM Workloads: Why LLM inference requires different load balancing strategies than traditional microservices due to variable request sizes.
- 8:50 — The Challenge of Auto-regressive Models: Understanding how the iterative nature of token generation impacts compute and state management.
- 13:15 — Scaling Large Models Across Hardware: Addressing the complexities of running models that are larger than a single hardware unit's capacity.
- 17:05 — Intelligent Inference Scheduling: An introduction to deployment patterns that use Gateway logic to optimize inference workloads.
- 21:00 — Standardizing Kubernetes APIs: The importance of defining consistent APIs between components like vLLM and the inference gateway.
- 28:40 — Optimizing the Network and Data Path: Managing the massive data throughput required when pushing large prompts across the network.
- 32:25 — The Future of Model Abstraction: How OpenAI-compatible APIs allow developers to swap model servers without changing application code.

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/kubernetes-podcast-from-google/episodes/llm-d-with-clayton-coleman-and-rob-shaw/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/kubernetes-podcast-from-google/llm-d-with-clayton-coleman-and-rob-shaw.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.