Stenobird

Episode

Multi-Cluster Orchestrator, with Nick Eberts and Jon Li

Podcast
Kubernetes Podcast from Google
Published
May 28, 2025
Duration seconds
1291
Processing state
processed
Canonical source
https://e780d51f-f115-44a6-8252-aed9216bb521.libsyn.com/multi-cluster-orchestrator-with-nick-eberts-and-jon-li
Audio
https://traffic.libsyn.com/secure/e780d51f-f115-44a6-8252-aed9216bb521/KPOD253.mp3?dest-id=3486674
JSON
/v1/public/podcasts/kubernetes-podcast-from-google/episodes/multi-cluster-orchestrator-with-nick-eberts-and-jon-li
Markdown
/podcast/kubernetes-podcast-from-google/multi-cluster-orchestrator-with-nick-eberts-and-jon-li.md

Actions

  • POST https://stenobird.com/v1/public/podcasts/kubernetes-podcast-from-google/episodes/multi-cluster-orchestrator-with-nick-eberts-and-jon-li/transcription-requests
    Idempotently request low-priority transcript generation for this episode.
  • GET https://stenobird.com/podcast/kubernetes-podcast-from-google/multi-cluster-orchestrator-with-nick-eberts-and-jon-li.md
    Read the agent-friendly Markdown representation of this episode resource.

Summary

The Multi-Cluster Orchestrator (MCO) solves the challenge of managing workloads across fragmented, non-uniform cloud resources. It provides a standardized way to scale AI inference engines from zero to one across multiple clusters based on real-time capacity and hardware availability.

Topics

  • Kubernetes
  • Multi-Cluster Orchestrator
  • AI Inference
  • GKE
  • Cloud Infrastructure
  • GPU Availability
  • Cluster Management
  • LLM Deployment

Highlights

  • Main idea: MCO addresses the shift from 'infinite' cloud capacity to a reality of hardware stockouts and non-uniform GPU availability
  • Practical takeaway: Use MCO to automate workload placement recommendations, allowing tools like Argo CD to scale workloads from zero to one in available regions
  • Technical detail: The system utilizes a normalized 'Cluster Profile' spec to unify cluster inventories across different providers and tools
  • Failure mode: Outdated cluster profiles can lead to MCO making recommendations based on stale capacity data
  • Practical takeaway: For AI inference, leverage custom metrics like KV cache utilization or queue depth to drive precise scaling decisions

Chapters

  1. 1:00 Kubernetes & GKE Updates: Brief overview of Kubernetes 1.33 availability in GKE Rapid channel and Kyverno 1.14.0 release.
  2. 2:35 Introduction to MCO: An introduction to the Multi-Cluster Orchestrator and the shift in cloud computing assumptions.
  3. 4:05 The Problem: Hardware Scarcity: Discussing why the assumption of infinite, uniform cloud capacity no longer holds true for modern workloads.
  4. 5:40 Defining Multi-Cluster Orchestrator: Explaining how MCO manages workloads across multiple clusters to optimize for proximity and availability.
  5. 7:15 Scaling AI Inference: How MCO handles the 'zero to one' scaling problem for compute-intensive inference engines.
  6. 8:45 Capacity and Metrics: Using open, accessible metrics to determine regional capacity and drive workload placement.
  7. 10:15 Latency in LLM Inference: The challenges of managing high-latency, multi-modal LLM requests compared to traditional microservices.
  8. 11:55 Standardizing Cluster Inventories: Using Cluster Profiles to create a normalized, open-source specification for multi-cluster management.