Episode
84: Trust But Canary: Configuration Safety at Scale
- Podcast
- Meta Tech Podcast
- Published
- Apr 2, 2026
- Duration seconds
- 2228
- Processing state
not_requested
Actions
POST https://stenobird.com/v1/public/podcasts/meta-tech-podcast-525691/episodes/84-trust-but-canary-configuration-safety-at-scale/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/meta-tech-podcast-525691/84-trust-but-canary-configuration-safety-at-scale.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
Have you ever wondered how Meta makes config rollouts safe at scale? In this episode, Pascal sits down with Ishwari and Joe to discuss Meta's approach for propagating changes across services in seconds and discuss why speed increases the need for strong safeguards. Catch the episode to discover canarying and progressive rollouts, the health checks and monitoring signals used to catch regressions early, and how incident reviews focus on improving systems rather than blaming people. We also hear how data and early AI/ML are slashing alert noise and speeding up bisecting when something goes wrong. Got feedback? Send it to us on Threads ( https://threads.net/@metatechpod ), Instagram ( https://instagram.com/metatechpod) and don't forget to follow our host Pascal ( https://mastodon.social/@passy, https://threads.net/@passy_ ). Fancy working with us? Check out https://www.metacareers.com/ . Links FFmpeg at Meta: Media Processing at Scale - https://engineering.fb.com/2026/03/02/video-engineering/ffmpeg-at-meta-media-processing-at-scale/ Reliably Changing Configuration @ Scale - https://atscaleconference.com/reliably-changing-configuration-scale/ Timestamps Intro 0:06 Introduction and Overview of Configuration Changes 2:31 Understanding Configurations in Distributed Systems 4:02 Meta's Configuration Management Systems 6:43 Safeguards and Incident Prevention 9:22 Deployment Mechanisms: Canary and Progressive Rollouts 12:06 Challenges in Configuration Consumption 14:39 Health Checks and Incident Response 17:13 Mitigation Strategies for Configuration Issues 19:18 Balancing Developer Velocity and Configuration Safety 21:09 Data-Driven Improvements in Incident Management 22:12 Leveraging AI for Change Detection 26:05 Challenges in Deployment and Testing 28:21 Reinventing Change Sa…