{"podcast":{"title":"Adventures in DevOps","slug":"adventures-in-devops","podcast_index_feed_id":686419,"rss_url":"https://adventuresindevops.com/episodes/rss.xml","website_url":"https://adventuresindevops.com","image_url":"https://d3wo5wojvuv7l.cloudfront.net/t_rss_itunes_square_1400/images.spreaker.com/original/2f474744f84e93eba827bee58d58c1c9.jpg","author":"Adventures in DevOps","episode_count":274,"summary":"Join us in listening to the experienced experts discuss cutting edge challenges in the world of DevOps. From applying the mindset at your company, to career growth and leadership challenges within engineering teams, and avoiding the common antipatterns. Every episode you'll meet a new industry veteran guest with their own unique story.","last_synced_at":null,"page_url":"https://stenobird.com/podcast/adventures-in-devops"},"episode":{"title":"How to build in Observability at Petabyte Scale","slug":"how-to-build-in-observability-at-petabyte-scale","published_at":"2025-09-07T00:00:00+00:00","page_url":"https://stenobird.com/podcast/adventures-in-devops/how-to-build-in-observability-at-petabyte-scale","show_page_url":"https://stenobird.com/podcast/adventures-in-devops","url":"https://adventuresindevops.com/episodes/2025/09/07/how-you-build-observability-that-scales-to-enterprise","audio_url":"https://dts.podtrac.com/redirect.mp3/api.spreaker.com/download/episode/67654497/download.mp3","summary":"Learn how Observe scales observability to petabytes of data per day by leveraging Snowflake's architecture instead of building a proprietary database. The discussion covers the technical trade-offs of using Kafka for stream processing and the strategic move toward open data formats like Iceberg.","meta_description":"Explore the architecture of petabyte-scale observability, from using Snowflake and Kafka to implementing multi-cloud, vendor-neutral data strategies.","key_points":["Main idea: Avoid the 'founding engineer instinct' of building a custom database to focus on delivering immediate user value","Architectural choice: Use Kafka as a buffer to smooth out massive data bursts before they hit Snowflake's batch-based engine","Strategic advantage: Leveraging open formats like Iceberg prevents vendor lock-in and allows customers to maintain true data ownership","Failure mode: Relying on proprietary cloud services like AWS Kinesis can create tight coupling that hinders multi-cloud (GCP/Azure) expansion","Practical takeaway: A usage-based pricing model for queries, paired with low-cost ingestion, prevents the 'bill shock' common in observability"],"chapters":[{"start_ms":60000,"title":"Context: Observability at Scale","summary":"Introduction to the challenges of managing petabyte-scale data streams and the evolution of observability expertise."},{"start_ms":260000,"title":"The Decision Against Proprietary Engines","summary":"Why building on top of Snowflake was a strategic choice to avoid the overhead of developing a custom execution engine."},{"start_ms":470000,"title":"Kafka as a Buffering Layer","summary":"Using Kafka to manage high-volume ingestion and bridge the gap between streaming data and batch-based processing."},{"start_ms":880000,"title":"Predictable Pricing Models","summary":"How separating ingestion costs from query usage helps customers avoid unexpected monthly billing spikes."},{"start_ms":1290000,"title":"Custom Stream Processing","summary":"The technical necessity of building custom stream processing layers to handle historical data reprocessing efficiently."},{"start_ms":1700000,"title":"Future-Proofing with Iceberg","summary":"Leveraging open data formats to enable data portability and multi-cloud interoperability."},{"start_ms":2110000,"title":"Security and Identity Risks","summary":"Discussing the risks of IAM trust policy exploitation and the importance of modern authentication like passkeys."}],"topics":["Observability","Snowflake","Kafka","Data Engineering","Cloud Architecture","Apache Iceberg","Petabyte Scale","Stream Processing","AWS S3"],"duration_seconds":2731,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/adventures-in-devops/episodes/how-to-build-in-observability-at-petabyte-scale/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/adventures-in-devops/how-to-build-in-observability-at-petabyte-scale.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}