{"podcast":{"title":"Screaming in the Cloud","slug":"screaming-in-the-cloud","podcast_index_feed_id":512714,"rss_url":"https://feeds.transistor.fm/screaming-in-the-cloud","website_url":"https://screaminginthecloud.com","image_url":"https://img.transistorcdn.com/sjY7QBiTinCDr8X80gOsgDaM4fMY0WuZn87UxNTh6Fw/rs:fill:0:0:1/w:1400/h:1400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS9zaG93/LzE0OTQvMTU4Mzg2/OTQ4My1hcnR3b3Jr/LmpwZw.jpg","author":"Corey Quinn","episode_count":673,"summary":"Screaming in the Cloud with Corey Quinn features conversations with domain experts in the world of Cloud Computing. Topics discussed include AWS, GCP, Azure, Oracle Cloud, and the \"why\" behind how businesses are coming to think about the Cloud.","last_synced_at":null,"page_url":"https://stenobird.com/podcast/screaming-in-the-cloud"},"episode":{"title":"Engineering Around Extreme S3 Scale with R. Tyler Croy","slug":"engineering-around-extreme-s3-scale-with-r-tyler-croy","published_at":"2026-01-13T11:00:00+00:00","page_url":"https://stenobird.com/podcast/screaming-in-the-cloud/engineering-around-extreme-s3-scale-with-r-tyler-croy","show_page_url":"https://stenobird.com/podcast/screaming-in-the-cloud","url":"https://share.transistor.fm/s/c1aea350","audio_url":"https://dts.podtrac.com/redirect.mp3/media.transistor.fm/c1aea350/5f9848e3.mp3","summary":"When S3 scale reaches hundreds of billions of objects, standard cloud operations like checksumming can cost six figures. R. Tyler Croy explains how Scribd engineers around these 'broken physics' by reducing object counts and building custom infrastructure.","meta_description":"Learn how Scribd manages extreme S3 scale, avoids massive metadata costs, and re-engineers storage architecture to handle 100 billion objects.","key_points":["Main idea: At extreme scale, simple S3 batch operations like checksumming can cost $100,000 due to per-object pricing","Practical takeaway: Reducing object count from 100 billion to 100 million is more effective than negotiating discounts","Failure mode: Relying on standard SDKs and default behaviors can lead to unmanageable metadata and request costs","Engineering strategy: Use technology-driven solutions to create new data capabilities rather than just seeking contract-based savings","Infrastructure insight: Modern AI and LLMs have increased the economic value of massive, legacy document archives"],"chapters":[{"start_ms":215000,"title":"The Scale of S3 Spend","summary":"An exploration of how S3 costs escalate when managing hundreds of millions of objects."},{"start_ms":365000,"title":"When Normal Physics Stop Working","summary":"Discussing the point where standard cloud engineering assumptions and cost models break down."},{"start_ms":515000,"title":"The High Cost of Metadata","summary":"Why interacting with legacy S3 buckets using modern SDKs can lead to massive unexpected expenses."},{"start_ms":665000,"title":"AI and the Value of Old Data","summary":"How large language models have transformed the utility and relevance of massive, older document archives."},{"start_ms":810000,"title":"Reducing Object Count","summary":"A strategy for bringing object counts down from 100 billion to 100 million to make costs manageable."},{"start_ms":965000,"title":"Engineering vs. Negotiating","summary":"Why building custom technical solutions is often more impactful than seeking enterprise discounts."},{"start_ms":1275000,"title":"The Unbounded Growth Problem","summary":"Addressing the challenges of managing data growth in an era of continuous accumulation."}],"topics":["S3 Scale","Cloud Economics","Infrastructure Engineering","Data Storage","Object Storage","Cost Optimization","Big Data","AWS"],"duration_seconds":2019,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/screaming-in-the-cloud/episodes/engineering-around-extreme-s3-scale-with-r-tyler-croy/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/screaming-in-the-cloud/engineering-around-extreme-s3-scale-with-r-tyler-croy.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}