{"podcast":{"title":"Adventures in Machine Learning","slug":"adventures-in-machine-learning","podcast_index_feed_id":2981332,"rss_url":"https://www.spreaker.com/show/6102041/episodes/feed","website_url":"https://topenddevs.com/podcasts/adventures-in-machine-learning","image_url":"https://d3wo5wojvuv7l.cloudfront.net/t_rss_itunes_square_1400/images.spreaker.com/original/230facb439840ff787c776d3ed78fcbd.jpg","author":"Charles M Wood","episode_count":209,"summary":"Machine Learning is growing in leaps and bounds both in capability and adoption. Listen to our experts discuss the ideas and fundamentals needed to succeed as a Machine Learning Engineer. Become a supporter of this podcast: https://www.spreaker.com/podcast/adventures-in-machine-learning--6102041/support .","last_synced_at":null,"page_url":"https://stenobird.com/podcast/adventures-in-machine-learning"},"episode":{"title":"A/B Testing with ML ft. Michael Berk - ML 181","slug":"a-b-testing-with-ml-ft-michael-berk-ml-181","published_at":"2025-01-02T11:00:00+00:00","page_url":"https://stenobird.com/podcast/adventures-in-machine-learning/a-b-testing-with-ml-ft-michael-berk-ml-181","show_page_url":"https://stenobird.com/podcast/adventures-in-machine-learning","url":"https://www.spreaker.com/episode/a-b-testing-with-ml-ft-michael-berk-ml-181--63548854","audio_url":"https://dts.podtrac.com/redirect.mp3/api.spreaker.com/download/episode/63548854/ml_181.mp3","summary":"Learn how to scale experimentation from simple control groups to automated A/B testing infrastructure. This episode explores the transition from manual data analysis to robust, automated frameworks for measuring feature impact.","meta_description":"Explore the evolution of A/B testing, from frequentist methods to automated infrastructure and the future of adaptive loss functions in ML.","key_points":["Main idea: Effective A/B testing requires a transition from simple manual analysis to automated, continuous integration-based experimentation frameworks","Practical takeaway: Use pre-intervention and post-intervention methods as early, low-cost alternatives to full-scale randomized control trials","Failure mode: Relying on fixed loss functions may limit the development of truly general AI that can adapt to new information","Main idea: In large-scale user bases, significant lift is often achieved by optimizing specific sub-metrics rather than attempting to move global retention rates","Practical takeaway: Start by determining necessary sample sizes and experiment durations based on expected lift to avoid wasting resources on non-significant results"],"chapters":[{"start_ms":70000,"title":"Introduction to Experimentation at Tubi","summary":"Michael Burke discusses his role in managing A/B testing infrastructure and ad configuration for Tubi."},{"start_ms":275000,"title":"Frequentist vs. Bayesian Approaches","summary":"A deep dive into the use of frequentist experimentation and the importance of statistical significance in randomized control trials."},{"start_ms":495000,"title":"Reducing Variance with Historical Data","summary":"Using pre-existing data to reduce variance and achieve clearer results in experimental testing."},{"start_ms":930000,"title":"Causal Inference and Robust Modeling","summary":"The importance of causal modeling and simulation in establishing true causality beyond simple correlation."},{"start_ms":1360000,"title":"Scaling A/B Testing Infrastructure","summary":"The journey from manual data collection to building automated, autonomous experimentation services within a CI/CD pipeline."},{"start_ms":2195000,"title":"Real-world Model Performance","summary":"The discrepancy between high training accuracy and real-world performance when models interact with live user data."},{"start_ms":2625000,"title":"The Future of Machine Learning","summary":"Discussion on adaptive loss functions, conformal prediction, and the evolution of general AI."}],"topics":["A/B Testing","Machine Learning","Causal Inference","Experimentation Infrastructure","Frequentist Statistics","Data Science","General AI","Feature Optimization"],"duration_seconds":2741,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/adventures-in-machine-learning/episodes/a-b-testing-with-ml-ft-michael-berk-ml-181/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/adventures-in-machine-learning/a-b-testing-with-ml-ft-michael-berk-ml-181.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}