# Shedding Light On Silent Model Failures With NannyML

Page: https://stenobird.com/podcast/ai-engineering-podcast/shedding-light-on-silent-model-failures-with-nannyml
Text version: https://stenobird.com/podcast/ai-engineering-podcast/shedding-light-on-silent-model-failures-with-nannyml.md
Podcast: [AI Engineering Podcast](https://stenobird.com/podcast/ai-engineering-podcast)
Published: 2022-09-14T02:00:00+00:00
Episode link: https://www.aiengineeringpodcast.com/nannyml-silent-model-failure-episode-11
Audio file: https://op3.dev/e/dts.podtrac.com/redirect.mp3/serve.podhome.fm/episode/f6ff0caa-931b-4c08-bfdd-08dc7f5cd336/6385305388920143521a1acb91-7368-4459-941f-8653a2d3bb9bv6.mp3
Processing state: failed
JSON: https://stenobird.com/v1/public/podcasts/ai-engineering-podcast/episodes/shedding-light-on-silent-model-failures-with-nannyml
Duration seconds: 3798

## Resource

Summary Because machine learning models are constantly interacting with inputs from the real world they are subject to a wide variety of failures. The most commonly discussed error condition is concept drift, but there are numerous other ways that things can go wrong. In this episode Wojtek Kuberski explains how NannyML is designed to compare the predicted performance of your model against its actual behavior to identify silent failures and provide context to allow you to determine whether and how urgently to address them. Announcements Hello and welcome to the Machine Learning Podcast, the podcast about machine learning and how to bring it from idea to delivery. Data powers machine learning, but poor data quality is the largest impediment to effective ML today. Galileo is a collaborative data bench for data scientists building Natural Language Processing (NLP) models to programmatically inspect, fix and track their data across the ML workflow (pre-training, post-training and post-production) – no more excel sheets or ad-hoc python scripts. Get meaningful gains in your model performance fast, dramatically reduce data labeling and procurement costs, while seeing 10x faster ML iterations. Galileo is offering listeners a free 30 day trial and a 30% discount on the product there after. This offer is available until Aug 31, so go to themachinelearningpodcast.com/galileo and request a demo today! Your host is Tobias Macey and today I’m interviewing Wojtek Kuberski about NannyML and the work involved in post-deployment data science Interview Introduction How did you get involved in machine learning? Can you describe what NannyML is and the story behind it? What is "post-deployment data science"?  How does it differ from the metrics/monitoring approach to managing the mod…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/ai-engineering-podcast/episodes/shedding-light-on-silent-model-failures-with-nannyml/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/ai-engineering-podcast/shedding-light-on-silent-model-failures-with-nannyml.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.