# Episode 60: 10 Things I Hate About AI Evals with Hamel Husain

Page: https://stenobird.com/podcast/vanishing-gradients-4989163/episode-60-10-things-i-hate-about-ai-evals-with-hamel-husain
Text version: https://stenobird.com/podcast/vanishing-gradients-4989163/episode-60-10-things-i-hate-about-ai-evals-with-hamel-husain.md
Podcast: [Vanishing Gradients](https://stenobird.com/podcast/vanishing-gradients-4989163)
Published: 2025-09-30T07:30:00+00:00
Episode link: https://hugobowne.substack.com/p/episode-60-10-things-i-hate-about-e83
Audio file: https://api.substack.com/feed/podcast/181324508/df8e23c2d8d560bd2bdd0f6995ea58c9.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/vanishing-gradients-4989163/episodes/episode-60-10-things-i-hate-about-ai-evals-with-hamel-husain
Duration seconds: 4396

## Resource

Most AI teams find "evals" frustrating, but ML Engineer Hamel Husain argues they’re just using the wrong playbook. In this episode, he lays out a data-centric approach to systematically measure and improve AI, turning unreliable prototypes into robust, production-ready systems. Drawing from his experience getting countless teams unstuck, Hamel explains why the solution requires a "revenge of the data scientists." He details the essential mindset shifts, error analysis techniques, and practical steps needed to move beyond guesswork and build AI products you can actually trust. We talk through: The 10(+1) critical mistakes that cause teams to waste time on evals Why "hallucination scores" are a waste of time (and what to measure instead) The manual review process that finds major issues in hours, not weeks A step-by-step method for building LLM judges you can actually trust How to use domain experts without getting stuck in endless review committees Guest Bryan Bischof's "Failure as a Funnel" for debugging complex AI agents If you're tired of ambiguous "vibe checks" and want a clear process that delivers real improvement, this episode provides the definitive roadmap. LINKS Hamel's website and blog ( https://hamel.dev/ ) Hugo speaks with Philip Carter (Honeycomb) about aligning your LLM-as-a-judge with your domain expertise ( https://vanishinggradients.fireside.fm/51 ) Hamel Husain on Lenny's pocast, which includes a live demo of error analysis ( https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill ) The episode of VG in which Hamel and Hugo talk about Hamel's "data consulting in Vegas" era ( https://vanishinggradients.fireside.fm/9 ) Upcoming Events on Luma ( https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk ) Watch the podcast video on YouTube ( https://yo…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/vanishing-gradients-4989163/episodes/episode-60-10-things-i-hate-about-ai-evals-with-hamel-husain/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/vanishing-gradients-4989163/episode-60-10-things-i-hate-about-ai-evals-with-hamel-husain.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.