Episode
Next Level AI Evals for 2026
- Podcast
- Vanishing Gradients
- Published
- Apr 23, 2026
- Duration seconds
- 3214
- Processing state
not_requested- Canonical source
- https://hugobowne.substack.com/p/next-level-ai-evals-for-2026
Actions
POST https://stenobird.com/v1/public/podcasts/vanishing-gradients-4989163/episodes/next-level-ai-evals-for-2026/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/vanishing-gradients-4989163/next-level-ai-evals-for-2026.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
There are a lot of reasons why we should do AI evals. For many companies doing AI evals is the way to build the feedback loop into the product development lifecycle. So it is like your compass. We’re using AI evals as a compass to guide product development and also product iteration. And also, many times we need evals to function as the pass or fail gate in release decisions. Whether this product is good enough for release or whether it is good enough for experiment, evals are also used in that. Stella Wenxing Liu , Head of Applied Science at ASU , and Eddie Landesberg , Staff Data Scientist at Google , join Hugo to talk about why AI evaluation is evolving from “vibe checks” into a rigorous, multi-disciplinary science and how causal inference will take AI evals to the next level in 2026 . Vanishing Gradients is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. They Discuss: * Team-Centric AI Evals , integrating product managers , data scientists , and SMEs under a “benevolent dictator” (or not!) to ensure comprehensive and effective evaluation; * Custom Evaluation Metrics , moving beyond generic vendor metrics to analyze raw data and identify specific failure modes , avoiding generic product outcomes; * AI as Policy Evaluation , framing AI evaluation as a causal inference problem to estimate counterfactual performance of new “policies” (prompts, models) and predict online AB test outcomes; * Clear Product Constraints , defining what an AI product should not do with strict guardrails to prevent misuse, control costs, and avoid brand dilution ; * Calibrated LLM Judges , statistically aligning LLM-as-a-judge with human experts using causal inference to ensure valid proxies for human welfare and business…