Episode
Episode 72: Why Agents Solve the Wrong Problem (and What Data Scientists Do Instead)
- Podcast
- Vanishing Gradients
- Published
- Mar 20, 2026
- Duration seconds
- 5619
- Processing state
not_requested
Actions
POST https://stenobird.com/v1/public/podcasts/vanishing-gradients-4989163/episodes/episode-72-why-agents-solve-the-wrong-problem-and-what-data-scientists-do-instead/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/vanishing-gradients-4989163/episode-72-why-agents-solve-the-wrong-problem-and-what-data-scientists-do-instead.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
I often see what I would consider to be b******t evals , especially in data, like write this dumb SQL . Almost every one of these dumb SQL questions that I’ve seen for benchmarks are just so either obviously easy or overwhelmingly adversarial. They just, they don’t feel valuable as a data scientist , it’s something that you probably would never ask a real data scientist to do. So I went out my way to create real ones. Let me read one to you. Bryan Bischof , Head of AI at Theory Ventures , joins Hugo to talk about what happened when 150 people spent six hours using AI agents to answer real data science questions across SQL tables , log files , and 750,000 PDFs . They Discuss: * Failure Funnels , pinpoint where agent reasoning breaks down using causal-chain binary evaluations instead of vague 1-5 scales; * Median Score: 23 out of 65 , what happened when world-class engineers turned agents loose on real data work, and why general-purpose coding agents with human prodding beat fancy frameworks; * Zero-Cost Submissions Kill Trust , without a penalty for wrong answers, agents hill-climb to correct submissions through brute force instead of building confidence; * Data Science is “Zooming” , moving beyond binary decisions to iterative problem framing , refining “does our inventory suck?” into a tractable hypothesis; * MCP as Semantic Layer , model your organization’s proprietary knowledge once and distribute it to whatever LLM interface your team prefers; * The Subagent vs. Tool Debate , a distinction that adds cognitive load without hiding complexity; * Self-Orchestration Gap , agents don’t yet realize they should trigger specialized extraction frameworks like DocETL instead of reading 750K PDFs one by one; * The Future of Evals , from vibe checks to objective functions and c…