# 903: LLM Benchmarks Are Lying to You (And What to Do Instead), with Sinan Ozdemir

Page: https://stenobird.com/podcast/super-data-science/903-llm-benchmarks-are-lying-to-you-and-what-to-do-instead-with-sinan-ozdemir
Text version: https://stenobird.com/podcast/super-data-science/903-llm-benchmarks-are-lying-to-you-and-what-to-do-instead-with-sinan-ozdemir.md
Podcast: [Super Data Science: ML & AI Podcast with Jon Krohn](https://stenobird.com/podcast/super-data-science)
Published: 2025-07-08T11:00:00+00:00
Episode link: https://www.podtrac.com/pts/redirect.mp3/chrt.fm/track/E581B9/arttrk.com/p/VI4CS/pscrb.fm/rss/p/traffic.megaphone.fm/SUPERDATASCIENCEPTYLTD4709835756.mp3?updated=1752060782
Audio file: https://www.podtrac.com/pts/redirect.mp3/chrt.fm/track/E581B9/arttrk.com/p/VI4CS/pscrb.fm/rss/p/traffic.megaphone.fm/SUPERDATASCIENCEPTYLTD4709835756.mp3?updated=1752060782
Processing state: failed
JSON: https://stenobird.com/v1/public/podcasts/super-data-science/episodes/903-llm-benchmarks-are-lying-to-you-and-what-to-do-instead-with-sinan-ozdemir
Duration seconds: 5300

## Resource

Has AI benchmarking reached its limit, and what do we have to fill this gap? Sinan Ozdemir speaks to Jon Krohn about the lack of transparency in training data and the necessity of human-led quality assurance to detect AI hallucinations, when and why to be skeptical of AI benchmarks, and the future of benchmarking agentic and multimodal models. Additional materials: ⁠⁠⁠⁠⁠www.superdatascience.com/903⁠⁠⁠⁠ This episode is brought to you by Trainium2, the latest AI chip from AWS, by ⁠⁠Adverity, the conversational analytics platform⁠⁠ and by the ⁠⁠Dell AI Factory with NVIDIA⁠⁠. Interested in sponsoring a SuperDataScience Podcast episode? Email natalie@superdatascience.com for sponsorship information. In this episode you will learn: (16:48) Sinan’s new podcast, Practically Intelligent (21:54) What to know about the limits of AI benchmarking (53:22) Alternatives to AI benchmarks (1:01:23) The difficulties in getting a model to recognize its mistakes

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/super-data-science/episodes/903-llm-benchmarks-are-lying-to-you-and-what-to-do-instead-with-sinan-ozdemir/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/super-data-science/903-llm-benchmarks-are-lying-to-you-and-what-to-do-instead-with-sinan-ozdemir.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.