# Everything Hard About Building AI Agents Today

Page: https://stenobird.com/podcast/mlops-community/everything-hard-about-building-ai-agents-today
Text version: https://stenobird.com/podcast/mlops-community/everything-hard-about-building-ai-agents-today.md
Podcast: [MLOps.community](https://stenobird.com/podcast/mlops-community)
Published: 2025-06-13T14:00:46+00:00
Episode link: https://podcasters.spotify.com/pod/show/mlops/episodes/Everything-Hard-About-Building-AI-Agents-Today-e346kqk
Audio file: https://anchor.fm/s/174cb1b8/podcast/play/104075540/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2025-5-13%2F402093896-44100-2-9fea7a8a6b789.mp3
Processing state: processed
JSON: https://stenobird.com/v1/public/podcasts/mlops-community/episodes/everything-hard-about-building-ai-agents-today
Duration seconds: 2822

## Resource

Willem Pienaar and Shreya Shankar discuss the challenge of evaluating agents in production where "ground truth" is ambiguous and subjective user feedback isn't enough to improve performance. The discussion breaks down the three "gulfs" of human-AI interaction—Specification, Generalization, and Comprehension—and their impact on agent success. Willem and Shreya cover the necessity of moving the human "out of the loop" for feedback, creating faster learning cycles through implicit signals rather than direct, manual review. The conversation details practical evaluation techniques, including analyzing task failures with heat maps and the trade-offs of using simulated environments for testing. Willem and Shreya address the reality of a "performance ceiling" for AI and the importance of categorizing problems your agent can learn to solve, or will likely never be able to solve. // Bio Shreya Shankar PhD student in data management for machine learning. Willem Pienaar Willem Pienaar, CTO of Cleric, is a builder with a focus on LLM agents, MLOps, and open source tooling. He is the creator of Feast, an open source feature store, and contributed to the creation of both the feature store and MLOps categories. Before starting Cleric, Willem led the open source engineering team at Tecton and established the ML platform team at Gojek, where he built high-scale ML systems for the Southeast Asian decacorn. // Related Links https://www.google.com/about/careers/applications/?utm_campaign=profilepage&utm_medium=profilepage&utm_source=linkedin&src=Online/LinkedIn/linkedin_page https://cleric.ai/ ~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~ Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore MLOps Swag/Merch: [ ht…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/mlops-community/episodes/everything-hard-about-building-ai-agents-today/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/mlops-community/everything-hard-about-building-ai-agents-today.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.