Stenobird

Episode

How can you test your code when you don’t know what’s in it?

Podcast
The Stack Overflow Podcast
Published
Mar 31, 2026
Duration seconds
1818
Processing state
processed
Canonical source
https://rss.art19.com/episodes/4322d3fc-87b7-4276-abce-c4b3c3b7da8a.mp3?rss_browser=BAhJIg90cmFuc2NyaWJyBjoGRVQ%3D--952c5701c84ad333c69d5faa668f8177091704f0
Audio
https://rss.art19.com/episodes/4322d3fc-87b7-4276-abce-c4b3c3b7da8a.mp3?rss_browser=BAhJIg90cmFuc2NyaWJyBjoGRVQ%3D--952c5701c84ad333c69d5faa668f8177091704f0
JSON
/v1/public/podcasts/the-stack-overflow-podcast/episodes/how-can-you-test-your-code-when-you-don-t-know-what-s-in-it
Markdown
/podcast/the-stack-overflow-podcast/how-can-you-test-your-code-when-you-don-t-know-what-s-in-it.md

Actions

  • POST https://stenobird.com/v1/public/podcasts/the-stack-overflow-podcast/episodes/how-can-you-test-your-code-when-you-don-t-know-what-s-in-it/transcription-requests
    Idempotently request low-priority transcript generation for this episode.
  • GET https://stenobird.com/podcast/the-stack-overflow-podcast/how-can-you-test-your-code-when-you-don-t-know-what-s-in-it.md
    Read the agent-friendly Markdown representation of this episode resource.

Summary

The shift toward Model Context Protocol (MCP) and LLM-driven agents introduces non-determinism that breaks traditional software testing. This discussion explores how developers can validate agentic workflows when the sequence of tool calls is decided on the fly by an AI.

Topics

  • Model Context Protocol
  • AI Agents
  • Software Testing
  • LLM-driven Development
  • MCP Servers
  • Agentic Workflows
  • Non-determinism
  • Software Architecture

Highlights

  • Main idea: MCP is becoming the new foundational layer for AI agents, shifting the abstraction level from LLM prompts to tool invocation
  • Failure mode: Relying on rigid, hard-coded workflows for AI agents defeats the purpose of the LLM's ability to decide the best path dynamically
  • Practical takeaway: Testing must move toward validating outcomes and bounds rather than checking specific, hand-authored code paths
  • Economic trade-off: AI-generated code may prioritize development velocity over performance, requiring a new pricing strategy to cover higher compute costs
  • Future outlook: As source code becomes a commodity, the value in engineering will shift toward data construction and managing the complexity of agentic interactions

Chapters

  1. 1:00 Guest Background: Fitz Nowlan discusses his journey from PhD research in distributed systems to leading AI architecture at SmartBear.
  2. 3:20 The Challenge of Non-deterministic Testing: The difficulty of testing MCP servers where the LLM decides the tool sequence on the fly, making rigid workflows impossible.
  3. 7:50 Prompt Engineering vs. Long-term Value: Why developers should avoid falling in love with specific prompts that may be rendered obsolete by newer, better models.
  4. 12:35 The Future of Unit Testing: A debate on whether traditional unit testing loses relevance when AI can effectively handle assertion-based testing.
  5. 21:20 Commoditization of CRUD Apps: How AI-driven development lowers margins for basic applications and shifts the focus toward more complex engineering problems.
  6. 27:50 MCP as the New Foundation: The transition from LLMs being the core focus to MCP providing the essential infrastructure for agentic tool use.