Stenobird

Episode

AI Engineering is Dead? Hectic Stag Dos, Event FOMO or Not to Go and Agent Evals are HARRDDD

Podcast
You've Been a Bad Agent
Published
Jun 5, 2025
Duration seconds
3012
Processing state
processed
Canonical source
https://share.transistor.fm/s/ed149948
Audio
https://media.transistor.fm/ed149948/956eb39a.mp3
JSON
/v1/public/podcasts/you-ve-been-a-bad-agent-7334728/episodes/ai-engineering-is-dead-hectic-stag-dos-event-fomo-or-not-to-go-and-agent-evals-are-harrddd
Markdown
/podcast/you-ve-been-a-bad-agent-7334728/ai-engineering-is-dead-hectic-stag-dos-event-fomo-or-not-to-go-and-agent-evals-are-harrddd.md

Actions

  • POST https://stenobird.com/v1/public/podcasts/you-ve-been-a-bad-agent-7334728/episodes/ai-engineering-is-dead-hectic-stag-dos-event-fomo-or-not-to-go-and-agent-evals-are-harrddd/transcription-requests
    Idempotently request low-priority transcript generation for this episode.
  • GET https://stenobird.com/podcast/you-ve-been-a-bad-agent-7334728/ai-engineering-is-dead-hectic-stag-dos-event-fomo-or-not-to-go-and-agent-evals-are-harrddd.md
    Read the agent-friendly Markdown representation of this episode resource.

Summary

A deep dive into the friction of modern engineering, from the difficulty of evaluating AI agents to the complexities of JavaScript build tools. The hosts also reflect on the psychological impact of consumer behavior and the challenges of tracing Python subprocesses.

Topics

  • AI Engineering
  • Python Tracing
  • JavaScript Build Tools
  • AI Agent Evaluation
  • Cloudflare Workers
  • Software Architecture
  • LLM Research
  • Developer Experience

Highlights

  • Main idea: Evaluating AI agents remains a significant technical hurdle due to the difficulty of creating reliable benchmarks
  • Practical takeaway: Use Python's 'sitecustomize' feature to implement deep tracing for subprocesses and middleware
  • Failure mode: Relying on strict rules and rigid workflows in AI tooling can be less effective than a 'tools, not rules' architecture
  • Technical insight: Integrating legacy JavaScript libraries with modern Vite plugins and Cloudflare workers often creates complex configuration conflicts
  • Observation: The shift toward 'models plus fetch' architectures simplifies agentic workflows by prioritizing tool access over rigid instruction sets

Chapters

  1. 1:00 Reflections on Las Vegas and Consumerism: A discussion on the overwhelming nature of Las Vegas and the contrast between high-value products and low-value slot machines.
  2. 5:00 Travels in Peru: Personal updates regarding high-altitude hiking in Peru and the physical toll of travel.
  3. 9:00 LLM Creativity and Naming: Comparing the ability of different models, like Claude Opus, to generate creative and coherent names.
  4. 12:00 The JavaScript Build Tooling Nightmare: The struggle of configuring JSON Path, Vite plugins, and Cloudflare workers in a monorepo environment.
  5. 16:00 Deep Research and Python Tracing: How AI-driven deep research uncovered the 'sitecustomize' feature to solve complex Python tracing challenges.
  6. 24:00 Architecting AI Agents: Moving away from strict rule-based systems toward flexible, tool-augmented model architectures.
  7. 39:00 The Difficulty of Agent Evaluations: The technical challenges of creating dynamic inputs and outputs for benchmarking AI performance.