{"podcast":{"title":"Data Engineering Podcast","slug":"data-engineering-podcast","podcast_index_feed_id":403671,"rss_url":"https://serve.podhome.fm/rss/1c0357c0-6aba-5766-a2d5-2090d8dab6bc","website_url":"https://www.dataengineeringpodcast.com","image_url":"https://assets.podhome.fm/f6ff0caa-931b-4c08-bfdd-08dc7f5cd336/638557928872209534cover.jpg","author":"Tobias Macey","episode_count":510,"summary":"This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.","last_synced_at":null,"page_url":"https://stenobird.com/podcast/data-engineering-podcast"},"episode":{"title":"Prompt Management, Tracing, and Evals: The New Table Stakes for GenAI Ops","slug":"prompt-management-tracing-and-evals-the-new-table-stakes-for-genai-ops","published_at":"2026-02-15T16:11:29+00:00","page_url":"https://stenobird.com/podcast/data-engineering-podcast/prompt-management-tracing-and-evals-the-new-table-stakes-for-genai-ops","show_page_url":"https://stenobird.com/podcast/data-engineering-podcast","url":"https://www.dataengineeringpodcast.com/openlit-open-source-llmops-episode-501","audio_url":"https://op3.dev/e/dts.podtrac.com/redirect.mp3/serve.podhome.fm/episode/f6ff0caa-931b-4c08-bfdd-08dc7f5cd336/63906715056530017229e94ae4-20eb-474e-a235-2d30233e840c.mp3","summary":"Moving LLM applications from prototype to production requires more than just a good prompt; it requires robust observability and evaluation. Aman Agarwal explains how using OpenTelemetry-native tools can eliminate the blind spots of opaque model behavior and runaway token costs.","meta_description":"Learn how to operationalize GenAI using OpenTelemetry-native observability, prompt management, and evaluation workflows to avoid vendor lock-in and high c…","key_points":["Main idea: Transitioning from frontier models to cheaper alternatives requires a robust evaluation framework to ensure performance doesn't degrade","Practical takeaway: Use OpenTelemetry-native instrumentation to create debuggable traces across models, tools, and data stores without vendor lock-in","Failure mode: Hard-coding prompts into application code creates massive management debt as use cases scale into the thousands","Main idea: Observability is critical even in the MVP phase to prevent unmonitored token usage from causing unexpected budget spikes","Practical takeaway: Implement systematic experimentation by visually comparing different models and prompts using standardized trace data"],"chapters":[{"start_ms":60000,"title":"The Need for AI Operational Investment","summary":"Introduction to the challenges of managing AI development workflows and the necessity of operational groundwork."},{"start_ms":280000,"title":"The Perils of Hard-coded Prompts","summary":"Discussing the difficulty of managing large-scale prompt libraries when they are embedded directly in application logic."},{"start_ms":510000,"title":"Avoiding Vendor Lock-in","summary":"Why developers need the flexibility to swap models and tools without rebuilding their entire observability stack."},{"start_ms":730000,"title":"Building Open-Source Infrastructure","summary":"The motivation behind creating OpenLit as an accessible, open-source tool for the AI engineering community."},{"start_ms":960000,"title":"Experimentation and Evaluation","summary":"How to use visual comparisons of different models and prompts to drive better engineering decisions."},{"start_ms":1180000,"title":"OpenTelemetry-native Design","summary":"The importance of adhering to open standards to ensure seamless integration with existing developer ecosystems."},{"start_ms":1630000,"title":"Managing Distributed Traces","summary":"The complexities of managing OTel collectors and the evolving landscape of AI observability."}],"topics":["GenAI Ops","OpenTelemetry","LLM Observability","Prompt Engineering","Model Evaluation","AI Infrastructure","Token Cost Management","Open Source"],"duration_seconds":3043,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/data-engineering-podcast/episodes/prompt-management-tracing-and-evals-the-new-table-stakes-for-genai-ops/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/data-engineering-podcast/prompt-management-tracing-and-evals-the-new-table-stakes-for-genai-ops.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}