{"podcast":{"title":"AI Engineering Podcast","slug":"ai-engineering-podcast","podcast_index_feed_id":5875646,"rss_url":"https://serve.podhome.fm/rss/c9abdd38-a5dc-5eb2-96fd-f833f93208a7","website_url":"https://www.aiengineeringpodcast.com","image_url":"https://assets.podhome.fm/f6ff0caa-931b-4c08-bfdd-08dc7f5cd336/638557211890591941ai_engineering_podcast_logo.jpg","author":"Tobias Macey","episode_count":79,"summary":"This show is your guidebook to building scalable and maintainable AI systems. You will learn how to architect AI applications, apply AI to your work, and the considerations involved in building or customizing new models. Everything that you need to know to deliver real impact and value with machine learning and artificial intelligence.","last_synced_at":null,"page_url":"https://stenobird.com/podcast/ai-engineering-podcast"},"episode":{"title":"From Blind Spots to Observability: Operationalizing LLM Apps with OpenLit","slug":"from-blind-spots-to-observability-operationalizing-llm-apps-with-openlit","published_at":"2026-02-15T19:23:44+00:00","page_url":"https://stenobird.com/podcast/ai-engineering-podcast/from-blind-spots-to-observability-operationalizing-llm-apps-with-openlit","show_page_url":"https://stenobird.com/podcast/ai-engineering-podcast","url":"https://www.aiengineeringpodcast.com/openlit-open-source-llmops-episode-77","audio_url":"https://op3.dev/e/dts.podtrac.com/redirect.mp3/serve.podhome.fm/episode/f6ff0caa-931b-4c08-bfdd-08dc7f5cd336/63906714955750011937985833-1425-4bb6-acfe-ce3b20759f52.mp3","summary":"LLM applications suffer from critical blind spots in model behavior, token costs, and prompt management. This episode explores how to use OpenTelemetry-native observability to move from opaque 'vibe coding' to production-ready AI engineering.","meta_description":"Learn how to operationalize LLM apps using OpenLit, focusing on OpenTelemetry, prompt management, and avoiding the hidden costs of AI development.","key_points":["Main idea: Observability must be established before the MVP phase to prevent runaway token costs and unmanageable model latency","Practical takeaway: Use OpenTelemetry-native tools to ensure vendor-neutral tracing across models, tools, and data stores","Failure mode: Hard-coding prompts into application code creates deployment bottlenecks and prevents effective experimentation","Technical strategy: Implement stepwise traces to understand how context modification affects final model responses","Design principle: Prioritize standard-compliant architectures to leverage existing community ecosystems like Grafana and Dash0"],"chapters":[{"start_ms":285000,"title":"The Dangers of Hard-coded Prompts","summary":"Discussing the operational friction caused by embedding prompts directly in application logic and the need for external management."},{"start_ms":520000,"title":"Moving Beyond Vibe Coding","summary":"The transition from experimental 'vibe coding' to structured, measurable AI development workflows."},{"start_ms":755000,"title":"Building the LLM Ops Stack","summary":"Defining the essential components for a full end-to-end LLM operations suite beyond simple hosting."},{"start_ms":970000,"title":"Experimentation and Evaluation","summary":"Using visual comparisons and traffic routing to evaluate different prompts and models effectively."},{"start_ms":1215000,"title":"Integrating with Existing Observability","summary":"How OpenLit integrates with established platforms like Grafana without adding environment complexity."},{"start_ms":1460000,"title":"The Importance of Early Observability","summary":"Why monitoring model performance and costs is critical even before reaching the MVP stage."},{"start_ms":2365000,"title":"Tackling Context Blind Spots","summary":"Addressing the difficulty of debugging how context changes impact model outputs."}],"topics":["LLM Observability","OpenTelemetry","AI Engineering","Prompt Management","LLMOps","Token Cost Optimization","OpenLit","Model Evaluation"],"duration_seconds":3036,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/ai-engineering-podcast/episodes/from-blind-spots-to-observability-operationalizing-llm-apps-with-openlit/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/ai-engineering-podcast/from-blind-spots-to-observability-operationalizing-llm-apps-with-openlit.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}