{"podcast":{"title":"The Data Exchange with Ben Lorica","slug":"the-data-exchange-with-ben-lorica","podcast_index_feed_id":1196000,"rss_url":"https://rss.buzzsprout.com/682433.rss","website_url":"https://thedataexchange.media/","image_url":"https://storage.buzzsprout.com/ljk0yj7r22pi61grsmelnsoa9084?.jpg","author":"Ben Lorica","episode_count":345,"summary":"A series of informal conversations with thought leaders, researchers, practitioners, and writers on a wide range of topics in technology, science, and of course big data, data science, artificial intelligence, and related applications. Anchored by Ben Lorica (@BigData), the Data Exchange also features a roundup of the most important stories from the worlds of data, machine learning and AI. Detailed show notes for each episode can be found on https://thedataexchange.media/ The Data Exchange podcast is a production of Gradient Flow [https://gradientflow.com/].","last_synced_at":null,"page_url":"https://stenobird.com/podcast/the-data-exchange-with-ben-lorica"},"episode":{"title":"The Junior Data Engineer is Now an AI Agent","slug":"the-junior-data-engineer-is-now-an-ai-agent","published_at":"2026-01-08T12:00:00+00:00","page_url":"https://stenobird.com/podcast/the-data-exchange-with-ben-lorica/the-junior-data-engineer-is-now-an-ai-agent","show_page_url":"https://stenobird.com/podcast/the-data-exchange-with-ben-lorica","url":"https://dts.podtrac.com/redirect.mp3/www.buzzsprout.com/682433/episodes/18437537-the-junior-data-engineer-is-now-an-ai-agent.mp3","audio_url":"https://dts.podtrac.com/redirect.mp3/www.buzzsprout.com/682433/episodes/18437537-the-junior-data-engineer-is-now-an-ai-agent.mp3","summary":"AI agents are moving beyond simple chat interfaces to perform complex, stateful data engineering tasks like building and testing pipelines. Matthew Glickman explains how Genesis Computing uses agentic technology to automate critical workflows and capture institutional knowledge.","meta_description":"Explore the rise of AI data agents and how they are automating the data engineering persona, from pipeline creation to managing complex distributed system…","key_points":["Main idea: AI agents are evolving from simple query interfaces into autonomous workers capable of executing multi-step data engineering workflows","Practical takeaway: Use AI to capture 'ambient knowledge' from senior engineers to prevent institutional memory loss during migrations","Failure mode: Automating entry-level tasks risks breaking the talent pipeline, as junior engineers need these foundational tasks to develop into seniors","Strategic lesson: Only build custom data infrastructure if it provides a core competitive advantage; otherwise, leverage specialized third-party experts","Technical insight: Unlike stateless software engineering, data engineering requires agents that can manage side effects across distributed systems like Spark and Kafka"],"chapters":[{"start_ms":60000,"title":"The Genesis of Genesis Computing","summary":"Matthew Glickman discusses his transition from Goldman Sachs and Snowflake to addressing the 'wall' enterprises hit when trying to deploy LLMs for data."},{"start_ms":320000,"title":"The Difficulty of the Last 10%","summary":"A look at why moving from flashy AI demos to deterministic, production-ready data pipelines is incredibly challenging."},{"start_ms":560000,"title":"Targeting the Data Engineer Persona","summary":"How Genesis Computing focuses specifically on automating the workflows of the data engineering persona rather than general business users."},{"start_ms":800000,"title":"Risks of Expanding the Engineering Pool","summary":"The implications of using AI to allow non-engineers to perform data engineering tasks and the potential for introducing errors."},{"start_ms":1090000,"title":"Capturing Institutional Knowledge","summary":"Using AI to ambiently acquire knowledge from experts so that pipeline specifics aren't lost when people leave the organization."},{"start_ms":1340000,"title":"The Burden of Legacy Migrations","summary":"Discussing the massive complexity of migrating legacy systems like SAP HANA and Oracle in large enterprises."},{"start_ms":1580000,"title":"Closing the Loop with Business Users","summary":"How agentic workflows allow business users to validate data logic and revenue calculations directly without a human middleman."}],"topics":["AI Agents","Data Engineering","Data Pipelines","Enterprise AI","Automation","Genesis Computing","Machine Learning Operations","Distributed Systems"],"duration_seconds":3273,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/the-data-exchange-with-ben-lorica/episodes/the-junior-data-engineer-is-now-an-ai-agent/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/the-data-exchange-with-ben-lorica/the-junior-data-engineer-is-now-an-ai-agent.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}