Episode
RAG Risks: Why Retrieval-Augmented LLMs are Not Safer with Sebastian Gehrmann - #732
- Published
- May 21, 2025
- Duration seconds
- 3429
- Processing state
failed
Actions
POST https://stenobird.com/v1/public/podcasts/twiml-ai-podcast/episodes/rag-risks-why-retrieval-augmented-llms-are-not-safer-with-sebastian-gehrmann-732/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/twiml-ai-podcast/rag-risks-why-retrieval-augmented-llms-are-not-safer-with-sebastian-gehrmann-732.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
Today, we're joined by Sebastian Gehrmann, head of responsible AI in the Office of the CTO at Bloomberg, to discuss AI safety in retrieval-augmented generation (RAG) systems and generative AI in high-stakes domains like financial services. We explore how RAG, contrary to some expectations, can inadvertently degrade model safety. We cover examples of unsafe outputs that can emerge from these systems, different approaches to evaluating these safety risks, and the potential reasons behind this counterintuitive behavior. Shifting to the application of generative AI in financial services, Sebastian outlines a domain-specific safety taxonomy designed for the industry's unique needs. We also explore the critical role of governance and regulatory frameworks in addressing these concerns, the role of prompt engineering in bolstering safety, Bloomberg’s multi-layered mitigation strategies, and vital areas for further work in improving AI safety within specialized domains. The complete show notes for this episode can be found at https://twimlai.com/go/732.