Episode
LLM Unpacked: A Deep Dive into Modern AI Architectures
- Published
- Jul 29, 2025
- Duration seconds
- 2501
- Processing state
not_requested
Actions
POST https://stenobird.com/v1/public/podcasts/smart-enterprises-ai-frontiers-7077810/episodes/llm-unpacked-a-deep-dive-into-modern-ai-architectures/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/smart-enterprises-ai-frontiers-7077810/llm-unpacked-a-deep-dive-into-modern-ai-architectures.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
Join us for an insightful exploration into the cutting-edge design of today's Large Language Models . Seven years on from the original GPT architecture, have we truly seen groundbreaking changes, or are we simply refining existing foundations? This podcast focuses on the architectural developments that define flagship open models in 2025, moving beyond benchmark performance or training algorithms. In this episode, we'll unpack the key ingredients contributing to LLM performance, examining how developers are pushing the boundaries of efficiency, memory management, and training stability. Discover the evolution and intricacies of: Attention Mechanisms: From Multi-Head Attention (MHA) to the more efficient Grouped-Query Attention (GQA), and innovative approaches like Multi-Head Latent Attention (MLA) used in DeepSeek-V3, which compresses key and value tensors for memory savings. We also delve into Sliding Window Attention from Gemma 3, which restricts context size for local efficiency. Normalization Layers: Explore the shift from LayerNorm to RMSNorm and the crucial placement of these layers (Pre-Norm, Post-Norm) as seen in OLMo 2 and Gemma 3, including the addition of QK-Norm for enhanced training stability. Mixture-of-Experts (MoE): Understand why this approach has seen a significant resurgence in 2025. Learn how MoE, as implemented in models like DeepSeek-V3, Llama 4, and Qwen3's sparse variants, allows for massive total parameter counts (e.g., DeepSeek-V3's 671 billion parameters) while activating only a small subset (e.g., 37 billion) per inference step for remarkable efficiency. Positional Embeddings: Discover how positional information is handled, from rotational positional embeddings (RoPE) to the radical concept of No Positional Embeddings (NoPE) in SmolLM3, whic…