# LLM Unpacked: A Deep Dive into Modern AI Architectures

Page: https://stenobird.com/podcast/smart-enterprises-ai-frontiers-7077810/llm-unpacked-a-deep-dive-into-modern-ai-architectures
Text version: https://stenobird.com/podcast/smart-enterprises-ai-frontiers-7077810/llm-unpacked-a-deep-dive-into-modern-ai-architectures.md
Podcast: [Smart Enterprises: AI Frontiers](https://stenobird.com/podcast/smart-enterprises-ai-frontiers-7077810)
Published: 2025-07-29T22:17:15+00:00
Episode link: https://podcasters.spotify.com/pod/show/ali-mehedi0/episodes/LLM-Unpacked-A-Deep-Dive-into-Modern-AI-Architectures-e3670qd
Audio file: https://anchor.fm/s/fcb30f54/podcast/play/106184973/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2025-6-29%2F404788191-44100-2-6d2666ff4b505.m4a
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/smart-enterprises-ai-frontiers-7077810/episodes/llm-unpacked-a-deep-dive-into-modern-ai-architectures
Duration seconds: 2501

## Resource

Join us for an insightful exploration into the cutting-edge design of today's Large Language Models . Seven years on from the original GPT architecture, have we truly seen groundbreaking changes, or are we simply refining existing foundations? This podcast focuses on the architectural developments that define flagship open models in 2025, moving beyond benchmark performance or training algorithms. In this episode, we'll unpack the key ingredients contributing to LLM performance, examining how developers are pushing the boundaries of efficiency, memory management, and training stability. Discover the evolution and intricacies of: Attention Mechanisms: From Multi-Head Attention (MHA) to the more efficient Grouped-Query Attention (GQA), and innovative approaches like Multi-Head Latent Attention (MLA) used in DeepSeek-V3, which compresses key and value tensors for memory savings. We also delve into Sliding Window Attention from Gemma 3, which restricts context size for local efficiency. Normalization Layers: Explore the shift from LayerNorm to RMSNorm and the crucial placement of these layers (Pre-Norm, Post-Norm) as seen in OLMo 2 and Gemma 3, including the addition of QK-Norm for enhanced training stability. Mixture-of-Experts (MoE): Understand why this approach has seen a significant resurgence in 2025. Learn how MoE, as implemented in models like DeepSeek-V3, Llama 4, and Qwen3's sparse variants, allows for massive total parameter counts (e.g., DeepSeek-V3's 671 billion parameters) while activating only a small subset (e.g., 37 billion) per inference step for remarkable efficiency. Positional Embeddings: Discover how positional information is handled, from rotational positional embeddings (RoPE) to the radical concept of No Positional Embeddings (NoPE) in SmolLM3, whic…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/smart-enterprises-ai-frontiers-7077810/episodes/llm-unpacked-a-deep-dive-into-modern-ai-architectures/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/smart-enterprises-ai-frontiers-7077810/llm-unpacked-a-deep-dive-into-modern-ai-architectures.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.