{"podcast":{"title":"Machine Learning Street Talk (MLST)","slug":"machine-learning-street-talk","podcast_index_feed_id":781643,"rss_url":"https://anchor.fm/s/1e4a0eac/podcast/rss","website_url":"https://podcasters.spotify.com/pod/show/machinelearningstreettalk","image_url":"https://d3t3ozftmdmh3i.cloudfront.net/staging/podcast_uploaded_nologo/4981699/4981699-1757416025703-f026fa81b6d04.jpg","author":"Machine Learning Street Talk (MLST)","episode_count":250,"summary":"Welcome! We engage in fascinating discussions with pre-eminent figures in the AI field. Our flagship show covers current affairs in AI, cognitive science, neuroscience and philosophy of mind with in-depth analysis. Our approach is unrivalled in terms of scope and rigour – we believe in intellectual diversity in AI, and we touch on all of the main ideas in the field with the hype surgically removed. MLST is run by Tim Scarfe, Ph.D (https://www.linkedin.com/in/ecsquizor/) and features regular appearances from MIT Doctor of Philosophy Keith Duggar (https://www.linkedin.com/in/dr-keith-duggar/).","last_synced_at":null,"page_url":"https://stenobird.com/podcast/machine-learning-street-talk"},"episode":{"title":"He Co-Invented the Transformer. Now: Continuous Thought Machines - Llion Jones and Luke Darlow [Sakana AI]","slug":"he-co-invented-the-transformer-now-continuous-thought-machines-llion-jones-and-luke-darlow-sakana-ai","published_at":"2025-11-23T17:36:59+00:00","page_url":"https://stenobird.com/podcast/machine-learning-street-talk/he-co-invented-the-transformer-now-continuous-thought-machines-llion-jones-and-luke-darlow-sakana-ai","show_page_url":"https://stenobird.com/podcast/machine-learning-street-talk","url":"https://podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/He-Co-Invented-the-Transformer--Now-Continuous-Thought-Machines---Llion-Jones-and-Luke-Darlow-Sakana-AI-e3bbt96","audio_url":"https://traffic.megaphone.fm/APO6903071163.mp3","summary":"Llion Jones, co-inventor of the Transformer, argues that current scaling laws are trapping AI in a local minimum of pattern matching rather than true reasoning. He and Luke Darlow introduce Continuous Thought Machines (CTM) as a biologically-inspired alternative that allows models to 'ponder' and process information step-by-step.","meta_description":"Transformer co-inventor Llion Jones discusses moving beyond scaling laws toward Continuous Thought Machines and adaptive computation at Sakana AI.","key_points":["Main idea: The Transformer architecture excels at pattern recognition but lacks the ability to genuinely 'think' through complex, multi-step problems","Failure mode: Current LLMs use 'brute force' scaling to mimic complex shapes or logic, effectively faking understanding through high-dimensional straight lines","Practical takeaway: Continuous Thought Machines (CTM) enable adaptive computation, allowing a model to spend more time on harder tasks by 'walking' through a problem","Technical insight: CTM uses a self-bootstrapping mechanism where the model is trained to predict only the next step in a sequence it has already partially mastered","Research philosophy: Moving away from 'architecture lottery' and fixed-compute models toward systems that can naturally backtrack and correct errors"],"chapters":[{"start_ms":65000,"title":"Stepping Back from Transformers","summary":"Llion Jones discusses the shift in AI research from the open-ended exploration of the Transformer era to the current era of reduced research freedom."},{"start_ms":400000,"title":"The Era of Technology Capture","summary":"An exploration of how the ubiquity of the Transformer architecture may be creating a 'local minimum' in AI development."},{"start_ms":1035000,"title":"The Limits of Scaling","summary":"A critique of how current models can produce clearly incorrect outputs despite massive scale, signaling a fundamental architectural flaw."},{"start_ms":1720000,"title":"Introducing Continuous Thought Machines","summary":"A deep dive into the CTM architecture and how it differs from the 'instantaneous' processing of standard Transformers."},{"start_ms":2040000,"title":"Adaptive Computation & Maze Solving","summary":"Using the maze analogy to explain how CTM can use attention to retrieve information and 'think' through steps sequentially."},{"start_ms":2380000,"title":"Technical Deep Dive: CTM Architecture","summary":"A technical look at neuron synchronization and measuring activations over time within the CTM framework."},{"start_ms":3345000,"title":"The Future of AI Research","summary":"Advice for young researchers on navigating the 'maze' of AI and the importance of pursuing passion-driven, bottom-up research."}],"topics":["Transformer Architecture","Continuous Thought Machines","Sakana AI","Adaptive Computation","Neural Network Architecture","Machine Learning Research","Artificial General Intelligence","Pattern Recognition"],"duration_seconds":4359,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/machine-learning-street-talk/episodes/he-co-invented-the-transformer-now-continuous-thought-machines-llion-jones-and-luke-darlow-sakana-ai/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/machine-learning-street-talk/he-co-invented-the-transformer-now-continuous-thought-machines-llion-jones-and-luke-darlow-sakana-ai.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}