{"podcast":{"title":"Machine Learning Street Talk (MLST)","slug":"machine-learning-street-talk","podcast_index_feed_id":781643,"rss_url":"https://anchor.fm/s/1e4a0eac/podcast/rss","website_url":"https://podcasters.spotify.com/pod/show/machinelearningstreettalk","image_url":"https://d3t3ozftmdmh3i.cloudfront.net/staging/podcast_uploaded_nologo/4981699/4981699-1757416025703-f026fa81b6d04.jpg","author":"Machine Learning Street Talk (MLST)","episode_count":250,"summary":"Welcome! We engage in fascinating discussions with pre-eminent figures in the AI field. Our flagship show covers current affairs in AI, cognitive science, neuroscience and philosophy of mind with in-depth analysis. Our approach is unrivalled in terms of scope and rigour – we believe in intellectual diversity in AI, and we touch on all of the main ideas in the field with the hype surgically removed. MLST is run by Tim Scarfe, Ph.D (https://www.linkedin.com/in/ecsquizor/) and features regular appearances from MIT Doctor of Philosophy Keith Duggar (https://www.linkedin.com/in/dr-keith-duggar/).","last_synced_at":null,"page_url":"https://stenobird.com/podcast/machine-learning-street-talk"},"episode":{"title":"ARC Prize v2 Launch! (Francois Chollet and Mike Knoop)","slug":"arc-prize-v2-launch-francois-chollet-and-mike-knoop","published_at":"2025-03-24T20:26:47+00:00","page_url":"https://stenobird.com/podcast/machine-learning-street-talk/arc-prize-v2-launch-francois-chollet-and-mike-knoop","show_page_url":"https://stenobird.com/podcast/machine-learning-street-talk","url":"https://podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/ARC-Prize-v2-Launch--Francois-Chollet-and-Mike-Knoop-e30k12p","audio_url":"https://anchor.fm/s/1e4a0eac/podcast/play/100319769/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2025-2-24%2F103fc293-4801-926b-df34-d8027ffb69d0.mp3","summary":"We are joined by Francois Chollet and Mike Knoop, to launch the new version of the ARC prize! In version 2, the challenges have been calibrated with humans such that at least 2 humans could solve each task in a reasonable task, but also adversarially selected so that frontier reasoning models can't solve them. The best LLMs today get negligible performance on this challenge. https://arcprize.org/ SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** TRANSCRIPT: https://www.dropbox.com/scl/fi/0v9o8xcpppdwnkntj59oi/ARCv2.pdf?rlkey=luqb6f141976vra6zdtptv5uj&dl=0 TOC: 1. ARC v2 Core Design & Objectives [00:00:00] 1.1 ARC v2 Launch and Benchmark Architecture [00:03:16] 1.2 Test-Time Optimization and AGI Assessment [00:06:24] 1.3 Human-AI Capability Analysis [00:13:02] 1.4 OpenAI o3 Initial Performance Results 2. ARC Technical Evolution [00:17:20] 2.1 ARC-v1 to ARC-v2 Design Improvements [00:21:12] 2.2 Human Validation Methodology [00:26:05] 2.3 Task Design and Gaming Prevention [00:29:11] 2.4 Intelligence Measurement Framework 3. O3 Performance & Future Challenges [00:38:50] 3.1 O3 Comprehensive Performance Analysis [00:43:40] 3.2 System Limitations and Failure Modes [00:49:30] 3.3 Program Synthesis Applications [00:53:00] 3.4 Future Development Roadmap REFS: [00:00:15] On the Measure of Intelligence, François Chollet https://arxiv.org/abs/1911.01547 [00:06:45] ARC Prize Foundation, François Chollet, Mike Knoop https://arcprize.org/ [00:12:50] OpenAI o3 model performance on ARC v1, ARC Prize Team https://arcprize.org/blog/oai-o3-pub-breakthrough [00:18:30] Chain-of-Thought…","meta_description":"We are joined by Francois Chollet and Mike Knoop, to launch the new version of the ARC prize! In version 2, the challenges have been calibrated with human…","key_points":[],"chapters":[],"topics":[],"duration_seconds":3255,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/machine-learning-street-talk/episodes/arc-prize-v2-launch-francois-chollet-and-mike-knoop/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/machine-learning-street-talk/arc-prize-v2-launch-francois-chollet-and-mike-knoop.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}