{"podcast":{"title":"Machine Learning Street Talk (MLST)","slug":"machine-learning-street-talk","podcast_index_feed_id":781643,"rss_url":"https://anchor.fm/s/1e4a0eac/podcast/rss","website_url":"https://podcasters.spotify.com/pod/show/machinelearningstreettalk","image_url":"https://d3t3ozftmdmh3i.cloudfront.net/staging/podcast_uploaded_nologo/4981699/4981699-1757416025703-f026fa81b6d04.jpg","author":"Machine Learning Street Talk (MLST)","episode_count":250,"summary":"Welcome! We engage in fascinating discussions with pre-eminent figures in the AI field. Our flagship show covers current affairs in AI, cognitive science, neuroscience and philosophy of mind with in-depth analysis. Our approach is unrivalled in terms of scope and rigour – we believe in intellectual diversity in AI, and we touch on all of the main ideas in the field with the hype surgically removed. MLST is run by Tim Scarfe, Ph.D (https://www.linkedin.com/in/ecsquizor/) and features regular appearances from MIT Doctor of Philosophy Keith Duggar (https://www.linkedin.com/in/dr-keith-duggar/).","last_synced_at":null,"page_url":"https://stenobird.com/podcast/machine-learning-street-talk"},"episode":{"title":"Prof. Randall Balestriero - LLMs without pretraining and SSL","slug":"prof-randall-balestriero-llms-without-pretraining-and-ssl","published_at":"2025-04-23T14:16:32+00:00","page_url":"https://stenobird.com/podcast/machine-learning-street-talk/prof-randall-balestriero-llms-without-pretraining-and-ssl","show_page_url":"https://stenobird.com/podcast/machine-learning-street-talk","url":"https://podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/Prof--Randall-Balestriero---LLMs-without-pretraining-and-SSL-e31ta64","audio_url":"https://anchor.fm/s/1e4a0eac/podcast/play/101672580/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2025-3-23%2Fc31e9f0d-1a79-4a2a-df8d-c8608ab858e1.mp3","summary":"Randall Balestriero joins the show to discuss some counterintuitive findings in AI. He shares research showing that huge language models, even when started from scratch (randomly initialized) without massive pre-training, can learn specific tasks like sentiment analysis surprisingly well, train stably, and avoid severe overfitting, sometimes matching the performance of costly pre-trained models. This raises questions about when giant pre-training efforts are truly worth it. He also talks about how self-supervised learning (where models learn from data structure itself) and traditional supervised learning (using labeled data) are fundamentally similar, allowing researchers to apply decades of supervised learning theory to improve newer self-supervised methods. Finally, Randall touches on fairness in AI models used for Earth data (like climate prediction), revealing that these models can be biased, performing poorly in specific locations like islands or coastlines even if they seem accurate overall, which has important implications for policy decisions based on this data. SPONSOR MESSAGES: *** Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. They are hiring a Chief Engineer and ML engineers. Events in Zurich. Goto https://tufalabs.ai/ *** TRANSCRIPT + SHOWNOTES: https://www.dropbox.com/scl/fi/n7yev71nsjso71jyjz1fy/RANDALLNEURIPS.pdf?rlkey=0dn4injp1sc4ts8njwf3wfmxv&dl=0 TOC: 1. Model Training Efficiency and Scale [00:00:00] 1.1 Training Stability of Large Models on Small Datasets [00:04:09] 1.2 Pre-training vs Random Initialization Performance Comparison [00:07:58] 1.3 Task-Specific Models vs General LLMs Efficiency 2. Learning Paradigms and Data Distribution [00:10:35] 2.1 Fair Language Mode…","meta_description":"Randall Balestriero joins the show to discuss some counterintuitive findings in AI. He shares research showing that huge language models, even when starte…","key_points":[],"chapters":[],"topics":[],"duration_seconds":2070,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/machine-learning-street-talk/episodes/prof-randall-balestriero-llms-without-pretraining-and-ssl/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/machine-learning-street-talk/prof-randall-balestriero-llms-without-pretraining-and-ssl.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}