{"podcast":{"title":"Latent Space: The AI Engineer Podcast","slug":"latent-space-ai-engineer","podcast_index_feed_id":6058902,"rss_url":"https://api.substack.com/feed/podcast/1084089.rss","website_url":"https://www.latent.space/podcast","image_url":"https://substackcdn.com/feed/podcast/1084089/ca7468da5614a246d2906ee8926f6de7.jpg","author":"Latent.Space","episode_count":204,"summary":"The AI Engineer newsletter + Top technical AI podcast. How leading labs build Agents, Models, Infra, & AI for Science. See https://latent.space/about for highlights from Greg Brockman, Andrej Karpathy, George Hotz, Simon Willison, Soumith Chintala et al!","last_synced_at":null,"page_url":"https://stenobird.com/podcast/latent-space-ai-engineer"},"episode":{"title":"[State of Code Evals] After SWE-bench, Code Clash & SOTA Coding Benchmarks recap — John Yang","slug":"state-of-code-evals-after-swe-bench-code-clash-sota-coding-benchmarks-recap-john-yang","published_at":"2025-12-31T16:00:00+00:00","page_url":"https://stenobird.com/podcast/latent-space-ai-engineer/state-of-code-evals-after-swe-bench-code-clash-sota-coding-benchmarks-recap-john-yang","show_page_url":"https://stenobird.com/podcast/latent-space-ai-engineer","url":"https://www.latent.space/p/state-of-code-evals-after-swe-bench","audio_url":"https://api.substack.com/feed/podcast/186610569/3c925296f32115c0d7354d7da15dcee7.mp3","summary":"From creating SWE-bench in a Princeton basement to shipping CodeClash , SWE-bench Multimodal , and SWE-bench Multilingual , John Yang has spent the last year and a half watching his benchmark become the de facto standard for evaluating AI coding agents—trusted by Cognition (Devin), OpenAI, Anthropic, and every major lab racing to solve software engineering at scale. We caught up with John live at NeurIPS 2025 to dig into the state of code evals heading into 2026: why SWE-bench went from ignored (October 2023) to the industry standard after Devin’s launch (and how Walden emailed him two weeks before the big reveal), how the benchmark evolved from Django-heavy to nine languages across 40 repos (JavaScript, Rust, Java, C, Ruby), why unit tests as verification are limiting and long-running agent tournaments might be the future (CodeClash: agents maintain codebases, compete in arenas, and iterate over multiple rounds), the proliferation of SWE-bench variants (SWE-bench Pro, SWE-bench Live, SWE-Efficiency, AlgoTune, SciCode) and how benchmark authors are now justifying their splits with curation techniques instead of just “more repos,” why Tau-bench’s “impossible tasks” controversy is actually a feature not a bug (intentionally including impossible tasks flags cheating), the tension between long autonomy (5-hour runs) vs. interactivity (Cognition’s emphasis on fast back-and-forth), how Terminal-bench unlocked creativity by letting PhD students and non-coders design environments beyond GitHub issues and PRs, the academic data problem (companies like Cognition and Cursor have rich user interaction data, academics need user simulators or compelling products like LMArena to get similar signal), and his vision for CodeClash as a testbed for human-AI collaboration —freeze model ca…","meta_description":"From creating SWE-bench in a Princeton basement to shipping CodeClash , SWE-bench Multimodal , and SWE-bench Multilingual , John Yang has spent the last y…","key_points":[],"chapters":[],"topics":[],"duration_seconds":1065,"processing_state":"processed","actions":[{"name":"request_transcript","method":"POST","url":"https://stenobird.com/v1/public/podcasts/latent-space-ai-engineer/episodes/state-of-code-evals-after-swe-bench-code-clash-sota-coding-benchmarks-recap-john-yang/transcription-requests","description":"Idempotently request low-priority transcript generation for this episode."},{"name":"read_markdown","method":"GET","url":"https://stenobird.com/podcast/latent-space-ai-engineer/state-of-code-evals-after-swe-bench-code-clash-sota-coding-benchmarks-recap-john-yang.md","description":"Read the agent-friendly Markdown representation of this episode resource."}]}}