Episode

Making deep learning perform real algorithms with Category Theory (Andrew Dudzik, Petar Velichkovich, Taco Cohen, Bruno Gavranović, Paul Lessard)

Podcast

Machine Learning Street Talk (MLST)

Published

Dec 22, 2025

Duration seconds

2637

Processing state

processed

Canonical source

https://podcasters.spotify.com/pod/show/machinelearningstreettalk/episodes/Making-deep-learning-perform-real-algorithms-with-Category-Theory-Andrew-Dudzik--Petar-Velichkovich--Taco-Cohen--Bruno-Gavranovi--Paul-Lessard-e3cmhua

Audio

https://anchor.fm/s/1e4a0eac/podcast/play/112985482/https%3A%2F%2Fd3ctxlq1ktw2nl.cloudfront.net%2Fstaging%2F2025-11-22%2F414834607-44100-2-d15a91341d3b9.mp3

JSON

/v1/public/podcasts/machine-learning-street-talk/episodes/making-deep-learning-perform-real-algorithms-with-category-theory-andrew-dudzik-petar-velichkovich-taco-cohen-bruno-gavranovi-paul-lessard

Markdown

/podcast/machine-learning-street-talk/making-deep-learning-perform-real-algorithms-with-category-theory-andrew-dudzik-petar-velichkovich-taco-cohen-bruno-gavranovi-paul-lessard.md

Actions

• POST https://stenobird.com/v1/public/podcasts/machine-learning-street-talk/episodes/making-deep-learning-perform-real-algorithms-with-category-theory-andrew-dudzik-petar-velichkovich-taco-cohen-bruno-gavranovi-paul-lessard/transcription-requests
  Idempotently request low-priority transcript generation for this episode.
• GET https://stenobird.com/podcast/machine-learning-street-talk/making-deep-learning-perform-real-algorithms-with-category-theory-andrew-dudzik-petar-velichkovich-taco-cohen-bruno-gavranovi-paul-lessard.md
  Read the agent-friendly Markdown representation of this episode resource.

Summary

Modern LLMs struggle with basic algorithmic tasks like addition because they rely on pattern recognition rather than internal computational logic. This discussion explores using Category Theory to move deep learning from empirical 'alchemy' to a principled engineering discipline.

Topics

• Category Theory
• Deep Learning
• Large Language Models
• Geometric Deep Learning
• Algorithmic Reasoning
• Neural Network Architecture
• Mathematical Logic
• Computational Complexity

Highlights

• Main idea: LLMs lack the internal machinery for operations like carrying digits, making them prone to failure on simple arithmetic
• Failure mode: Relying on tool-use or chain-of-thought acts as a patch rather than fixing the underlying architectural inability to process algorithms
• Practical takeaway: Moving from 'Analytic' to 'Synthetic' mathematics can help models internalize rules rather than just observing data
• Main idea: Category Theory offers a 'Periodic Table' for neural networks, allowing for the composition of complex operations from simpler ones
• Technical vision: The goal is to build neural networks that can execute computation in a way that is mathematically reasonble and robust

Chapters

1. 1:00 The Failure of LLM Addition: Why language models fail at basic arithmetic despite appearing competent through pattern recognition.
2. 4:35 Broadening Geometric Deep Learning: The need to expand the lens of geometric deep learning to include more complex algorithmic structures.
3. 7:35 Invariance and Preconditions: Discussing the challenges of maintaining computational invariants when pushing data through functions.
4. 10:40 Compositionality and Color Violations: Using the analogy of 'colors' to explain how types must match for successful algebraic composition.
5. 14:00 Moving Beyond Deep Learning Alchemy: The transition from ad hoc architectural design choices to a principled, theory-driven foundation.
6. 17:35 Aligning Models to Algorithms: The difficulty of aligning neural architectures with classical algorithmic computation and permutation invariance.
7. 21:00 The Complexity of Multiplication: Why scaling inputs makes the translation between embeddings and algorithms increasingly prone to failure.
8. 24:20 Synthetic Mathematics: A shift toward a mathematical framework where principles are abstracted rather than just observed.