# SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Page: https://stenobird.com/podcast/daily-paper-cast-7079649/smelt-scaling-laws-for-compute-matched-moe-looped-transformers
Text version: https://stenobird.com/podcast/daily-paper-cast-7079649/smelt-scaling-laws-for-compute-matched-moe-looped-transformers.md
Podcast: [Daily Paper Cast](https://stenobird.com/podcast/daily-paper-cast-7079649)
Published: 2026-09-02T08:18:16+00:00
Episode link: https://share.transistor.fm/s/d8dde809
Audio file: https://media.transistor.fm/d8dde809/8f49784f.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/smelt-scaling-laws-for-compute-matched-moe-looped-transformers
Duration seconds: 1268

## Resource

🤗 Upvotes: 53 | cs.LG Authors: Shaowen Wang, Ge Zhang, Kairong Luo, Yuhao Wu, Shaofan Liu, Jiaheng Liu, Wenhao Huang, Shen Yan, Jian Li Title: SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers Arxiv: http://arxiv.org/abs/2609.01343v1 Abstract: Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0\% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/smelt-scaling-laws-for-compute-matched-moe-looped-transformers/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/daily-paper-cast-7079649/smelt-scaling-laws-for-compute-matched-moe-looped-transformers.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.