# DataPrep-Bench: Benchmarking LLMs as Training Data Preparators

Page: https://stenobird.com/podcast/daily-paper-cast-7079649/dataprep-bench-benchmarking-llms-as-training-data-preparators
Text version: https://stenobird.com/podcast/daily-paper-cast-7079649/dataprep-bench-benchmarking-llms-as-training-data-preparators.md
Podcast: [Daily Paper Cast](https://stenobird.com/podcast/daily-paper-cast-7079649)
Published: 2026-07-28T03:37:14+00:00
Episode link: https://share.transistor.fm/s/d4f30016
Audio file: https://media.transistor.fm/d4f30016/f39b630b.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/dataprep-bench-benchmarking-llms-as-training-data-preparators
Duration seconds: 1296

## Resource

🤗 Upvotes: 44 | cs.LG, cs.CL Authors: Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang Title: DataPrep-Bench: Benchmarking LLMs as Training Data Preparators Arxiv: http://arxiv.org/abs/2607.20465v1 Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout, "quality" refers to downstream training utility rather than surface-level textual properties. We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD betw…

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/dataprep-bench-benchmarking-llms-as-training-data-preparators/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/daily-paper-cast-7079649/dataprep-bench-benchmarking-llms-as-training-data-preparators.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.