Episode
DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
- Podcast
- Daily Paper Cast
- Published
- Jul 28, 2026
- Duration seconds
- 1296
- Processing state
not_requested- Canonical source
- https://share.transistor.fm/s/d4f30016
Actions
POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/dataprep-bench-benchmarking-llms-as-training-data-preparators/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/daily-paper-cast-7079649/dataprep-bench-benchmarking-llms-as-training-data-preparators.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
🤗 Upvotes: 44 | cs.LG, cs.CL Authors: Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, Zhaoyang Han, Xiaochen Ma, Bohan Zeng, Ruichuan An, Conghui He, Wentao Zhang Title: DataPrep-Bench: Benchmarking LLMs as Training Data Preparators Arxiv: http://arxiv.org/abs/2607.20465v1 Abstract: The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout, "quality" refers to downstream training utility rather than surface-level textual properties. We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD betw…