Episode
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
- Podcast
- Daily Paper Cast
- Published
- May 15, 2026
- Duration seconds
- 1519
- Processing state
not_requested- Canonical source
- https://share.transistor.fm/s/3a90cf54
Actions
POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/eva-bench-a-new-end-to-end-framework-for-evaluating-voice-agents/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/daily-paper-cast-7079649/eva-bench-a-new-end-to-end-framework-for-evaluating-voice-agents.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
đŸ¤— Upvotes: 57 | cs.SD, cs.AI, cs.CL, cs.LG Authors: Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin, Joseph Marinier, Hari Subramani, Anil Madamala, Sridhar Krishna Nemala, Srinivas Sunkara Title: EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents Arxiv: http://arxiv.org/abs/2605.13841v1 Abstract: Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to different agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both…