# EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents Page: https://stenobird.com/podcast/daily-paper-cast-7079649/eva-bench-a-new-end-to-end-framework-for-evaluating-voice-agents Text version: https://stenobird.com/podcast/daily-paper-cast-7079649/eva-bench-a-new-end-to-end-framework-for-evaluating-voice-agents.md Podcast: [Daily Paper Cast](https://stenobird.com/podcast/daily-paper-cast-7079649) Published: 2026-05-15T05:00:54+00:00 Episode link: https://share.transistor.fm/s/3a90cf54 Audio file: https://media.transistor.fm/3a90cf54/3a348941.mp3 Processing state: not_requested JSON: https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/eva-bench-a-new-end-to-end-framework-for-evaluating-voice-agents Duration seconds: 1519 ## Resource đŸ¤— Upvotes: 57 | cs.SD, cs.AI, cs.CL, cs.LG Authors: Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin, Joseph Marinier, Hari Subramani, Anil Madamala, Sridhar Krishna Nemala, Srinivas Sunkara Title: EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents Arxiv: http://arxiv.org/abs/2605.13841v1 Abstract: Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to different agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both… ## Actions - request_transcript: `POST https://stenobird.com/v1/public/podcasts/daily-paper-cast-7079649/episodes/eva-bench-a-new-end-to-end-framework-for-evaluating-voice-agents/transcription-requests` — Idempotently request low-priority transcript generation for this episode. - read_markdown: `GET https://stenobird.com/podcast/daily-paper-cast-7079649/eva-bench-a-new-end-to-end-framework-for-evaluating-voice-agents.md` — Read the agent-friendly Markdown representation of this episode resource. A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed. ## Transcript Full transcripts are not published on public pages unless there is a clear rights basis.