AssemblyAI released Voice Standard Bench, an MIT-licensed Go CLI that compares LLMs on four voice transcript tasks: summarization, information extraction, text correction, and dictation. It reports quality and latency by transcript length, from short messages to hour-long meetings.
A new arXiv paper introduces MTVA-Bench, a benchmark that evaluates the language model inside cascaded voice agent pipelines under realistic phone-call conditions. It tests 49 agents across 490 reviewed scenarios in 7 languages, combining deterministic tool-call checks with two transcript-citing LLM judges. In a seven-model study, similar tool-selection accuracy masked a 24.4-point spread in overall scores, driven by argument values, action ordering, and rule compliance.
A new arXiv paper introduces MTVA-Bench, a benchmark that evaluates the language model inside cascaded voice agent pipelines under realistic phone-call conditions. It tests 49 agents across 490 reviewed scenarios in 7 languages, combining deterministic tool-call checks with two transcript-citing LLM judges. In a seven-model study, similar tool-selection accuracy masked a 24.4-point spread in overall scores, driven by argument values, action ordering, and rule compliance.
A developer built smokebench, an open-source TUI for quickly benchmarking LLMs on local setups, addressing the gap between public leaderboards and real-world quantized or fine-tuned models. It supports any OpenAI or Anthropic compatible endpoint and tests across 8 categories including math, code, and creative writing.
A developer built smokebench, an open-source TUI for quickly benchmarking LLMs on local setups, addressing the gap between public leaderboards and real-world quantized or fine-tuned models. It supports any OpenAI or Anthropic compatible endpoint and tests across 8 categories including math, code, and creative writing.
A new benchmark called Real World VoiceEQ has been launched to measure the human quality of voice AI, addressing the gap between traditional metrics like word error rate and perceived naturalness. The benchmark includes leaderboards and audio samples for speech models, aiming to push voice AI beyond current saturation in established benchmarks.
A new benchmark called Real World VoiceEQ has been launched to measure the human quality of voice AI, addressing the gap between traditional metrics like word error rate and perceived naturalness. The benchmark includes leaderboards and audio samples for speech models, aiming to push voice AI beyond current saturation in established benchmarks.
Developer releases LLM Speedtest, an open-source desktop app for benchmarking local LLM inference
Developer releases LLM Speedtest, an open-source desktop app for benchmarking local LLM inference
A benchmarking study evaluates GPT-4.1 and GPT-5 as LLM-as-a-Judge for voice-agent conversations in telecom and retail. It tests three evaluation configurations for conversational quality and safety. This research informs practitioners on reliability and calibration of automated judgments versus human oversight.
A benchmarking study evaluates GPT-4.1 and GPT-5 as LLM-as-a-Judge for voice-agent conversations in telecom and retail. It tests three evaluation configurations for conversational quality and safety. This research informs practitioners on reliability and calibration of automated judgments versus human oversight.