Scale AI launches Voice Showdown, the first real-world benchmark for voice AI — and the results are humbling for some top models

Voice AI is moving faster than the tools we use to measure it. Every major AI lab — OpenAI, Google DeepMind, Anthropic, xAI — is racing to ship voice models capable of natural, real-time conversation.

But the benchmarks used to evaluate those models are largely still running on synthetic speech, English-only prompts, and scripted test sets that bear little resemblance to how people actually talk.

Scale AI, the large data annotation startup whose founder was poached by Meta last year to lead its Superintelligence Lab, is still going strong and tackling the problem head on: today it launches Voice Showdown, what it calls the first global preference-based arena designed to benchmark voice AI through the lens of real human interaction.

This product offers a unique strategic value to users: free access to the world’s leading frontier models. Through Scale’s ChatLab platform, users can interact with high-tier models—which typically require multiple $20-per-month subscriptions—at no cost. In exchange, users participate in occasional blind, head-to-head "battles" to choose which of two anonymized leading voice models offers a better experience, providing data for the industry’s most authentic, human-preference leaderboard of voice AI models.

"Voice AI is really the fastest moving frontier in AI right now," said Janie Gu, product manager for Showdown at Scale AI. "But the way that we evaluate voice models hasn't kept up."

The results, drawn from thousands of spontaneous voice conversations across more than 60 languages, reveal capability gaps that other benchmarks have consistently missed.

How Scale's Voice Showdown works

Voice Showdown is built on ChatLab, Scale's model-agnostic chat platform where users can freely interact with whichever frontier AI model they choose — for free — within a single app. The platform has been available to Scale's global community of over 500,000 annotators, with roughly 300,000 having submitted at least one prompt. Scale is opening the platform to a public waitlist today.

The evaluation mechanism is elegant in its simplicity: while a user is having a natural voice conversation with a model, the system occasionally — on fewer than 5% of all voice prompts — surfaces a blind side-by-side comparison. The same prompt is sent to a second, anonymous model, and the user picks which response they prefer.

This design solves three problems that plague existing voice benchmarks.

First, every prompt comes from real human speech — with accents, background noise, half-finished sentences, and conversational filler — rather than synthesized audio generated from text.

Second, the platform spans more than 60 languages across 6 continents, with over a third of battles occurring in non-English languages including Spanish, Arabic, Japanese, Portuguese, Hindi, and French.

Third, because battles occur within users' actual daily conversations, 81% of prompts are conversational or open-ended — questions without a single correct answer. That rules out automated scoring and makes human preference the only credible signal.

Voice Showdown currently runs two evaluation modes: Dictate (users speak, models respond with text) and Speech-to-Speech, or S2S (Speech-to-Speech, users speak, models talk back). A third mode — Full Duplex, which captures real-time, interruptible conversation — is in development.

Incentive-aligned voting

One design detail sets Voice Showdown apart from Chatbot Arena (LM Arena), the text benchmark it most closely resembles. In LM Arena, critics have noted that users sometimes cast throwaway votes with little stake in the outcome. Voice Showdown addresses this directly: after a user votes for the model they preferred, the app switches them to that model for the rest of their conversation. If you voted for GPT-4o Audio over Gemini, you're now talking to GPT-4o Audio. That alignment of consequence with preference discourages casual or dishonest voting.

The system also controls for confounds that could corrupt comparisons: both model responses begin streaming simultaneously (eliminating speed bias), voice gender is matched across both options (eliminating gender preference bias), and neither model is identified by name during voting.

The new Voice AI leaderboard every enterprise decision-maker should pay attention to

Voice Showdown launches with 11 frontier models evaluated across 52 model-voice pairs as of March 18, 2026. Not all models support both evaluation modes — the Dictate leaderboard includes 8 models, while S2S includes 6.

Dictate Leaderboard (Speech-In, Text-Out)

In this mode, users provide a spoken prompt and evaluate two side-by-side text responses. Here are the baseline scores:

1. Gemini 3 Pro (1073)

2. Gemini 3 Flash (1068)

3. GPT-4o Audio (1019)

4. Qwen 3 Omni (1000)

5. Voxtral Small (925)

6. Gemma 3n (918)

7. GPT Realtime (875)

8. Phi-4 Multimodal (729)

Note: Gemini 3 Pro and Gemini 3 Flash are statistically tied for the top rank.

Speech-to-Speech (S2S) Leaderboard

In this mode, users speak to the model and evaluate two competing audio responses. Also baselines:

1. Gemini 2.5 Flash Audio (1060)

2. GPT-4o Audio (1059)

3. Grok Voice (1024)

4. Qwen 3 Omni (1000)

5. GPT Realtime (962)

6. GPT Realtime 1.5 (920)

Note: Gemini 2.5 Flash Audio and GPT-4o Audio are statistically tied for the top rank in baseline evaluations.

Dictate rankings are led by Google's Gemini 3 Pro and Gemini 3 Flash, which are statistically tied at #1 with Elo scores around 1,043-1,044 after style controls.

GPT-4o Audio holds a clear third place. Open-weight models including Gemma3n, Voxtral Small, and Phi-4 Multimodal trail significantly.

Speech-to-Speech (S2S) rankings show a tighter race at the top, with Gemini 2.5 Flash Audio and GPT-4o Audio statistically tied at #1 in the baseline rankings.

After adjusting for response length and formatting — factors that can inflate perceived quality — GPT-4o Audio pulls ahead (1,102 Elo vs. 1,075 for Gemini 2.5 Flash Audio).

Grok Voice jumps to a close second at 1,093 under style controls, suggesting its raw #3 ranking undersells its actual performance quality.

Qwen 3 Omni, the open-weight model from Alibaba's Qwen team, performs better on pure preference than its popularity would suggest — ranking fourth in both modes, ahead of several higher-profile names.

"When people come in, they go for the big names," Gu noted. "But for preference, lesser-known models like Qwen actually pull ahead."

Surprised revealed by real-world preference data

Beyond rankings, Voice Showdown's real value is in the failure diagnostics — and those paint a more complicated picture of voice AI than most leaderboards reveal.

The multilingual gap is worse than you think

Language robustness is the starkest differentiator across models. In Dictate, Gemini 3 models lead across essentially every language tested.

In S2S, the winner depends heavily on which language is being spoken: GPT-4o Audio leads in Arabic and Turkish; Gemini 2.5 Flash Audio is strongest in French; Grok Voice is competitive in Japanese and Portuguese.

But the more alarming finding is how frequently some models simply stop responding in the user's language at all.

GPT Realtime 1.5 — OpenAI's newer real-time voice model — responds in English to non-English prompts roughly 20% of the time, even on high-resource, officially supported languages like Hindi, Spanish, and Turkish.

Its predecessor, GPT Realtime, mismatches at about half that rate (~10%). Gemini 2.5 Flash Audio and GPT-4o Audio sit at ~7%.

The phenomenon runs both directions: some models carry non-English context from earlier in a conversation into an English turn, or simply mishear a prompt and generate an unrelated response in the wrong language entirely.

User verbatims from the platform capture the frustration bluntly: "I said I have an interview today with Quest Management and instead of answering, it gave me information about 'Risk Management.'"

"GPT Realtime 1.5 thought I was speaking incoherently and recommended mental health assistance, while Qwen 3 Omni correctly identified I was speaking a Nigerian local language."

The reason existing benchmarks miss this: they're built on synthetic speech optimized for clean acoustic conditions, and they're rarely multilingual. Real speakers in real environments — with background noise, short utterances, and regional accents — break speech understanding in ways lab conditions don't anticipate.

Voice selection is more than aesthetics

Voice Showdown evaluates models not just at the model level but at the individual voice level — and the variance within a single model's voice catalog is striking.

For one unnamed model in the study, the best-performing voice won 30 percentage points more often than the worst-performing voice from the same underlying model. Both voices share the same reasoning and generation backend. The difference is purely in audio presentation.

The top-performing voices tend to win or lose on audio understanding and content completeness — whether the model heard you correctly and answered fully. But speech quality remains a deciding factor at the voice selection level, particularly when models are otherwise comparable. "Voice directly shapes how users evaluate the interaction," Gu said.

Models degrade in conversation

Most benchmarks test a single turn. Voice Showdown tests how models hold up across extended conversations — and the results aren't flattering.

On Turn 1, content quality accounts for 23% of model failures. By Turn 11 and beyond, it becomes the primary failure mode at 43%. Most models see their win rates decline as conversations extend, struggling to maintain coherence across multiple exchanges.

GPT Realtime variants are an exception, marginally improving on later turns — consistent with their known strengths on longer contexts, and their documented weakness on the brief, noisy utterances that dominate early interactions.

Prompt length shows a complementary pattern: short prompts (under 10 seconds) are dominated by audio understanding failures (38%), while long prompts (over 40 seconds) shift the primary failure toward content quality (31%). Shorter audio gives models less acoustic context to parse; longer requests are understood but harder to answer well.

Why some voice AI models lose

After every S2S comparison, users tag why they preferred one response over the other across three axes: audio understanding, content quality, and speech output. The failure signatures differ meaningfully by model.

Qwen 3 Omni's losses cluster around speech generation — its reasoning is competitive, but users are put off by how it sounds. GPT Realtime 1.5's losses are dominated by audio understanding failures (51%), consistent with its language-switching behavior on challenging prompts. Grok Voice's failures are more balanced across all three axes, indicating no single dominant weakness but no particular strength either.

What's next

The current leaderboard covers turn-based interaction — you speak, the model responds, repeat. But real voice conversations don't work that way. People interrupt, change direction mid-sentence, and talk over each other.

Scale says Full Duplex evaluation — designed to capture these real-time dynamics through human preference rather than scripted scenarios or automated metrics — is coming to Showdown next. No existing benchmark captures full-duplex interaction through organic human preference data.

The leaderboard is live at scale.com/showdown. A public waitlist to join ChatLab and vote on comparisons is open today, with users receiving free access to frontier voice models including GPT-4o, Gemini, and Grok in exchange for occasional preference votes.