🏆 Nvidia Nemotron 3.5 Transcribe ranks #3 of 10 on the open-source Voice of India STT leaderboard at 6.59% OI-WER in Hindi.
Ahead of SraVaani, Gemma, OmniASR and Qwen.
Ahead of SraVaani, Gemma, OmniASR and Qwen.
Voice Of India Benchmark
Verified account This account is verified. Learn more@JoshTalksAI
Building the infrastructure to measure and improve AI that works for India in collaboration with AI4Bhārat at IIT Madras
302 Following 24K Followers 11K Posts
Latest posts 5
Can an AI voice agent handle a real Indian phone call as well as a person?
We tested 4 of them: Gemini, OpenAI, Grok and Inworld. A trained human agent took the same calls.
The answer, for now, is no. Not one AI sounded as human as the person.
Gemini came closest. But put Gemini and the human agent side by side, and listeners would still pick the human about 4 times out of 5.
OpenAI came third. Grok and Inworld are tied for fourth, too close to separate.
Based on 1,543 blind votes from Indian listeners, on calls in Indian English.
We tested 4 of them: Gemini, OpenAI, Grok and Inworld. A trained human agent took the same calls.
The answer, for now, is no. Not one AI sounded as human as the person.
Gemini came closest. But put Gemini and the human agent side by side, and listeners would still pick the human about 4 times out of 5.
OpenAI came third. Grok and Inworld are tied for fourth, too close to separate.
Based on 1,543 blind votes from Indian listeners, on calls in Indian English.
Ever wondered why there's no Mean Opinion Score (MOS) on the Voice of India leaderboards?
Because scores are subjective. One person's 5 is another person's 7. Ask thousands of people to rate models out of 10 and you end up averaging their different scales, and that average can rank models the wrong way.
Blind pairwise evaluation takes most of that subjectivity out. Raters see two outputs with no model names and answer one question: "Which one is better?" Not "What score would you give each one?" Picking between two options leaves far less room for personal scales than putting a number on each.
On our Text-to-Image board alone: 22,865 comparisons by native speakers across 11 languages, and every one of them asked the same question:
"Pick the better image based on the prompt provided."
Because scores are subjective. One person's 5 is another person's 7. Ask thousands of people to rate models out of 10 and you end up averaging their different scales, and that average can rank models the wrong way.
Blind pairwise evaluation takes most of that subjectivity out. Raters see two outputs with no model names and answer one question: "Which one is better?" Not "What score would you give each one?" Picking between two options leaves far less room for personal scales than putting a number on each.
On our Text-to-Image board alone: 22,865 comparisons by native speakers across 11 languages, and every one of them asked the same question:
"Pick the better image based on the prompt provided."
Gemini (Duplex) is the most human-sounding AI voice agent in #Indian English - 1,101 Elo across 1,543 blind human votes.
It still trails a real human by 223 Elo.
Elo is a rating system used to rank performance in head-to-head matchups. Here, it measures how often an AI voice is judged more human than another voice.
Voice of India's Speech-to-Speech Arena scores full-duplex voice agents on humanness: two agents hold a natural conversation, blind listeners pick which one feels human. One of the five entrants is an actual person: the Human Anchor.
Here’s where the board stands:
➤ Human Anchor: 1,324 Elo, wins 90% of its matchups. The nearest AI sits 223 Elo back.
➤ Gemini (Duplex)- @GoogleDeepMind: 1,101 Elo, 65% win rate. The top machine, +204 clear of the next.
➤ OpenAI (Duplex)- @OpenAI: 897 Elo, 36% win rate.
➤ Grok (Duplex)- @xAI: 844, and Inworld (Duplex)- @inworld: 833. Inside each other's error bars (±22–23); ranks 4 and 5 interchange.
The gap that matters here isn't between the models. It's the 223 Elo between the best AI and one human voice. That's a line no other voice arena can draw, because no other arena puts a human on the board.
Bradley-Terry Elo from 1,543 blind pairwise votes, 14% ties. Provisional, and moving as votes come in.
Full board, and cast your own vote → voiceofindia.ai
Congrats to @GoogleDeepMind on the top machine slot, and to @OpenAI, @xAI and @inworld for stepping into a human-anchored arena.
It still trails a real human by 223 Elo.
Elo is a rating system used to rank performance in head-to-head matchups. Here, it measures how often an AI voice is judged more human than another voice.
Voice of India's Speech-to-Speech Arena scores full-duplex voice agents on humanness: two agents hold a natural conversation, blind listeners pick which one feels human. One of the five entrants is an actual person: the Human Anchor.
Here’s where the board stands:
➤ Human Anchor: 1,324 Elo, wins 90% of its matchups. The nearest AI sits 223 Elo back.
➤ Gemini (Duplex)- @GoogleDeepMind: 1,101 Elo, 65% win rate. The top machine, +204 clear of the next.
➤ OpenAI (Duplex)- @OpenAI: 897 Elo, 36% win rate.
➤ Grok (Duplex)- @xAI: 844, and Inworld (Duplex)- @inworld: 833. Inside each other's error bars (±22–23); ranks 4 and 5 interchange.
The gap that matters here isn't between the models. It's the 223 Elo between the best AI and one human voice. That's a line no other voice arena can draw, because no other arena puts a human on the board.
Bradley-Terry Elo from 1,543 blind pairwise votes, 14% ties. Provisional, and moving as votes come in.
Full board, and cast your own vote → voiceofindia.ai
Congrats to @GoogleDeepMind on the top machine slot, and to @OpenAI, @xAI and @inworld for stepping into a human-anchored arena.
Nano Banana 2 is the most culturally accurate image model on Indian prompts.
1,112 Elo across 22,865 blind human votes.
Elo is a rating system used to rank performance in head-to-head matchups. Here, it measures which model produces the more culturally accurate image when people judge the outputs blind.
And even the leader loses roughly a third of its matchups to models ranked below it.
Voice of India's Text-to-Image Leaderboard scores cultural truth, not aesthetics: 1,000 prompts written by people living in each state about something specific to where they live, then voted blind by raters from that same state. The people who notice if the drape, the sweet, the script or the temple is wrong.
➤ Nano Banana 2- @GoogleDeepMind: 1,112 Elo, 67% win rate. The leader, +45 clear of the next.
➤ Grok Imagine- @xai: 1,067 Elo, 61% win rate.
➤ GPT Image 2 High- @OpenAI: 1,039 Elo, 56% win rate.
➤ FLUX.2 Max- @bfl_ml: 993, MAI Image 2.5- @MSFTAI: 985, and Krea 2 Medium- @krea_ai: 983. Inside each other's error bars (±6 to 7); ranks 4 to 6 interchange.
➤ Recraft v4.1 Utility- @recraftai: 914, 36% win rate, and Reve 1.0- @reve_image: 907, 35%.
The gap that matters here isn't between the models. It's between a beautiful image and a real one.
Every system on this board can render a gorgeous Indian street; 205 Elo separates them on whether it's the street the prompt actually asked for, or the default sepia Rajasthan lane a web corpus taught them to draw.
Bradley-Terry Elo from 22,865 blind pairwise votes, ties counted as half a win each, intervals from resampling the prompt set. Provisional, and moving as votes come in.
Full board, state-wise breakdowns, and cast your own vote → voiceofindia.ai
Congrats to @GoogleDeepMind on the top slot, and to @xai, @OpenAI, @bfl_ml, @MSFTAI, @krea_ai, @recraftai and @reve_image for putting their models on a board that scores accuracy over polish.
1,112 Elo across 22,865 blind human votes.
Elo is a rating system used to rank performance in head-to-head matchups. Here, it measures which model produces the more culturally accurate image when people judge the outputs blind.
And even the leader loses roughly a third of its matchups to models ranked below it.
Voice of India's Text-to-Image Leaderboard scores cultural truth, not aesthetics: 1,000 prompts written by people living in each state about something specific to where they live, then voted blind by raters from that same state. The people who notice if the drape, the sweet, the script or the temple is wrong.
➤ Nano Banana 2- @GoogleDeepMind: 1,112 Elo, 67% win rate. The leader, +45 clear of the next.
➤ Grok Imagine- @xai: 1,067 Elo, 61% win rate.
➤ GPT Image 2 High- @OpenAI: 1,039 Elo, 56% win rate.
➤ FLUX.2 Max- @bfl_ml: 993, MAI Image 2.5- @MSFTAI: 985, and Krea 2 Medium- @krea_ai: 983. Inside each other's error bars (±6 to 7); ranks 4 to 6 interchange.
➤ Recraft v4.1 Utility- @recraftai: 914, 36% win rate, and Reve 1.0- @reve_image: 907, 35%.
The gap that matters here isn't between the models. It's between a beautiful image and a real one.
Every system on this board can render a gorgeous Indian street; 205 Elo separates them on whether it's the street the prompt actually asked for, or the default sepia Rajasthan lane a web corpus taught them to draw.
Bradley-Terry Elo from 22,865 blind pairwise votes, ties counted as half a win each, intervals from resampling the prompt set. Provisional, and moving as votes come in.
Full board, state-wise breakdowns, and cast your own vote → voiceofindia.ai
Congrats to @GoogleDeepMind on the top slot, and to @xai, @OpenAI, @bfl_ml, @MSFTAI, @krea_ai, @recraftai and @reve_image for putting their models on a board that scores accuracy over polish.
