Debates over AI benchmarks — and the way they’re reported by AI labs — are spilling out into public view.
This week, an OpenAI worker accused Elon Musk’s AI firm, xAI, of publishing deceptive benchmark outcomes for its newest AI mannequin, Grok 3. One of many co-founders of xAI, Igor Babushkin, insisted that the corporate was in the fitting.
The reality lies someplace in between.
In a post on xAI’s blog, the corporate printed a graph exhibiting Grok 3’s efficiency on AIME 2025, a group of difficult math questions from a current invitational arithmetic examination. Some consultants have questioned AIME’s validity as an AI benchmark. However, AIME 2025 and older variations of the check are generally used to probe a mannequin’s math capacity.
xAI’s graph confirmed two variants of Grok 3, Grok 3 Reasoning Beta and Grok 3 mini Reasoning, beating OpenAI’s best-performing accessible mannequin, o3-mini-high, on AIME 2025. However OpenAI workers on X have been fast to level out that xAI’s graph didn’t embrace o3-mini-high’s AIME 2025 rating at “cons@64.”
What’s cons@64, you would possibly ask? Properly, it’s quick for “consensus@64,” and it mainly provides a mannequin 64 tries to reply every downside in a benchmark and takes the solutions generated most incessantly as the ultimate solutions. As you may think about, cons@64 tends to spice up fashions’ benchmark scores fairly a bit, and omitting it from a graph would possibly make it seem as if one mannequin surpasses one other when in actuality, that’s isn’t the case.
Grok 3 Reasoning Beta and Grok 3 mini Reasoning’s scores for AIME 2025 at “@1” — which means the primary rating the fashions acquired on the benchmark — fall beneath o3-mini-high’s rating. Grok 3 Reasoning Beta additionally trails ever-so-slightly behind OpenAI’s o1 model set to “medium” computing. But xAI is advertising Grok 3 because the “world’s smartest AI.”
Babushkin argued on X that OpenAI has printed equally deceptive benchmark charts prior to now — albeit charts evaluating the efficiency of its personal fashions. A extra impartial celebration within the debate put collectively a extra “correct” graph exhibiting almost each mannequin’s efficiency at cons@64:
Hilarious how some individuals see my plot as assault on OpenAI and others as assault on Grok whereas in actuality it’s DeepSeek propaganda
(I truly consider Grok appears to be like good there, and openAI’s TTC chicanery behind o3-mini-*excessive*-pass@”””1″”” deserves extra scrutiny.) https://t.co/dJqlJpcJh8 pic.twitter.com/3WH8FOUfic— Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) (@teortaxesTex) February 20, 2025
However as AI researcher Nathan Lambert pointed out in a post, maybe a very powerful metric stays a thriller: the computational (and financial) value it took for every mannequin to realize its finest rating. That simply goes to point out how little most AI benchmarks talk about fashions’ limitations — and their strengths.
