Hallucinations are very damaging to a model’s utility. But doesn’t the Omniscience Index focus on knowledge-based queries? To me, using LLMs for their memorized knowledge is very 2023 and suboptimal.

IMO, what really makes a model useful is its ability to process information within its context reliably and faithfully. I don’t care if it hallucinates George Washington’s favorite color, but I do care about it hallucinating the results of tool calls.

I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.

> A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses.

It's not useful because the benchmarks often measure the wrong thing. They're here yapping about AGI and yet the benchmarks treat it like a trained dog. Fetch this. 100 points.

Each "problem" in these benchmarks likely has more than 1 solution that can be considered correct and even should be graded in many ways. Yet we see in many benchmarks higher effort (or thinking levels) don't help because the benchmark penalizes for doing "more" than what the answers asks for. So what did you ask for?

In human school you often get marks on the process and not just the end result. Thinking tokens have been cut. All we group on is things like cost, turns and time but not the what else.

This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.

is any of this 'scientific'? does AA allow peer review of its processes? are these published in journals of at least medium impact? what are the sample sizes? how grounded is their mechanistic reasoning?

like they have words that are dressed in scientific language on their site like

"We estimate a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1% - based on experiments with >10 repeats on certain models for all evaluation datasets included in Artificial Analysis Intelligence Index v4.2."

but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for?

there's a major difference between scientific sounding and being truly empirically rigorous. the 'research' in AI intelligence feels somehow even less trustworthy than supplement-funded studies because those are at least subjected to scrutiny by peers without profit motives

Given how many private benchmarks they're using now, its likely they just tested different combos until they got the result they wanted.

Completely discredits the index if it just gets modified to match social media vibes.

In real life, there is always this feedback edge from the results to the methodology.

Theoretically it's unscientific to do tweaks like this but in reality this is what actual science is because you need to see the results understand the problems in your experimental designs.

Now the interesting thing is that you could repair the bias problem. The main issue is that you will do these tweaks after a bad result not after they look fine.

Maybe we could commit ahead of time that the experiment will be reanalyzed after results regardless of what they are. Instead of only when the results disprove the hypothesis.

So like artificial analysis committing to a fixed cadence of index updates instead of when the results start looking jank.

> Astra is way better than Sol

This needs to be evaluated per task, jagged frontier yadda yadda. I would be not at all surprised if sol was better at some things than Astra just like people still use opus 4.6 and for good reasons.

What other benchmarks do you recommend that are more accurate?

Because it's the best option currently available.

It's really easy to shit on AI benchmarks, but that noise is useless unless you're offering a solution or a better benchmark.

As the saying goes, Artificial Analysis is the worst benchmarking company, except for all the others.

That would be the fastest way for them to completely torch their company.

The only thing they are selling and why people look at them is trust that they do honest evaluations.

The apparent advantage is exaggerated by them running Astra at six different effort levels, and almost everything else at just the maximum available effort.

I don't really understand why they keep doing this. Either run and report everyone at multiple effort levels, or run everyone at only one.

But alsi, token efficiency seems pretty artificial? For example tokenizers are different from model to model. The cost/perf Pareto frontier seems a lot more meaningful (and Astra does very well at that too, just to be clear. It seems to be a great model.)

GPT-6 has a looped transformer / recurrent depth architecture, so it gets some internal CoT reasoning "for free" with no output tokens.

This also makes it much harder to monitor its reasoning.

Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of course it's going to dominate the perf at given output token level. The truly sensible comparison is perf -vs- thinking flops (because each model might be a different unknown size, but labs are very secretive about what they're actually running under the hood) or perhaps cost (which can be misleading because of subsidies, but is at least practically relevant in the moment).

The timing is related to the fact that their benchmark was saying it was the same as Sol, and below Opus 5, when anecdotal reports and other benchmarks strongly disagree. It looked bad for them for their benchmark to disagree with people's lived experience so hard.