I was a little suspicious last time, but even moreso this time. The slant their scores take, and the release timing is very suspect. On the other hand, new models drop on a near daily basis, so maybe there's no way to appear entirely fair.
An issue I think all benchmarks suffer from is regarding precision. What does a one point difference mean? How many points of difference is needed to translate into a meaningful distinction between models?
Among fellow humans, we often find ourselves picking what we consider to be most intelligent and then designing the tests around that. I've noticed a similar thing in AI, where whatever OAI/Anthropic decide to focus on improving becomes the new measure of intelligence. With their latest models, this seems to be 3d modeling/game making capabilities. Unsurprisingly, because that isn't what the Chinese companies have been focusing on, they appear behind on "frontier" reasoning tests. On top of that, older tests end up being discarded for being saturated. Makes me think that maybe unintentionally, the methodology used for these scores might be biased towards Western companies.
This would also fit with the widely discussed decline in writing quality from Opus and the worse readability of code produced by one of OAI's recent models, despite benchmarks continuing to show progress on all fronts.
Plus, if they're going to update the benchmark so frequently to discard saturated components, is there even a research-related point to keeping it private?
The writeup fails to disclose that they quietly redacted several open-weight models from the new benchmark. Popular ones. They're not on the charts; they're not available to manually add to the charts; and their individual model pages now display scores of "N/A".
This might be the first time I’ve favourited a post without upvoting it.
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
The natural language choice in multilingual index seems pretty strange and biased to me, they tried to encompass a broad swath of Asian languages, but then there's like four European romance languages. They didn't include any Slavic one i.e Russian, or Nordic one like Swedish, or Finno-Ugric. And these romance ones are probably overrepresented in the training data anyways
It is embarrassing how much they suck up to american labs while those models only perform the first few days after release. With open weights you can be sure the quality stays the same unlike the closed source crap. How about a retest after some time?
I think it's pretty evident that Artificial Analysis loves to change the benchmark to place US models on the top, it's no the first time either.
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
I'm still using DS-flash - it beats frontier model on speed and cost keeping good enough code quality.
It depends on the harness and constraints you give to the model but for most of the tasks its a great fit, I have not switched from DS since couple of weeks already (previously I would switch to Opus for some heavier work but I stopped recently).
My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.
Glancing over the methodology[1] made me curious about three things:
1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?
2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?
3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a senior developer?
I find it fascinating that some of the main areas people are interested in are using benchmarks with quite few questions: ~200 for coding and ~150 for science.
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
An issue I think all benchmarks suffer from is regarding precision. What does a one point difference mean? How many points of difference is needed to translate into a meaningful distinction between models?
Among fellow humans, we often find ourselves picking what we consider to be most intelligent and then designing the tests around that. I've noticed a similar thing in AI, where whatever OAI/Anthropic decide to focus on improving becomes the new measure of intelligence. With their latest models, this seems to be 3d modeling/game making capabilities. Unsurprisingly, because that isn't what the Chinese companies have been focusing on, they appear behind on "frontier" reasoning tests. On top of that, older tests end up being discarded for being saturated. Makes me think that maybe unintentionally, the methodology used for these scores might be biased towards Western companies.
This would also fit with the widely discussed decline in writing quality from Opus and the worse readability of code produced by one of OAI's recent models, despite benchmarks continuing to show progress on all fronts.
Plus, if they're going to update the benchmark so frequently to discard saturated components, is there even a research-related point to keeping it private?
GLM 5.3-flash: N/A
https://artificialanalysis.ai/models/glm-5-3-flash/
Qwen 3.8-flash-next: N/A
https://artificialanalysis.ai/models/qwen3-8-flash-next
This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.
Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.
That seems improbable
The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.
This is what happens when the benchmark team is funded by VC capital.
My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.
1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?
2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?
3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a senior developer?
[1] https://artificialanalysis.ai/methodology/intelligence-bench...
As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.
I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.
Kimi K3.1 Pro: −9
DeepSeek V4.1 Flash 1017: −13
GLM-5.3: −13
MiMo-V2.6-Pro: −16
Proprietary models
GPT-6 Astra: +4
GPT-6.1 Sol: +3
GPT-6.1 Terra: +3
Claude Opus 5.5: 0
Claude Sonnet 5.5: −2
Claude Fable 5.1: −3
Gemini 4 Argon (high): −3
Muse Wonder Full: −6
Claude Haiku 5.5: −7
Muse Wonder: −9
Gemini 3.8 Flash (high): −10
Grok 4.7 (high): −11
Is there affiliation with OpenAI or coincidence?
Will an LLM topping this bench be the end of humanity? The doomer rhetoric has no place in what should be an objective measure.