The Daily Newsstand · Free, Always
Wednesday, November 18, 2026

To rank the world's biggest supercomputers, we're gonna need a bigger bench

Translate

AI has birthed some truly colossal supers, they just aren't on the Top500

For decades, the Top500 list has tirelessly ranked the world’s biggest, most powerful, and often most unique publicly-known supercomputers.

With the rise of generative AI, the list no longer lives up to its name. China’s Lineshine supercomputer at 2.2 exaFLOPS of double precision compute in the High-Performance Linpack (HPL) benchmark may be the top system on the bi-annual ranking, but it’s nowhere near the biggest compute cluster in the world.

Superclusters like SpaceXAI’s Colossus, Meta’s Promethius, and OpenAI’s Stargate pack hundreds of thousands of GPUs, dwarfing anything on the Top500 today.

REG AD

“These [companies] are not interested in submitting to the Top500,” Eth Zürich’s Torsten Hoefler said during his Supercomputing address this week.

REG AD

It’s a dilemma that the founders of the Top500 ranking are now grappling with.  And some believe it may be time for a rethink.

“What’s really needed for tracking that more appropriately is not just a different benchmark but also different methodology,” Top500 author Erich Strohmaier said.

For a long time, Strohmaier explains, compute capability was roughly aligned with capacity.

What happened in the last few years with AI is “capacity really pulled ahead of capability in terms of market size,” he said. “We were and we still are very focused on capability.”

By capability, Strohmaier is referring to the ability to run one code across the entire cluster.

“Our prime rule for benchmarking is, you have to benchmark your whole system with one code and you cannot just add two parts up to make up a number,” he explained.

This is problematic for the large clusters major hyperscalers and cloud providers are now building to fuel the AI boom. “All those systems are capacity systems,” Strohmaier said.

They’re not designed to run one piece of code but have instead been architected to run many, many smaller or medium-sized code instantiations, which may include training, fine-tuning, reinforcement learning, and of course inference.

REG AD

Because of this, Strohmaier says, it wouldn’t be reasonable to expect SpaceXAI to run something High-Performance Linpack (HPL) on a cluster like Colossus.

But these clusters are still worth tracking. They'll just need a new benchmark better suited to judging the capability of these arrays while at the same time preventing operators from gaming the system.

What shape or form such a benchmark might take remains to be seen. Developing a new benchmark or methodology is no small task, and changing the existing benchmarks isn’t ideal. ”If we do anything to the benchmark itself, we’re going to lose the historic trends, which is extremely valuable,” Hoefler said.

Allowing for lower precisions — HPL-MxP already does this to some extent — is one avenue for addressing the fact that these accelerators are increasingly trading FP64 performance for lower-accuracy floating-point data types like FP8 or FP4.

Another avenue is developing a separate AI benchmark. This would be immediately applicable to these clusters, but with the pace of change in model architecture, building such a benchmark is challenging.

“The AI workloads change every two weeks, so if you pick something today, it’s going to be irrelevant in at most a year, and these benchmarks need to live for multiple decades,” Hoefler said. ®

View the original on The Register →

KioskNews shows a cleaned-up reading view extracted from the publisher’s page — the original always lives on their site, not ours.