The AI folks do not seem to understand intelligence

Utopian and apocalyptic predictions about AI appear to ignore potential bottlenecks in intelligence — the more you know, the harder it becomes to learn more

• By Gautam Mukunda / Bloomberg Opinion

• • Add TT as Preferred Source

  • 

  • 

  • 

Just three years after OpenAI said its models were good at grade-school math, the company announced they had solved the Navier-Stokes Millennium Prize problem in 88 hours, something that had stumped humanity’s greatest mathematical minds since the 1930s. Project that trend forward and you see why OpenAI chief executive officer Sam Altman thinks artificial intelligence (AI) will cure all diseases — and why people are afraid that a rogue AI might harm humanity.

Santa Fe Institute scientists Melanie Mitchell and David Krakauer described AI’s mantra as “scale is all you need.” What that means is the bigger the training sets and the more compute you have, the better AI becomes.

Altman’s vision of AI’s exponential growth curve, and the related hopes and fears, are similarly driven by an unspoken assumption that intelligence is all you need. If you are smart enough, you can cure cancer — or destroy the world — but that is not how power and intelligence work.

Illustration: Yusha

Anthropic PBC says Claude, its AI model, already writes more than 80 percent of the code its engineers merge. Push that to 100 percent and AI would be building its own successors. It could keep racing ahead until finally it understood everything and could control everything.

That is the fear and, for some, the hope. AI could create what Anthropic chief executive officer Dario Amodei calls “a country of geniuses in a datacenter.”

The US ran that experiment at Los Alamos. It put legendary physicists such as Niels Bohr, Richard Feynman and Robert Oppenheimer to work building the atomic bomb. However, Alex Wellerstein, a historian of science and nuclear weapons at the Stevens Institute of Technology, calculated that Los Alamos accounted for just 4 percent of the Manhattan Project’s almost US$2 billion cost. Eighty percent of the budget went to plants in Tennessee and Washington that produced uranium and plutonium.

In his 1945 Congressional testimony, Oppenheimer said that without scientists there would have been no bomb, but “if there had been only scientists, there also would be no atomic bomb.”

AI has changed coding, mathematics and weather forecasting. They all have a vast pool of available training data, rapid feedback, and clear right and wrong answers. These keep models moored to reality, not spiraling off to disaster as the result of compounding small errors.

Constant corrections are why we remember the Wright brothers. The best-funded attempt at flight in 1903 was made by Samuel Langley, secretary of the Smithsonian Institution, who had US$50,000 in government funds and an engine four times as powerful as the one in the hands of the Wrights. Langley designed a plane that was inherently stable, but his brilliant design plunged into the Potomac River because the models he built did not scale.

The Wright brothers were bicycle mechanics. They did not have his resources, but they knew that bikes were unstable machines kept upright by constant corrections from the rider. They used that insight, plus wind tunnel tests of as many as 200 wing shapes, to beat their better-funded rival.

Intelligence needs input. The faster and clearer the input, the more powerful intelligence becomes. Math, coding and weather forecasting? Check. Financial markets qualify, too. Hedge fund Two Sigma Investments boasts that its computing power would rank among the world’s top five supercomputer sites, and that is not an accident.

Writing fails the tests, hence the deluge of AI slop on LinkedIn. So does curing cancer, because it requires experiments on living systems that take years to respond. Intelligence might improve your hit rate, but it cannot get you around that limit.

Intelligence gets used first in the places with the fastest and clearest feedback. What is left are the problems where it is worth less. That is why its power is unlikely to keep climbing. The more you know, the harder it becomes to learn even more. We ran that experiment in theoretical physics.

In the early days after the quantum mechanics revolution, progress leapt forward. Over time, despite better instruments and tools, foundational breakthroughs got harder. In a 1975 lecture, Nobel Prize-winning physicist Paul Dirac said: “It was very easy in those days for any second-rate physicist to do first-rate work. There has not been such a glorious time since then. It is very difficult now for a first-rate physicist to do second-rate work.”

Research-heavy industries show the same pattern. In semiconductors, Stanford University economist Nicholas Bloom and three coauthors found that doubling chip density takes more than 18 times as many researchers as in the early 1970s. Even Tom Davidson and Tom Houlden of the research group Forethought, whose model gives about 60 percent odds that fully automated AI research would compress three years of progress into one, assume the returns fall as software approaches its limits.

So, when Amodei predicts that powerful AI could compress 50 to 100 years of biological progress into five to 10, and that reductions of 95 percent or more in cancer deaths and new cases seem possible, skepticism is appropriate. The key question, one AI leader wrote, is how much progress each additional increment of intelligence buys, and which other inputs become the constraint when intelligence is plentiful.

That leader was Amodei himself, in his 2024 essay “Machines of Loving Grace.” In the same essay, he wrote that today’s particle physicists have no shortage of theories, but lack the accelerator data to choose between them, and it is not clear superintelligence would change that.

Other Anthropic researchers said that Amdahl’s Law dictates that speeding up one part of a process tends to move bottlenecks, not eliminate them. They even forecast that this would apply to a world dominated by self-improving AI.

That is why we all need to take a breath before plunging into the utopian and apocalyptic forecasts about AI, and to watch the bottlenecks instead. An advantage in thinking is not an army, a power grid or a supply chain, and it cannot build them overnight. Doing that would take more time and investment than the roughly US$760 billion in capital spending Google parent Alphabet Inc, Amazon.com Inc, Meta Platforms Inc and Microsoft Corp alone expect this year.

Altman’s curve is real in mathematics. Maybe not far beyond it. Transforming coding is important. It is not the first sign of a cybernetic god.

Gautam Mukunda writes about corporate management and innovation. He teaches leadership at the Yale School of Management and is the author of Indispensable: When Leaders Really Matter. This column reflects the personal views of the author and does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.