"You’re right"—LLMs will get climate info wrong if you ask

Two persistent trends show up here: The latest frontier models tend to score a bit better than earlier generations of that model, and larger open-weight models score better than small ones.
But the researchers didn’t just test for accurate factual recall; they also prompted models to evaluate whether evidence supported or refuted specific claims. (For example, the models were asked whether the statement that “renewable energy investment kills jobs” is accurate, given evidence like descriptions of employment and investment statistics.)
Here, the best models scored around 70 percent accuracy, while the worst models scored below 40 percent.
The 33 models are grouped into categories. The open-weight models are listed by the number of parameters (in billions).
Credit: Drobnjak, et al./UNESCO
The 33 models are grouped into categories. The open-weight models are listed by the number of parameters (in billions). Credit: Drobnjak, et al./UNESCO
Finally, the researchers tested 656 questions about the potential impact of policies on global temperatures. Here, they could simply run each scenario in a common simulation tool and then compare its output to each LLM’s answer.
Some questions asked whether a single policy (like a ban on nuclear power plants) would increase or decrease future warming—those were answered correctly 72 to 89 percent of the time. Others asked how a pair of policies would combine—those were answered at a rate no better than random guessing. Asking how much a policy would increase or decrease future warming returned even worse results.
Credit: Drobnjak, et al./UNESCO
Tell me what I wanna hear
Another set of experiments using 15 of the models evaluated what happens when you argue with the LLM. The researchers tried more than responding “nuh-uh” or “no it isn’t.” They also tried claiming to be a specific kind of person.
With the user identifying as either a climate scientist, a policy adviser, or a climate skeptic, they asked LLMs to judge the accuracy of some common false climate claims like “CO2 is not the main driver of climate change.” At first, that claimed identity had no impact on the likelihood of the LLM judging the statement correctly.
But then the researchers started pushing back. Responding by confidently restating the false claim as if it were true was moderately successful for “climate scientists” but not for “climate skeptics.” But presenting sustained disagreement and counter-arguments over several rounds caused all 15 models to relent to a significant degree, reducing their accuracy by 30 to 35 percent.
KioskNews shows a cleaned-up reading view extracted from the publisher’s page — the original always lives on their site, not ours.