Sol Hsiang writes:

We have a comment coming out in Nature next week that is going to cause the retraction of a high-profile paper by Kotz et al. from last year (the second most cited climate paper in the news in 2024).

Basically, Kotz et al claimed that climate change was already costing the world economy a huge amount and would cost 300% of what prior estimates claimed (which was already large). This result got enormous attention in Europe, in particular.

We couldn’t reproduce their findings and realized that it was all driven by weird data from Uzbekistan. If you remove Uzbekistan from their data set, the result falls apart. The costs are still large, but not the extreme numbers that made headlines around the world.

One reason we think this is important is because these data were previously being used by central banks around the world to run stress tests for the effects of climate change.

Here’s the retraction note, in full:

The authors have retracted this paper for the following reasons: post-publication, the results were found to be sensitive to the removal of one country, Uzbekistan, where inaccuracies were noted in the underlying economic data for the period 1995–1999. Furthermore, spatial auto-correlation was argued to be relevant for the uncertainty ranges. The authors corrected the data from Uzbekistan for 1995–1999 and controlled for data source transitions and higher-order trends as present in the Uzbekistan data. They also accounted for spatial auto-correlation. These changes led to discrepancies in the estimates for climate damages by mid-century, with an increased uncertainty range (from 11–29% to 6–31%) and a lower probability of damages diverging across emission scenarios by 2050 (from 99% to 90%).

The authors acknowledge that these changes are too substantial for a correction, leading to the retraction of the paper. An updated version of the paper with these changes, which has yet to undergo peer review, is publicly available with continued open access to its data and methodology (https://doi.org/10.5281/zenodo.15984134). The authors intend to submit a revised version of the paper for peer review. If and when published, this retraction note will be updated to include a link to the new publication. The authors appreciate the corrective role of the global scientific community and thank Thomas Bearpark, Dylan Hogan, Solomon Hsiang and Christof Schötz for bringing these issues to their attention. All authors agree to this retraction.

Good for them. And here’s the story in Retraction Watch.

How science advances when data and methods are open

Jonathan Falk independently pointed me to this story and wrote:

Imagine how uphill it would have been without access to the original data/methods.

Good point!

He also pointed to this news article which summarized the story:

If Uzbekistan were excluded . . . the damages would look similar to earlier research. Instead of a 62 percent decline in economic output by 2100 in a world where carbon emissions continue unabated, global output would be reduced by 23 percent. . . .

Wait—the estimate declines by almost a factor of 3 after removing just one data point? Uzbekistan’s not a tiny country but it’s not huge either (population 40 million); it doesn’t seem like its data should have so much influence as all that.

I went back to the original paper and it has some scatterplots, but (a) it’s hard to see that any one point would be so influential, and (b) the countries aren’t labeled so I don’t see which one is Uzbekistan.

A question of influence

What happened with the data? Is there some sort of scatterplot that would’ve indicated a concern?

To put it another way, if the data from a single medium-sized country could have that much of an impact on the findings, that would’ve been worth reporting from the get-go in the original paper. Even had there not been any data problems, we’d want to know that the results were so sensitive to one data point.

So the meta-question is: What data analysis should’ve been done originally, either to flag the problem with Uzbekistan’s data, or at least to reveal the extreme sensitivity of the headline results to that one data point?

I posed this question to Hsiang, who responded:

We noticed this issue because we were looking at several papers and running some basic diagnostics on all of them. One thing we were doing was just dropping one country at a time and rerunning the models to make sure things weren’t being driving by a single country. We were surprised that this turned up. There are many issues with this paper conceptually, but it’s not even really possible to discuss any of them until you deal with the UZB issue. We had a lot of dialogue with the authors, and it turned out that they really hadn’t run much quality control on the more granular data. When we traced back this issue, it seemed like their research assistants had faithfully converted some numbers from a PDF document, but those numbers were just implausible.

There is a scatterplot in their data paper that is supposed to provide technical validation of their data set. We wanted to see why Uzbekistan didn’t jump out, so we reproduced it in our comment (Extended Data Fig 1). It turns out that Uzbekistan wasn’t even the biggest outlier, but that the version they had published had the axes cropped so you couldn’t see the outliers (see red boxes in our version). This seemed indicative of a different issue, which is why we documented it in the comment.

I guess those graphs should be on the log scale?

It still seems crazy that the data from a single mid-sized country could have such a big effect of a global estimate. That’s something that the original researchers should’ve been aware of, and what it suggests to me is that there is a larger methodological problem that this didn’t get looked at automatically during the research process.