Guest Post by Willis Eschenbach

There’s a recent study in AGU Atmospheres entitled “Proxy evidence for China’s monsoon precipitation response to volcanic aerosols over the past seven centuries”, by Zhou et al, paywalled here. The study was highlighted by Anthony here. It makes the claim that volcanic eruptions cause droughts in China. Is this possible? Sure. But have they made their case? … well, that is far from sure.

As is far too common, the authors have not archived either the data as used or the code as used. So, again as usual, I’ve gone to take a look at the data. Their main dataset is a reconstruction of the Palmer Drought Severity Index (PDSI) for China since the year 1300 … and how do they know what the PDSI was in China in the year say 1492?  

… Well … they are using a PDSI reconstruction called the Monsoon Asian Drought Atlas (MADA), by Cook et al. The MADA reconstruction and the description are here. Cook et al. say we can reconstruct the PDSI using tree rings. Now, this is a step above using tree rings for temperature, but still … they’re trying to reconstruct the PDSI on a 2.5° x 2.5° grid basis. Here’s the problem:

Figure 1. Cook et al. Figure 1B. ORIGINAL CAPTION: More detailed view of the MADA domain; the 534 grid points of instrumental PDSIs (red crosses) were reconstructed by the 327-series tree-ring chronology network (green dots).

As you can see, there are huge areas of both Asia in general and China in particular that do not have any trees within a long, long ways of the grid points. Cook discusses the various methods that were used to do the reconstruction, and you can read about them here, but I can’t claim that I was impressed. No matter how clever you are, you have more grid points with no data than grid points with data … sketchy.

In any case, I went to look at the MADA data. The study itself doesn’t show a graphic of the data … and given what the data looks like, I can’t say I’m surprised that they didn’t show it:

Figure 2. Area-weighted average of the MADA reconstruction of the Palmer Drought Severity Index (PDSI). Negative numbers indicate drought. According to Zhou et al, “positive MADA values stand for wet conditions while negative values represent dry conditions; droughts develop while MADA values fall below 0.5.”

I’m sure that you can see the problem. According to the tree ring proxies, Asia is in the midst of the worst drought in seven hundred years … curious how that hasn’t made the headline news. How do Zhou et al. deal with that? Well … they don’t comment on it at all.

That alone would disqualify or at least raise serious questions about the validity of their study (and the MADA reconstruction) for me, but it gets worse. In the abstract, Zhou et al. say:

Results show a statistically significant (at 90% confidence level) drying trend over mainland China from year 1 to year 4 after the eruptions.

A 90% confidence level? Ninety percent?

Ninety percent confidence gives me no confidence at all. But it gets worse. Here are their results:

Figure 3. Zhou et al. results of the stacked “Superposed Epoch Analysis” (SEA) of the MADA drought data, along with the Chinese Historic Index of droughts (CHI, panel b). INH2p shows northern hemisphere or global volcanoes twice the strength of Pinatubo, INH1p shows volcanoes the strength of Pinatubo or greater, INH1/2p shows volcanoes half the strength of Pinatubo or more, ISH shows volcanoes only affecting the southern hemisphere. Note that the sense of the MADA and CHI indexes are reversed, with positive CHI values indicating more drought and positive MADA values indicating wetter conditions.

They are using “stacked” data, what they call a “Superposed Epoch Analysis”. In this type of analysis, you align the data from the years surrounding each eruption at the dates of the eruptions, and then average them from five years before the eruptions to five years after the eruptions.

So what’s not to like?

Well, the first thing that I don’t like is that there are no error bars on the results. Bad scientists, no cookies. Considering that there are only six INH2p eruptions (Injections of sulfates in the Northern Hemisphere twice Pinatubo strength) in the IVI2 eruption dataset that they used, this is a huge omission.

Next thing I don’t like is that they’ve got data from 1300 to 2005, but they are only using data from 1400 to 1900 … not sure why they did that, but it does make me suspectful.

Next thing I don’t like is that the CHI index shows no effect at all the first year after the eruption, with the maximum effect of the NH volcanoes in the second year only, and things returning to pre-eruption values in the third year. But the MADA data shows the effects starting the year after the volcano, peaking during the third year, and not returning to normal until after the fifth year … there is no discussion of this difference.

Final thing I don’t like? I can’t replicate their results. Here’s what I get for the MADA data (I haven’t bothered with the CHI data).

Figure 4. My results for the Superimposed Epoch Analysis of the MADA drought data. Error bars show the 95% confidence interval for each point. Points are offset slightly to show the individual error bars.

This shows the importance of error bars. Out of the 44 points in the four analyses, there are only two which are significantly different from zero. With forty-four data points, random chance alone should give us up to 5 data points which are “significant” at a 95% confidence level … so they’re not doing better than random.

One final issue. They say:

Results obtained from the above methods are based on the assumption that there is no temporal correlation of the precipitation proxies. CHI is obtained from historical documents and thus is independent in time and space. MADA is reconstructed from the tree ring data which should be determined largely by the meteorological conditions of the individual growing season so the data should have no correlation in time.

Autocorrelation is very important in assessing the statistical significance of a result. As a consequence, it seems to me that these assumptions about the autocorrelation of the reconstructions need to be tested and verified rather than assumed and asserted … especially given that the PDSI for the US 1895-2009 has a lag-1 autocorrelation of about 0.5.

CONCLUSIONS:

• The MADA reconstruction of the Palmer Drought Severity Index shows a very strange dropoff at the end (see Figure 2), which indicates that there is something seriously wrong with or omitted from either the proxy data, with the reconstruction method, or with the underlying assumption of linearity. Curiously, it appears to be a version of the “divergence problem” seen in other tree-ring reconstructions, and which no one has ever satisfactorily explained. In temperature reconstructions the narrow rings are said to represent cool temperatures in recent years, despite the acknowledged slight warming. If the same narrow rings appear in a drought reconstruction, on the other hand, it would be interpreted as dry conditions.

• The lack of error bars renders their results meaningless.

• The differences between the MADA and the CHI results indicates further problems with their datasets. Strangely, they show no direct comparison of the MADA and CHI data.

• In my analysis including error bars, only two of the forty-four data points resulting from the SEA analysis are significantly different from zero at the 95% level.

• They use a very low 90% significance level for their overall results.

• They have assumed a lack of autocorrelation of the MADA and CHI data, but apparently have made no effort to actually calculate the autocorrelation.

• I am unable to replicate their results.

• Since they have not posted either their data as used or their code as used, I am unable to determine the reason for the difference between their results and mine. It may be my error entirely, perhaps my foolish error, but without their data and code I can’t tell. However, even if I could replicate their results, the other errors invalidate the study on their own.

There are other problems with the study, but I downloaded the data last night in Salem, Oregon, and I’ve done this entire analysis on a train rolling south to the middle of California. Right now it’s 11:45PM, and I can’t be bothered to mess with this nonsense any more. I gotta say, these kinds of pseudo-scientific studies are getting old … in any case, at present their results are useless.

Best regards to all, and don’t believe everything that you read in the “scientific” journals,

w.

THE USUAL: If you disagree with someone, please QUOTE THEIR EXACT WORDS THAT YOU DISAGREE WITH. Don’t just reference their entire comment, quote only and exactly where you think they went off the rails. This avoids all manner of problems and misunderstandings.

DATA AND CODE: The source for the MADA data is given above. That data, plus the IVI2 volcano data used in the study and the R code, is in a folder called “MADA folder.zip“.