To put this in perspective: - ADH (2015) was written by Ivy League professors. - It was published in the prestigious American Journal of Political Science. - It has 1,504 citations on Web of Science and 3,415 on Google Scholar. https://onlinelibrary.wiley.co... 2/20

In other words, people have been reading and citing ADH (2015) for a decade without, it seems, anyone stopping to think “Oh wait, West Germany’s real GDP per capita probably didn’t grow by 8% per year from 1960 to 1990. That would be a miracle.” 3/20

Why did no one notice? I fear that it is because scholars are not encouraged to look for such mistakes. Peer review is supposed to be the arbiter of truth. If something is published in a top journal, it must be right. The dominant epistemology assumes that peer review works. 4/20

Yet peer review is broken. It only encourages conformity and reinforces hierarchy, as well as wasting everyone’s time. https://x.com/a_m_mastroianni/... 5/20

ADH (2015) shows how dysfunctional the current system is. As I detail in the first of two new papers, they made a simple data-handling error. Rather than use real GDP per capita, they accidentally used nominal GDP per capita: https://github.com/joefrancis5... 6/20

In the paper, I demonstrate this by comparing their data to the current OECD data, but really you can see it just by looking at their chart above. It’s obvious that West Germany’s real GDP per capita could not have grown at 8% per year from 1860 to 1990. 7/20

I have also replicated ADH (2015) using real GDP per capita data from the World Bank and the German national accounts. Otherwise, the data are the same as in the original. And the results aren’t great. 8/20

The p-value calculated from the permutation test goes up from 0.059 to 0.118. I do not believe ADH’s results would have been published if they had used the correct variable of interest. As such, their article probably ought to be retracted (cc. @RetractionWatch ). 9/20

Furthermore, I do not think the problems in ADH (2015) stop with data handling. This footnote is a major red flag, in my opinion. It shows that they may have cherry-picked their donor pool to suit their hypothesis. 10/20

The risk is that ADH have excluded countries that, like West Germany, did poorly in the 1990s, thereby ensuring that West Germany’s poor performance would seem significant compared to the comparators they had selected. 11/20

ADH then justify this choice by saying that it did not affect their identification strategy, but––worryingly––do not seem to be aware that it has implications for their causal inference strategy. 12/20

In a second new paper, I then use the case of a recent paper in the Journal of Political Economy to show how the method that ADH use has such researcher degrees of freedom that it is wide open to abuse. https://github.com/joefrancis5... 13/20

Indeed, I argue, the “Synthetic Control Method” is so flimsy that economists do not even need to be aware that they have p-hacked their research design. It can happen by default because everything is left to subjective choice. 14/20

And, finally, I explain how junk articles persist: the incentives that researchers face to use trendy new methods; the dysfunctional peer-review system that rubber stamps them as truth; the journals that are unwilling to retract articles even when they are shown to be junk. 15/20

In the case of ADH (2015), people have been reading and citing it for a decade, accepting it because it had been published in a top journal, etc. Nonetheless, I first looked at on Saturday afternoon, and it took me about 24 hours to see the problem. 16/20

Why so quick? Because I have a bad attitude: I’m critical, aggressive, don’t respect my superiors, etc. This meant that I realized early on that I was ill-suited to academia, where disobedience is not often rewarded. 17/20

For my sins, then, I am here on my hill in Wales, an “independent scholar.” The University of Birmingham has been kind enough to give me an institutional affiliation, but it does not provide any funding. 18/20

Hence, if you like what I do and can afford it, please make a donation at: https://paypal.com/donate/?bus... And if you want to pay me to investigate a study or methodology that you suspect is problematic, my DMs are open. Even if you can’t pay me, I would still be interested! 19/20

I believe that there is a lot of peer-reviewed junk floating around out there, some of it highly cited. But it is time to cast down the idols. Let’s go! 20/20

p.s. These are the direct download links of my two papers: https://raw.githubusercontent.... https://raw.githubusercontent.... And this is ADH’s replication data, in case you want to check for yourselves: https://dataverse.harvard.edu/...