An interesting profile “The Dismal Science’s Crusading Voice: Economist Bruce McCullough Continues a Largely Solitary Push to Require Replication in Research” just published in the Wall Street Journal highlights the importance of replication in economics and how elusive that goal has been in practice at economic journals.  To understand fully the problem one must go back further than the past decade covered in the article.

Nearly three decades ago the American Economic Review (AER, 1986, Vol. 76, No. 4) published a rather devastating article by Bill Dewald, Jerry Thursby and Richard Anerson called “Replication in Empirical Economics.” As the authors stated in the abstract, the

paper examines the role of replication in empirical economic research. It presents the findings of a two-year study that collected programs and data from authors and attempted to replicate their published results. Our research provides new and important information about the extent and causes of failures to replicate published results in economics. Our findings suggest that inadvertent errors in published empirical articles are a commonplace rather than a rare occurrence.

Because of that study the editors of the AER at that time (Orley Ashenfelter, Bob Haveman, John Riley, and I) published an Editorial Statement in the same issue of the AER putting forth a new replication policy:

It is the policy of the American Economic Review to publish papers only where the data used in the analysis are clearly and precisely documented, are readily available to any researcher for purposes of replication, and where details of the computations sufficient to permit replication are provided. The Managing Editor should be notified at the  time of submission if the data used in a paper are proprietary, or if, for some other reason, the above requirements cannot be met.

What is worrisome is that years later in 2003 McCullough found that this replication policy was not being enforced. He reported that the authors of half the AER papers he examined declined to provide the data and computer code need to replicate, forcing Ben Bernanke the editor at that time to step up the enforcement.  

Recently there are some good signs of more interest in replication in the economics profession, with Volker Wieland’s macro model data base in Frankfurt being a prime example. But let’s hope that today’s journal editors are taking their policies seriously.  The major difficulties are that there is little incentive to replicate other peoples work in economics and it is hard to make replication easy. There’s a lot of economic policy that is being based on this research.