Over the last few years we have learned a lot about the reliability of scientific evidence in a range of fields through large scale ‘meta-research’ projects. Such projects take a scientific approach to studying science itself, and can help shed light on whether science is progressing in the cumulative fashion we might hope for.
One well known meta-research example is The Reproducibility Project in Psychology. A group of 270 psychological scientists embarked on a worldwide collaboration to undertake a full direct replication of 100 published studies, in order to test the average reliability of findings. Results showed over half of those 100 replications failed to produce the same results as the original. Similar studies have been conducted in other fields too—biomedicine, economics —with equally disappointing results.
It’s tempting to think that this kind of replication happens all the time. But it doesn’t. Studies of other disciplines tell us that only 1 in every 1,000 papers published is a true direct replication of previous research. The vast majority of published findings never face the challenge of replication.
As yet, there have not been any meta-research projects in ecology and evolution, so we don’t know whether the same low reproducibility rates plague our own discipline. In fact, it’s not just that the meta-research hasn’t been done yet, it is quite unlikely to ever happen, at least in the form of direct replication discussed above. This is because the spatial and temporal dependencies of ecological processes, the long time frames and other intrinsic features make direct replication attempts difficult at best, and often impossible.
But there are real reasons to be concerned about what that meta-research would show, if it was possible. The aspects of the scientific culture and practice that have been identified as direct causes of the reproducibility crisis in other disciplines exist in ecology and evolution too. For example, there’s a strong bias towards only publishing novel, original research which automatically pushes replication studies out of the publication cycle. The pragmatic difficulties of experimental and field research mean that the statistical power of those studies is often low, and yet there are a disproportionate number of ‘positive’ or ‘significant’ studies in the literature—another kind of publication bias towards ‘significant’ results. The rate of full data and material sharing in many journals is still low, despite this being one of the easiest and most obvious solutions to reproducibility problems.
In our paper, we argue that the pragmatic difficulties with direct replication projects shouldn’t scare ecologists and evolutionary biologists off the idea of meta-research projects altogether. We discuss other approaches that could be used for replicating ecological research. We also propose several specific projects that could serve as ‘proxies’ or indicator measures of the likely reproducibility of the ecological evidence base. Finally, we argue that it’s particularly important for the discipline to take measure to safe guard against the known causes of reproducibility problems, in order to maintain public confidence in the discipline, and the important evidence base it provides for important environmental and conservation decisions.
Paper citation:
Fidler F., Chee Y.E., Wintle B.C., Burgman M.A., McCarthy M.A., Gordon A. (2017) Meta-research for Evaluating Reproducibility in Ecology and Evolution. Bioscience. doi: 10.1093/biosci/biw159l available at https://doi.org/10.1093/biosci/biw159
