Massive collaboration testing reproducibility of psychology studies publishes findings

Dr. Russ Clay and 269 co-authors investigating the reproducibility of psychological science published their findings today in Science Magazine.

Launched nearly four years ago, the Reproducibility Project: Psychology has produced the most comprehensive investigation ever done about the rate and predictors of reproducibility in a field of science.

The project conducted replications of 100 published findings of three prominent psychology journals.  They found that regardless of the analytic method or criteria used, fewer than half of their replications produced the same findings as the original study.  Mallory Kidwell, one of the project coordinators from the Center for Open Science, concluded, “The results provide suggestive evidence toward the challenges of reproducing research findings, including identifying predictors of reproducibility and practices to improve it.”

The extent to which the findings promote a change in research practices across the many sub-disciplines of psychology will be seen in the months and years to come.

Dr. Clay, with the Department of Psychology at The City University of New York’s College of Staten Island, commented that, “Being a part of this project has been a tremendous learning experience.  My hope is that the field of psychology views these findings as an opportunity to improve the way that we conduct research.  Doing so will inspire confidence in the research findings that we produce, not only within our field, but from the public at large as well.”

Read real-time news coverage updates from the international media>

Science is unique from other ways of gaining knowledge by relying on reproducibility to gain confidence in ideas and evidence.  Reproducibility means that the results recur when the same data are analyzed again, or when new data are collected using the same methods.  As noted by Angela Attwood, team member from University of Bristol, “Scientific evidence does not rely on trusting the authority of the person that made the discovery.  Rather, credibility accumulates through independent replication and elaboration of the ideas and evidence.”

The team emphasized that a failure to reproduce does not necessarily mean the original report was incorrect.  Elizabeth Gilbert, team member from the University of Virginia, noted, “A replication team must have a complete understanding of the methodology used for the original research, and shifts in the context or conditions of the research could be unrecognized but important for observing the result.”  Belen Fernandez-Castilla, team member from Universidad Complutense de Madrid, added, “Scientists investigate things that are not yet understood, and initial observations may not be robust.”

Yet a problem for psychology and other fields is that incentives for scientists are not consistently aligned with reproducibility.  “What is good for science and what is good for scientists are not always the same thing.  In the present culture, scientists’ key incentive is earning publications of their research, particularly in prestigious outlets,” said Ljiljana Lazarević, team member from the University of Belgrade.  Research with new, surprising findings is more likely to be published than research examining when, why, or how existing findings can be reproduced.  As a consequence, it is in many scientists’ career interests to pursue innovative research, even at the cost of reproducibility of the findings.

Fewer than half of the original findings were successfully replicated.  This held true across multiple different criteria of success.  The team noted three basic reasons this might occur: 1) Even though most replication teams worked with the original authors to use the same materials and methods, small differences in when, where, or how the replication was carried out might have influenced the results.  2) The replication might have failed to detect the original result by chance. 3) The original result might have been a false positive.  Johanna Cohoon, another Project Coordinator from the Center for Open Science, concluded that “the findings demonstrate that reproducing original results may be more difficult than is presently assumed, and interventions may be needed to improve reproducibility.”  In keeping with the goals of openness and reproducibility, every replication project posted its methods on a public website, and later added their raw data and computer code for reproducing their analyses.

Many organizations, funders, journals, and publishers are already working on improving reproducibility.  For example, in 2014, the journal Psychological Science – one of the journals included in this study – implemented practices such as badges to acknowledge open sharing of materials and data to improve reproducibility.  “Efforts include increasing transparency of original research materials, code, and data so that other teams can more accurately assess, replicate, and extend the original research, and pre-registration of research designs to increase the robustness of the inferences drawn from the statistical analyses applied to research results,” said Denny Borsboom, a team member from the University of Amsterdam who was also involved in creation of the Transparency and Openness Promotion (TOP) Guidelines published in Science in June.

Since the Reproducibility Project began in 2011, similar projects have emerged in other fields such as the Reproducibility Project: Cancer Biology.  And, a discipline of metascience is emerging – scientific research about scientific research.  These and the widespread efforts to improve research transparency and reproducibility are indications that, as suggested by team member Susann Fiedler from the Max Planck Institute for Research on Collective Goods, “science is actively self-examining and self-correcting to maximize the quality and efficiency of the research process in the service of building knowledge for the public good.”