In the research world what often counts more is not the number of research papers produced but the number of times a paper is cited. With citations, what matters is that suitable papers are cited.
A new review has found that papers published in the fields of psychology, economics and general science, where the results presented cannot be replicated, are more likely to be cited papers in academic research. This is the reverse of what is desirable.
The study comes from the University of California San Diego’s Rady School of Management and the researchers involved refer to this as the ongoing “replication crisis” with reference to the concerns that arise when independent researchers try to repeat experiments reported by others.
Across different disciplines, based on a review of a selection of published papers, in psychology, only 39 percent could be successfully replicate; in economics, 61 percent; and with science, 62 percent could be replicated.
The degree of variation with the likelihood of citations is that papers that can be replicated are cited 153 times less than those that failed.
It also stands that some of the papers that are not repeatable begin to build up higher citation scores over time, in that their influence grows. Typically papers that failed to replicate were cited 16 times more per year than papers that could be replicated.
This state of academic affairs begs the questions: ‘Why are these paper so frequently cited?’ and ‘Why are non-replicable papers accepted for publication in the first place?’
The authors of the study suggest this is because the studies that are not reproducible tend to be the most interesting, attracting more readers. A short of scientific ‘click bait’. The economic model for many science journals is that they are expected to make money, either by charging authors or charging for access.
Such papers with more headline grabbing topics also tend to be those more readily covered more by the general media as well as social media platforms like Twitter.
Data for the study was drawn from Google Scholar, where citation scores were gathered. The variable remained the topicality of the subject. The gender of the authors and location did not make any significant difference.
The effects of non-reproducible work can be very impactful. Take the study by Andrew Wakefield published in The Lancet in 1998 (redacted in 2000). This led to tens of thousands of parents around the world not allowing their children to receive the measles, mumps and rubella vaccine, based on an unreproducible claim connecting vaccinations and autism. Globally, rates of measles rose.
The research is titled “Nonreplicable publications are cited more than replicable ones”, published in the journal Science Advances.
