It’s often easier to conduct a science study, assemble the data and review, and come to a conclusion, than it is to replicate the results. The replication process, if it is robust and able to stand by itself, should be conducted at an independent institution, involving a separate group of scientists. The best scientific studies are those that can be repeated and have similar results obtained. Failure to do so not only results in poor science, it can also, in the case of something like a medicine, lead to patient harm if the medicinal product has been approved and released.
When a science paper is looked at, it is important to ask:
Can the results of the study be replicated?
Has a different research group attempted this, and, if so, what did they find?
If a different conclusion was reached, this potentially undermines the original claims.
Some branches of science are more vulnerable to the process of replication than others, it seems. This dates from a time when Professor Brian Nosek (University of Virginia) ran the “Reproducibility Initiative” project. Here he attempted to re-run some 100 psychological experiments. He was only able to successfully replicate 40 percent of them.
Following this, looking at the specific area of loss of willpower, Professor Martin Hagger (Curtin University in Australia) co-ordinated with 24 different research laboratories in order to replicate a published study. The outcome was that none succeeded, thereby casting doubt on the original study.
The original study that Hagger challenged was published in 1990s. It involved taking some fresh-baked cookies and placing them on a plate. Next to the plate was a bowl of radishes. Students were then taken into the room containing these foods. The students were split into two group and left on their own. The students were told to eat either cookies or radishes.
After this, the students were given a puzzle and told to solve it. The puzzle was designed so that it was impossible to complete. It was found that the students who eat the cookies worked on the puzzle for 19 minutes, on average, before throwing in the towel; those who had eaten the radishes only lasted about 8 minutes before giving up.
The outcome was called “ego depletion.” The inference was we each have a limited amount of willpower and this decreases with overuse. So the students who could eat only radishes while having to look at the cookies, had used up more willpower and therefore were not able to continue with the study for along as the students who had enjoyed the cookies. Thus willpower is seen as akin to muscle, and something that can become exhausted.
Except that Professor Martin Hagger and his 24 independent labs couldn’t replicate this, in fact they found willpower to be zero time based. So, was the original paper flawed? It could be. The problem is this wasn’t some long-forgotten paper, it was a major one that had received over 3,000 citations (see: “Ego Depletion: Is the Active Self a Limited Resource?”, published in the Journal of Personality and Social Psychology, 1998.)
Why does psychology, as a social scientific discipline, seem to get things wrong most often? The possibilities have been examined by Olivia Goldhill, writing for Quartz. These are:
The pressure from the universities that employ academics to publish in journals;
The desire from journals to publish more exciting and sensationalist research. Here “novelty” trumps “robustness”;
Desire by journals only to publish science that shows a positive outcome rather than research that confirms something that has gone before or even disprove something;
Wild claims made from tiny sample sets.
The problem affecting psychology affects many other fields of science too. Digital Journal recently ran an extensive review of “bad science”, looking at why scientific findings are sometimes unfounded (to counterbalance this we also looked at examples of “good” science too.)
Much bad science, we found, was due to misinterpretation of results; seeing correlation as the same as causation; not having a sufficient number of subjects; not having control groups; being selective with the evidence and so on. We also, like the psychology issue, found there were problems with replication.
The case for good science involves having established control groups; statistically significant sample sizes; using representative test subjects; focusing on a research topic and not attempting to come up with a “universal theory of everything.” Above all, it should be reproducible.
