Replication is the act of conducting a study again. If the study gets the same result twice (or more), we say that the study's findings are replicable. Chapter 14 in the book describes three types of replications: Direct replication, conceptual replication, and replication-plus-extension. This blog post is about direct replications.
In the last few years, psychologists have been focused a lot on the replicability of the field's major findings. The effort came to some fruition last month, when the results of a large-scale project, "The Reproducibility Project" were published in the journal Science. Here's how Smithsonian Magazine introduced the topic:
Academic journals and the press regularly serve up fresh helpings of fascinating psychological research findings. But how many of those experiments would produce the same results a second time around?
According to work presented today in Science, fewer than half of 100 studies published in 2008 in three top psychology journals could be replicated successfully. The international effort included 270 scientists who re-ran other people's studies as part of The Reproducibility Project: Psychology, led by Brian Nosek of the University of Virginia.
That's the first, majorpart of the story--the outcome that 60% of the 100 studies could not replicate when an independent group of researchers conducted direct replications of them.
The next part of the story is about what this outcome means. Stop for a moment and consider what it means if the majority of these 100 studies cannot be replicated. What explanations can you come up with?
One interpretation, given that some social scientists have been in the news for fabricating results (such as in this example or this example, is thatthe non-replicable findings were fabricated (did you assume this?). However, most scientists believe that fabrication is rare, and they stress that if a finding was not replicated, that does not mean it was fraudulent.
Another interpretation of this outcome is that psychology is "broken"--that it is in crisis. One version of that argument says that people will not trust certain findings or subfields any more, and that psychologists who do the unreplicable studies will have trouble getting jobs. For one version of this argument, see this story.
Finally, though, most scientists who've reflected on the finding that 60% of the studies did not replicate, have reached the conclusion that it's a typical stage in the scientific process. Here's how the Smithsonian story described it:
The eye-opening results don't necessarily mean that those original findings were incorrect or that the scientific process is flawed. When one study finds an effect that a second study can't replicate, there are several possible reasons, says co-author Cody Christopherson of Southern Oregon University. Study A's result may be false, or Study B's results may be false—or there may be some subtle differences in the way the two studies were conducted that impacted the results.
In fact, the idea that there are subtle differences between the studies was also elaborated on by Dr. Lisa Feldman Barrett in an editorial:
...the failure to replicate is not a cause for alarm; in fact, it is a normal part of how science works. [Sometimes] study A produces the predicted phenomenon, whereas Study B does not. We have a failure to replicate. Does this mean that the phenomenon in question is necessarily illusory? Absolutely not. If the studies were well designed and executed, it is more likely that the phenomenon from Study A is true only under certain conditions. The scientist’s job now is to figure out what those conditions are, in order to form new and better hypotheses to test.
Later in her editorial, Feldman-Barrett explains the importance of context in scientific findings:
Much of science still assumes that phenomena can be explained with universal laws and therefore context should not matter. But this is not how the world works. Even a simple statement like “the sky is blue” is true only at particular times of day, depending on the mix of molecules in the air as they reflect and scatter light, and on the viewer’s experience of color.