Everyday Research Methods

Psychology's Reproducibility Project

Replication is the act of conducting a study again. If the study gets the same result twice (or more), we say that the study's findings are replicable. Chapter 14 in the book describes three types of replications: Direct replication, conceptual replication, and replication-plus-extension. This blog post is about direct replications.

In the last few years, psychologists have been focused a lot on the replicability of the field's major findings. The effort came to some fruition last month, when the results of a large-scale project, "The Reproducibility Project" were published in the journal Science. Here's how Smithsonian Magazine introduced the topic:

Academic journals and the press regularly serve up fresh helpings of fascinating psychological research findings. But how many of those experiments would produce the same results a second time around?

According to work presented today in Science, fewer than half of 100 studies published in 2008 in three top psychology journals could be replicated successfully. The international effort included 270 scientists who re-ran other people's studies as part of The Reproducibility Project: Psychology, led by Brian Nosek of the University of Virginia.

That's the first, majorpart of the story--the outcome that 60% of the 100 studies could not replicate when an independent group of researchers conducted direct replications of them.

The next part of the story is about what this outcome means. Stop for a moment and consider what it means if the majority of these 100 studies cannot be replicated. What explanations can you come up with?

One interpretation, given that some social scientists have been in the news for fabricating results (such as in this example or this example, is thatthe non-replicable findings were fabricated (did you assume this?).  However, most scientists believe that fabrication is rare, and they stress that if a finding was not replicated, that does not mean it was fraudulent.

Another interpretation of this outcome is that psychology is "broken"--that it is in crisis. One version of that argument says that people will not trust certain findings or subfields any more, and that psychologists who do the unreplicable studies will have trouble getting jobs. For one version of this argument, see this story.

Finally, though, most scientists who've reflected on the finding that 60% of the studies did not replicate, have reached the conclusion that it's a typical stage in the scientific process.  Here's how the Smithsonian story described it:

The eye-opening results don't necessarily mean that those original findings were incorrect or that the scientific process is flawed. When one study finds an effect that a second study can't replicate, there are several possible reasons, says co-author Cody Christopherson of Southern Oregon University. Study A's result may be false, or Study B's results may be false—or there may be some subtle differences in the way the two studies were conducted that impacted the results.

In fact, the idea that there are subtle differences between the studies was also elaborated on by Dr. Lisa Feldman Barrett in an editorial:

...the failure to replicate is not a cause for alarm; in fact, it is a normal part of how science works. [Sometimes] study A produces the predicted phenomenon, whereas Study B does not. We have a failure to replicate. Does this mean that the phenomenon in question is necessarily illusory? Absolutely not. If the studies were well designed and executed, it is more likely that the phenomenon from Study A is true only under certain conditions. The scientist’s job now is to figure out what those conditions are, in order to form new and better hypotheses to test.

Later in her editorial, Feldman-Barrett explains the importance of context in scientific findings:

Much of science still assumes that phenomena can be explained with universal laws and therefore context should not matter. But this is not how the world works. Even a simple statement like “the sky is blue” is true only at particular times of day, depending on the mix of molecules in the air as they reflect and scatter light, and on the viewer’s experience of color.

Why is the reproducibility project considered news at all, you might ask? Don't psychologists routinely make sure their results are replicable by conducting direct replications of every result?  Here's an answer to that question, by author Cody Christopherson:

 

Unfortunately there are disincentives to pursuing this kind of research, he says: “To get hired and promoted in academia, you must publish original research, so direct replications are rarer. I hope going forward that the universities and funding agencies responsible for incentivizing this research—and the media outlets covering them—will realize that they've been part of the problem, and that devaluing replication in this way has created a less stable literature than we'd like.”

a. What kinds of policies and incentives might work to make direct replications more frequent and common?

 

b. Where do moderators come in here? Re-read Dr. Feldman Barrett's argument above, and see if you can detect the concept of moderators in her reasoning.

Preschool in Tulsa: Success story....or not?

Why might it be difficult to study the long-term impact of quality preschool programs? Photo: Shutterstock

An NPR feature story recently covered Tulsa, Oklahoma's free preschool program, which began about 10 years ago. Like some other cities around the U.S., Tulsa introduced a free pre-K program, with the goal of helping children from all economic backgrounds get ready to succeed in school.  Developmental psychologist Deborah Phillips of Georgetown University has tracked children who participated in the Tulsa program, and is especially interested in how the kids are doing as they start high school. According to the story:

"These children did show huge gains in early math and early literacy skills," says Deborah Phillips, a developmental psychologist at Georgetown University who has been overseeing the study. "They were more likely to be engaged in school, less timid in the classroom and more attentive."

Phillips says preschool gave them a good, strong boost into elementary school, Today, as eighth-graders, says Phillips, most of these kids are still doing really well.

Phillips didn't just look at grades and test scores. Her team looked at student mobility, whether kids were in advanced or special education classes. They examined retention rates, absenteeism, and they even surveyed students' attitudes about school.

Researchers then compared these eighth-graders to a large sample of Tulsa eighth- and seventh-graders who did not attend preschool. They found that those students were not doing nearly as well.

a) What kind of study is this? What are its independent and dependent variables? This is one of those studies that has multiple dependent variables. Make sure you include all of them in your answer!

Although Dr. Phillips' conclusion is that Tulsa's preschool improved kids' abilities to do well in school, the NPR piece interviewed some folks who objected. Here's one example:

....Russ Whitehurst, senior fellow with the Center on Children and Families at the Brookings Institution, a Washington, D.C., think tank....says he's looked closely at the Tulsa study and takes issue with the way researchers compared kids who were in the program with those who were not.

"What Dr. Phillips and her colleagues have done is scrounge up a bunch of kids who for whatever reason — and they don't know that reason — did not attend pre-K at all," he says.

"We don't know if they were similar to the kids who went to pre-K," Whitehurst adds. "That's why the design [raises] question marks about the ability to conclude that pre-K had the affects attributed by Dr. Phillips."

b. What kind of criticism is Dr. Whitehurst raising about this study? (What terms from the text can you apply here?) What, if anything could have been done to prevent the problem?What are the practical and ethical limitations to such research?

Here's another example of a person who is skeptical of the study's results:

...[high school] principal Nanette Coleman ... says she doesn't know how many of her ninth-graders this year attended Tulsa's preschool program, and has not seen the research. But she has a hard time believing that preschool, no matter how good the program, is going to have an impact on a student 10 years later.

"They're going to struggle when they get to me because there are so many outliers that can have a student not be successful," she says. "Let me be clear," she adds, "I've never made a direct linkage between a pre-K program and their high school success."

c) On what source of information (from Chapter 2) is Principal Coleman basing her beliefs? What are some problems with that source of information?