Everyday Research Methods

Replication: "Don't ditch the laptop just yet"

A mini meta-analysis found that the average effect size comparing laptop to handwritten notes was negligible. Credit: insta_photos/Shutterstock

One of the major empirical examples in Chapter 10 (in the 3rd and 4th editions of the textbook) is a study that tested whether laptop or handwritten notes are better for people's learning from a video lecture. As you may recall, the study (Mueller & Oppenheimer, 2014) found that people who took notes by hand generally remembered more--they scored better on conceptual questions (though not better on factual questions) about a TED talk they had watched. 

Last month, a replication study was published in the journal Psychological Science--the same journal that published the original Mueller and Oppenheimer study. The replication was lead by Dr. Heather Urry of Tufts University. The coauthors on the paper are her entire research methods class, who conducted the work as a class project (How cool is that? Shout out to Dr. Urry's students!). 

The work is an excellent example of a replication study. It was covered in a few media outlets . Here is one summary, as reported by the APS website:

But although previous work suggested that students who write notes by hand may retain information better than those who type them out on a laptop, attempts to replicate this research indicate that taking notes longhand may offer no measurable benefit. 

“We found only small, statistically nonsignificant differences in quiz performance as a function of note-taking medium,” whether students used a laptop or pen, wrote APS Fellow Heather L. Urry (Tufts University) and colleagues in Psychological Science. 

Here's some coverage of the same study, in a blog post by Dr. Regan Gurung, on Psychology Today:

There was no difference in how well the students performed on the factual recall questions as a function of how they took notes. This was similar to the 2014 study. The difference, and failure to replicate, came when Urry et al. analyzed performance on the applied questions. Again, there was no difference in scores based on note-taking style.

In the APS summary, it is reported:

Urry and colleagues reached this conclusion through a direct replication of a 2014 study by Pam A. Mueller (Princeton University) and APS Fellow Daniel M. Oppenheimer (University of California, Los Angeles), in addition to a mini meta-analysis of eight related studies. 

a) As a review, what were the IV (and its levels) and DVs in the original study? (hint: There are two DVs)

b) What does the term "direct replication" mean in the previous quote?

c) How does direct replication compare to a conceptual replication? (You'll find these terms in Chapter 14)

 

Now let's talk about the term "mini meta-analysis" in the quote, "...in addition to a mini meta-analysis of eight related studies."   Here's a summary from the Psychology Today blog:

Urry et al. (2021) also found 8 similar studies, that also aimed to replicate the 2014 results.  The studies compared also used similar methodologies with slight variations. The effect of taking notes longhand was not statistically significant.

d) Do you think the eight studies  in the mini meta-analysis were direct replications or conceptual replications? How can you tell? 

e) Now explain, in your own words, how they might have done the mini meta-analysis? What would the authors have done, once they found the eight studies? (Why do you think it was called "mini"?)

f) Dr. Urry's team preregistered their replication study--what does preregistration mean in this context? 

g) While the original Mueller and Oppenheimer study was covered by multiple popular press outlets (such as here, here, and here), the replication study has, so far, not really broken through to the popular press. Why do you think that is? 

You should check out forest plots of the mini meta-analysis!  Here's the link to the journal article. If you're locked out, you can access a preprint of the replication study here. 

People believe claims of sexual harassment more when women are attractive

The coded stereotypicality of the drawing was a dependent variable in one of the studies. Photo: Courtesy of Dr. Jin Goh. Citation: Goh, J. X., Bandt-Law, B., Cheek, N. N., Sinclair, S., & Kaiser, C. R. (2021). Narrow prototypes and neglected victims: Understanding perceptions of sexual harassment. Journal of Personality and Social Psychology. Advance online publication. https://doi.org/10.1037/pspi0000260

Sexual harassment (unwelcome and inappropriate sexual remarks or physical contact) harms people in schools and workplaces. One potential problem with sexual harassment is whether people who report it are taken seriously. Unfortunately, when people report sexual harassment, their claims are not always believed, which means perpetrators can go uninvestigated and victims' harassment might continue. 

The press recently covered an important series of studies that investigated the role that physical appearance plays in the reporting (and believing) process. One of the summaries appeared in the Seattle Times; another  appeared on the APA website.  The Seattle Times journalist summed up the studies like this: 

The study found people are more apt to believe sexual harassment claims by women who are young, “conventionally attractive,” and appear and act feminine. Women who don’t fit that prototype not only are less likely to be believed, but also are presumed to be unharmed by harassing behavior, the study said. 

The APA website does a better job detailing the methods:

The researchers conducted a series of 11 multimethod experiments, involving more than 4,000 total participants, designed to investigate the effect a victim’s fit to the concept of a typical woman had on participants’ view of sexual harassment and the consequences of that mental association.

In five of the experiments, participants read scenarios in which women either did or did not experience sexual harassment. Participants then assessed the extent to which these women fit with the idealized image of women, either by drawing what they thought the woman might look like or selecting from a series of photos. Across all the experiments, participants perceived the targets of sexual harassment as more stereotypical than those who did not experience harassment.

The details above describe a set of experiments (do you know why they are experiments?)

a) What was the main independent variable (IV) in the five studies above? What were its levels? 

b) What was the main dependent variable (DV) in the five studies? What do you think its levels were? 

c) The image (with the two drawings, above) that goes with this post came from the first study (Study 1), in which participants read a scenario in which " women either did or did not experience sexual harassment" and made a "drawing what they thought the woman might look like". Which drawing do you think came from each condition? 

d) The drawings from Study 1 had to be coded for stereotypical femininity. Since this variable required coding, inter-rater reliability would have been important. How do you think the researchers established inter-rater reliability for their coders? (Hint: you can find relevant information in Chapter 5.)

e) Sketch a small bar graph depicting the results of these studies, based on the journalist's description. 

Now here's some more description from the APA website:

In the next four experiments, participants [read] ambiguous sexual harassment scenarios, such as a boss inquiring about a woman’s dating life. These scenarios were paired with descriptions or photos of women who were either stereotypical or not. The participants then rated the likelihood that the incident constituted sexual harassment.

f) What was the main independent variable (IV) in these four studies? What were its levels? 

g) What was the main dependent variable (DV) in the four studies? What do its levels seem to have been? 

h) Sketch a small bar graph depicting the results based on the journalist's description. 

i) Assess the plausibility of this claim: "Being more stereotypically feminine causes people to think a woman's sexual harassment claim is true"  Apply the three causal criteria (covariance, temporal precedence, and internal validity). 

Instructors: Students are likely to enjoy reading the studies in the original journal article, which appears on my list of journal articles that are student-friendly.