Everyday Research Methods

Update: Portion size studies

See updates from October 2018 and August 2019 below. 

My textbook describes several studies from Dr. Brian Wansink's lab. They make excellent teaching examples because students are able to understand the theory and hypotheses almost immediately and therefore focus on the methodological details. For example, in Chapter 10 of the 2nd and 3rd editions, I feature studies in which pasta was served from either large or small serving bowls (van Kleef, Shimizu, & Wansink, 2012). And in the Supplemental Chapters on statistics, I feature a study involving stale and fresh popcorn serving sizes (Wansink & Kim, 2005).

Instructors and students should know that Wansink's lab has come under intense scrutiny over the past year. First, he was attacked for publishing a (seemingly innocent) blog post admitting to questionable research practices (including HARKing--see Chapter 14 of the 3rd edition). Second, some researchers have alleged impossible values in data tables which suggest some sloppy statistical reporting.  Third, he has admitted to using the same wording in more than one publication (sometimes called self-plagiarizing) and publishing some of his data in two places. According to reports, Wansink appears open to checking all past work and publishing corrections as needed. This story in The Chronicle summarizes the issues in a fairly balanced report (from March, 2017), and this story explains the results of an inquiry by Wansink's institution, Cornell University (from April, 2017). 

In two of my own classes, I conducted demonstration versions of portion size studies, and I have obtained the predicted pattern, with large effect sizes, both times. In my opinion, the portion size effect is real (August 2019 update--see below for a new study that suggests my belief may be wrong!) However, it's definitely worth telling students about the alleged problems with Wansink's work.  

So far, the study in Chapter 10 (van Kleef et al., 2012) has not been identified as problematic. However, the popcorn study (Wansink & Kim, 2005) was alleged to have reported impossible values on a key table; the problems were described as "relatively minor" (Source). I changed that table for Figure S1.7 of the 3rd Edition in order to delete the ANOVA values that were found to be problematic. The entire table and discussion will be omitted from the 4th edition. However, despite my changes to the table, I think instructors should use Figure S1.7 only as an example for how to read data tables, and not endorse it as a replicable scientific finding. 

Here's an example of a rather vehement blog-based attack on Wansink's work, with several links to related posts. 

Update: October 2018

Dr. Wansink has agreed to resign from Cornell University after six of his articles were retracted from the journal JAMA and after his university concluded that he had engaged in academic misconduct. Here's a CNN story on the situation. 

Update: August 2019

One lab conducted a large, preregistered, experimental test of the portion size effect (using large and small plates). The study found no effect of plate size on number of calories consumed, amount served, number of bites taken, or time spent eating. Here's the article. 

Viewing scenes in slow-motion changes how we think

Is defensive end Allen Bailey intentionally hurting running back Carlos Hyde? Your answer may depend on whether you watch the scene in slo mo. Photo Credit: Peter G. Aiken/Getty Images

When people view videos in slow-motion, they attribute more intent and purpose to the actors in the video compared to when they watch the same scene at real-time, according to the results of a series of experiments. Slo-mo video might be especially important in how juries might interpret crime scene evidence, and even suggest that watching NFL replays in slow motion could make tackles seem more intentionally forceful. 

As you read this paragraph from the Scientific American summary, see if you can identify at least two studies the authors conducted. 

Researchers designed experiments based on a place where slow-motion video comes up a lot: the courtroom. They asked subjects to imagine themselves as jurors and watch a video of a convenience store robbery and shooting, either in slow motion or in real time. Those who watched the slow-motion video reported thinking the robber had more time to act and was acting with greater intent. The effect persisted even when the researchers displayed a timer on the screen to emphasize exactly how much time was passing, and it was reduced yet still present when subjects watched a combination of real-time and slow-motion videos of the crime (as they might in an actual courtroom). Participants also ascribed greater intent to a football player ramming an opponent when they viewed the play in slow motion.

a) What are the independent and dependent variables in the first experiment? Do you think the IV was manipulated as between groups or within groups? Assuming that the IV had three or four levels, what would they probably be? 

b) What are the independent and dependent variables in the second (football) experiment? Do you think the IV was manipulated as between groups or within groups? 

c) By conducting one study using a crime-scene situation and another using a football game, the researchers were able to build which of the four big validities?

d) The football player study could be seen as a replication of the crime-scene study. Is this a direct replication, a conceptual replication, or a replication-plus-extension study? 

e) If you were an attorney or sportscaster, how might you use (or even exploit?) the results of this study? 

You can read the full text of the original scientific paper here! You'll see that there were actually four studies in the published paper. 

 

Online resource: Video about random sampling

Pew Research is my favorite polling resource, partly because they ask such interesting questions, and partly because they are so transparent about sharing their methodology. (For examples, see their Methods page or click on the full Report Materials for a study they did on gun ownership in America.) They make their sampling techniques and question wording easily available.

Now Pew has shared a video that explains how a sample of 1,000 can be used to draw inferences about the population. Instructors: Save this one for when you teach Chapter 7!

 

Vacations and job stress

To what extent does the evidence support a causal influence of vacations on happiness and stress? Photo: Syda Productions/Shutterstock

Here are some quasi-experimental and correlational studies on vacations, just in time for the end of summer. The APS website describes a few studies consistent with the argument that vacations can be good for your mental health. Here's one study by researchers Sabine Sonnentag and Jana Kühnel:

The researchers surveyed 131 teachers before and after a two week break from school.

First, they had the teachers complete a measure of exhaustion—how emotionally drained and burned out they felt the day before heading out for vacation. The teachers then completed weekly surveys on how engaged they were with their work, relaxed, and stressed they felt four weeks after returning from vacation.

As predicted, the results indicated that vacationing had a beneficial effect. Not only did the teachers report feeling less tired and emotionally burned out, they also reported feeling more engaged and positive about their work.

a) This is a quasi-experiment. What is the study's "independent" variable? What is/are its dependent variable(s)?

b) Would you call the design a non-equivalent groups posttest only? non-equivalent groups pretest/posttest? Interrupted time series? Or non-equivalent groups interrupted time series? 

c) Consider the 12 internal validity threats in Table 11.1. Which threats can this study rule out? Which threats might still apply?

d) Sketch a graph of the results of the study, incorporating this (more negative) message: 

But, these benefits were fairly short-lived, particularly for those teachers who came back to especially difficult students and heavy workloads. Within four weeks, the vacation’s positive benefits had faded and teachers were back to their initial levels of stress and emotional exhaustion.

The article also suggests that when it comes to spending money, money spent on vacations is associated with more happiness than money spent on material goods. 

...psychological scientists Amit Kumar and Thomas Gilovich of Cornell University and Matthew Killingsworth of University of California, San Francisco tracked moment-to-moment data from 2,266 adults as part of a large-scale experience-sampling project. Participants received notifications from the researchers on their iPhones at random times throughout the day.

Comparing data from individual participants across different times, Gilovich and colleagues found that people were happier at times when they were thinking about a future experiential purchase, like a ski trip, than they were at times when they weren’t thinking about a purchase at all. There was no relative increase or decrease in happiness when people were thinking about a future material purchase.

e) The above study is a correlational one, with a twist. The researchers computed a correlation for each individual person, using "experience" as the unit of analysis. Given the results described above, what might a bar graph depict for a typical person in this study? (what would be on each axis, and what would the results pattern depict?)

Update: Extraneous factors in judicial decisions (Danzinger et al.)

This post isn't a replication update per se. However, this blog post by Daniel Lakens (2017, July 3) challenges an interrupted time series design featured in Chapter 13 (Figure 13.2). The original study, by Danziger et al. (2011),  found that judges were more likely to grant parole at the beginning of the day or after a snack break. 

Lakens's reasoning is that the effect is so large (d = 2.0) as to be impossible.  Keep in mind that this is a blog post, and thus not formally peer-reviewed. However, the point is compelling. 

One peer reviewed critique of the original paper appeared, by Glockner (2016) in Judgment and Decision Making. Another critique is by Margel and Shapard (2011) in the form of a published letter to PNAS, the journal where the original article appeard. 

I wish to thank Dr. Stephen Lindsay for bringing this example to my attention!

New Section: Replication Updates

Science progresses one study at a time. As scientists conduct research and make the results public, we enable others to build upon, replicate, and critique our work, improving the field and building a body of knowledge. Even the studies in the textbook are not certain "truths," but rather steps on a scientific path, selected at one moment in time. 

As the third edition of the textbook explains (Chapter 14), psychologists are investing new energy in improving our field. More than ever, psychologists are conducting replication studies,  improving data analysis techniques, and making science open and transparent. 

In same spirit, we've added a new section of the blog called Replication Updates. When I learn that a study that is featured in the textbook has not been replicated or has had its conclusions questioned, I will devote a post to the issue so that instructors can stay informed. To read them, click the appropriate filter on the blog menu.

I hope that as teachers, we will find ways to include students in thinking about issues of replication, scientific openness, and progress. As we do, let's keep in mind that some (but not all) of the critiques appear in non-peer reviewed outlets, and discuss this aspect with them as well. 

Going forward, I hope readers will let me know when they hear updates on studies featured in the book!