Everyday Research Methods

"Want to raise kind, generous kids? Take them to an art museum"

Did the study manipulate exposure to art museums?
Photo: AChubykin/Deposit Photos

On my first read of this headline, I was skeptical. "Probably a correlational study behind this one," I thought. As it turns out we are actually looking at an experiment. However, there's another problem with the journalist's coverage of this study. Read on.

The journalist writes for Inc.com, which  summarizes research for the business world. The headline was "Want to raise kind, generous kids? Take them to an art museum" The original empirical article appeared in Psychological Science. Let's work through it. The introduction starts out:

Everyone wants to raise successful kids, but a big part of being successful is being a kind and generous person. So how can parents give their kids the best while also doing their best to ensure they raise decent, thoughtful little people? 

They introduce the idea that art might make kids more generous: 

You may be wondering what possible connection a day viewing Van Gogh or Picasso could have to raising an empathetic kid. Certainly, museums make children more worldly and cultured, but how do they make them kinder? The link, according to the new study, recently published in Psychological Science, is awe. 

a) Before you read more, make a prediction. Given this introduction, what independent variable do you think the researchers probably manipulated? What variable do you think was measured?

Now let's read the details. There were two studies. In the first one, 

..the research team asked 159 volunteers aged 8 to 13 to watch short movie clips. Some of these clips were neutral, others cheerful, and others awe-inspiring. . 

In the second study, about 350 kids participated. The researchers used one of the same three clips for the awe variable (neutral, joy, and awe), but in this case, the generosity variable was operationalized differently:

The researchers then asked the kids [in Study 1] to determine how many items in a list of foods should be donated to a food drive for needy families. Alternately [in Study 2], the kids were given the option of donating their reward for participating in the study--a ticket to a local art museum--to a refugee family. 

(By the way, the original article reports that "The awe-eliciting clip was from the film Song of the Sea, in which the main character turns into a seal and transforms the city while flying above it.")

Back to the Inc.com story for the results: 

"Children who watched the awe-inspiring video chose to count 50 percent more items for the food drive than children who watched the joy-inspiring clip and more than twice as many items as children who watched the neutral clip. [in Study 2], children in the awe-inspiring condition were also 2 to 3 times more likely to donate their museum tickets than children in the joyful or neutral conditions," reports the Association for Psychological Science blog. 

b) Take a moment to classify the two variables in Study 1:

Variable name	What are the variable's possible levels?	Is this manipulated or measured?	Is this an IV or a DV?	For IVs: is it within groups or independent groups?

c) Which kind of experiment is Study 1: Posttest only? Pretest/posttest? Concurrent measures? Repeated measures? 

d) And now for Study 2:

Variable name	What are the variable's possible levels?	Is this manipulated or measured?	Is this an IV or a DV?	For IVs: is it within groups or independent groups?

e) Which kind of experiment is Study 2: Posttest only? Pretest/posttest? Concurrent measures? Repeated measures? 

Look back at the description of the study. Did either Study 1 or Study 2 manipulate "taking kids to an art museum"? Nope. In fact, Study 1 took place online (kids watched video clips on their own computers). And Study 2 took place in a science museum. Neither of the studies manipulated art museum exposure. Art museum tickets were used in Study 2, but only as a way of operationalizing generosity.

That factual error didn't stop the journalist from asserting:

Parents ... can always go to their local art museum to give their kids a guaranteed dose of awe. 

Guaranteed dose of awe? That conclusion does not really follow from this study!  So here are a couple more questions. 

f) What could be an alternative headline for the Inc.com story--one that is more accurate to what the study actually manipulated?

g) The journalist appears to have generalized from one setting (watching video clips) to another (going to an art museum). Which of the four big validities is the journalist treading on here? 

h) Here's a construct validity question: The two studies operationalized the dependent variable, "generosity" in two different ways. Were each of them a good way to measure this variable? Why or why not? What are the strengths of operationalizing the variable more than one way?

i) Now for a question related to internal validity: The main independent variable was the content of the film clip the children saw. To ensure internal validity, what should the researchers hold constant about the film clips used in the three conditions? 

 

 

 

"Fetuses smile for carrots but grimace over kale"

This fetus's facial expression would be judged as more of a cry or grimace. Photo courtesy of FETAP (Fetal Taste Preferences) Study, Fetal and Neonatal Research Lab, Durham University

This fetus's facial expression would be judged as smiling (or lip-pulling). Photo courtesy of FETAP (Fetal Taste Preferences) Study, Fetal and Neonatal Research Lab, Durham University

Even though kale is having a moment as a vegetable (and kale salad is certainly one of my favorites), most folks would agree that carrots taste better than kale.  A new study, covered by multiple news outlets, has demonstrated that even in utero, fetuses agree. They seem to be able to tell the difference between carrots and kale, and they like carrots better!

As you can imagine, this study was a media darling. Here are some excerpts of a report on this study as covered by journalists at CNN. 

The scientists used in-utero facial expressions to infer how babies felt about what their parent had eaten: 

Fetuses create more of a “laughter-face” in the womb when exposed to the flavor of carrots consumed by their mother and create more of a “cry-face” response when exposed to kale [...]

The study looked at the healthy fetuses of 100 women between the ages of 18 and 40 years who were between 32 and 36 weeks pregnant in northeast England.

From this, 35 women were [assigned at random] into an experimental group that consumed an organic kale capsule, 35 were put into a group that took a carrot capsule, and 30 were put into a control group that was not exposed to either flavor.

After a waiting period of 20 minutes post-consumption, the women underwent 4D ultrasound scans, which were compared to 2D images of the fetuses.

Lip corner pulling, suggestive of a smile or laughter, was significantly higher in the carrot group compared to the kale and control group. Whereas ... casting down of the lower lip...– suggestive of a crying face – [was] much more common in the kale group than the other groups.

Questions

1. Using the table below, classify the three main variables mentioned in this study (hint: One variable was manipulated and two were measured)

Variable name (conceptual variable)	How was this variable operationalized?	What are this variable's levels?	Was this a manipulated or measured variable?	Was this an IV or a DV?

2. Was the independent variable manipulated as independent groups or within-groups? 

3. Which of the four basic experimental designs was used: posttest only? pretest-posttest? repeated measures? concurrent measures?  Explain your answer. 

4. Sketch a little bar graph of the results of the "lip pulling" result.

5. Can this study support the causal claim that "eating a carrot pill causes fetuses to make a smiling face"? Apply the three causal criteria. 

6.  The CNN study mentioned the following in their reporting. Which of the four big validities is this addressing? 

Participants were asked not to consume any food or flavored drinks one hour before their scans. The mothers also did not eat or drink anything containing carrot or kale on the day of their scans to make sure it would not influence the results.

7. The 100 fetuses in this study were not a random sample of any population--they were a convenience sample. Why do you think these researchers did not prioritize external validity in this study? Do you think that's a major problem, or not? 

8. Use the Internet to find other media coverage of this study--note any similarities and differences in how the journalists discuss the findings. 

Here's a link to the original empirical article (paywalled). 

Time alone with your thoughts?

Credit: Ground Picture/Shutterstock

Imagine sitting alone with your thoughts for 20 minutes. You wouldn't be allowed to distract yourself with a book, phone, or even walking around. How much do you think you would you enjoy it?  

When this question was asked of participants in a series of studies, most reported that they'd not enjoy time alone very much.  But this study also tested how well their predictions matched their actual experience. The empirical research was published in the Journal of Experimental Psychology: General, and  summarized in a trade magazine for the American Psychological Association. According to the journalist, 

...study lead author Aya Hatano, PhD, of Kyoto University in Japan[, said] “Our research suggests that individuals have difficulty appreciating just how engaging thinking can be. That could explain why people prefer keeping themselves busy with devices and other distractions, rather than taking a moment for reflection and imagination in daily life.”

The research team conducted six studies. Here is an overview of some of them: 

In a series of six experiments with a total of 259 participants, the researchers compared people’s predictions of how much they would enjoy simply sitting and thinking with their actual experience of doing so. In the first experiment, they asked people to predict how much they would enjoy sitting alone with their thoughts for 20 minutes, without being allowed to do anything distracting such as reading, walking around or looking at a smartphone. Afterward, participants reported how much they had enjoyed it.

The researchers found that people enjoyed spending time with their thoughts significantly more than they had predicted. This held true across variations of the experiment in which participants sat in a bare conference room or in a small, dark tented area with no visual stimulation; variations in which the thinking period lasted for three minutes or for 20 minutes; and one variation in which the researchers asked people to report on their enjoyment midway through the task instead of after it was over. In every case, participants enjoyed thinking more than they had expected to.

a) The researchers conducted six different studies. Do these sound like direct replications? Replication-plus-extensions? or Conceptual replications? (you can skip this question if you haven't done Chapter 14 yet)

b) The journalist identifies these as "experiments," so let's assume that each study contains a manipulated variable. Take this study, as described by the journalist:

The researchers found that people enjoyed spending time with their thoughts significantly more than they had predicted [in a variation of] the experiment in which participants sat in a bare conference room or in a small, dark tented area with no visual stimulation. 

In this study, one of the independent variables was "predicted vs. actual".  What is the other main independent variable, or IV? What is the dependent variable, or DV? 

c) Do the same for this study:

variations in which the thinking period lasted for three minutes or for 20 minutes;

In this study, one of the independent variables was "predicted vs. actual". What is the other IV? What is the DV?

d) And for this one: 

In another experiment, the researchers compared one group of participants’ predictions of how much they would enjoy thinking with another group’s predictions of how much they would enjoy checking the news on the internet. Again, the researchers found that people underestimated their enjoyment of thinking. The thinking group expected to enjoy the task significantly less than the news-checking group, but afterward, the two groups reported similar enjoyment levels.

In this study, one of the independent variables was "predicted vs. actual". What is the other IV? What is the DV?

e) Challenge question: For the study above (comparing time alone to looking at news on the internet)  and graph the result. You can compare your graph to Figure 6 in the empirical article (available next to the APA journalist's story, here).

Here's a final caveat about the results. I appreciated this statement because it puts the statistical outcome in concrete and realistic terms: 

It is important to note that participants did not rate thinking as an extremely enjoyable task, but simply as more enjoyable than they thought it would be, according to Murayama. On average, participants’ enjoyment level was around 3 to 4 on a 7-point scale. Future research should delve into which types of thinking are most enjoyable and motivating, according to Murayama. “Not all thinking is intrinsically rewarding, and in fact some people are prone to vicious cycles of negative thinking,” he said.

 

 

Where do you get more creative ideas?

The research tested whether people came up with more, and more creative, ideas when working face-to-face compared to Zoom.
Photo credit: MBI/Alamy Stock Photo

Apparently, researchers have found that Zoom can stifle pairs' creativity. Here's how an empirical study on this topic was introduced by journalists writing for The Guardian:

While the benefits of Zoom and other videoconferencing tools made them indispensable in the pandemic, the research suggests that heavy reliance on the technology comes at a cost to creative thinking.

a) Before moving on, how might you design an experiment to test which type of meeting would be better for creativity? What would your study's independent variable (IV) be? What would the dependent variable (DV) be? 

Let's see how the researchers did it. According to the journalist,  

... the researchers recruited more than 600 volunteers who were paired up to tackle a creativity task either together in the same room, or virtually over Zoom.

The pairs had five minutes to come up with creative uses for a Frisbee or bubble wrap and a minute to select their best idea. Independent judges ruled that turning a Frisbee into a plate was less creative than using it to knock fruit from a tree, while using bubble wrap to send morse code messages was more creative than using it to protect a baby. Overall, those who worked over Zoom had 20% fewer ideas than those who met face to face.

b) In the study above, what was the IV and what were its levels?

c) Was the IV manipulated as between groups or within-groups?

d) What was the DV and what were its possible levels?

e) What simple experiment is this: Posttest-only? Pretest-posttest? Repeated measures? Concurrent measures? 

Now here's a second experiment: 

The same effect was apparent in the real world. In a field study, the researchers analysed ideas for new products generated by 1,490 engineers for a multinational company. The engineers, who were in Finland, Hungary, India, Israel and Portugal, were randomly paired up and given an hour or so to brainstorm products either in person or over Webex videoconferencing. They then selected their best idea.

...the researchers report that the engineers produced more ideas, and more innovative ideas, when working face to face. “They are not only generating a larger number of creative ideas, but their best idea is better,” 

f) This experiment was billed as a "field study." What does that mean? 

g) In the study above, what was the IV and what were its levels?

h) Was the IV manipulated as between groups or within-groups?

i) What were the two DVs in this study?

j) What simple experiment is this: Posttest-only? Pretest-posttest? Repeated measures? Concurrent measures? 

k) The second study is a replication of the first one. What role does replication play in the scientific process? If you've studied Chapter 14, you could identify whether this was a direct replication, a replication-plus-extension, or a conceptual replication.

Interestingly, the journalist notes, "it is unclear whether the impact on creativity holds for larger teams." This statement is a good example of external validity as it applies to situations (rather than the most common focus of external validity, which is on a study's generalizability to participants).

l) What other situations might we wonder about as we consider the external validity of this research? 

The original empirical study was published in Nature. The original study contains a number of interesting additional DVs, including potential mediators for the overall effect on creativity. 

How much weekly exercise can prevent depression?

The weekly recommendation for exercise is 150 minutes.
Photo credit: blasbike/Deposit Photos

A journalist wrote about a recent meta-analysis in JAMA Psychiatry which evaluated the connection between weekly time spent exercising and levels of depression. The headline of the Washington Post summary read, "Exercising even half the recommended amount can help prevent depression". Let's delve into this research, as covered by the journalist in their brief report. You can access the original empirical work here. 

Here's the opening message:

Already known to help ease depression, regular exercise may also help prevent it, with people who exercised just half the recommended weekly amount lowering their risk for depression by 18 percent.... However, those who were more active, meeting at least the minimum recommended physical activity level, reduced their risk for depression by 25 percent, compared with inactive people.

The empirical research was a meta analysis, and it seems that all of the studies involved longitudinal designs: 

The findings stem from the analysis of data from 15 studies, involving 191,130 adults who were tracked for at least three years.

a) What are the steps you'd typically follow if you were conducting a meta-analysis? 

According to the journalist, the recommended amount of weekly exercise is 150 minutes. Therefore, it seems that working out only 75 minute per week (by walking, for example) does predict lower rates of depression over time. 

Let's apply the four big validities to this study. 

b) Construct validity: We'd need to address the construct validity of each major variable--exercise and depression. It's likely you can measure exercise in multiple ways (self-report, observational, physiological). Can you tell from the journalist's report how the studies measured exercise? 

What about depression--can you tell from the report how the studies measured depression?

I would assume that across the 15 studies, there were a variety of ways that depression and exercise were measured. Is this variability of operationalizations a strength of the meta-analysis, or a weakness? 

c) Statistical validity. The journalist merely reports a single estimate:  "people who exercised just half the recommended weekly amount lowering their risk for depression by 18 percent." But the empirical article gives us a 95% CI for this estimate: [13%, 23%].  In addition the journalist writes, "meeting at least the minimum recommended physical activity level, reduced their risk for depression by 25 percent, compared with inactive people." Again the empirical article gives the 95% CI as [18%, 32%%]. 

In your own words, explain what these two 95% CIs mean.

d) External validity. As we consider external validity, you might be tempted to focus on the total number of participants in the meta-analysis (more than 191,000 over 15 studies). Why would it be incorrect to use sample size to make decisions about external validity? 

e) Internal validity. All 15 studies in this analysis were prospective, or longitudinal. What does that mean? Use the term "temporal precedence" in your response.

Now to focus on internal validity specifically: Can you think a possible third variable that might be associated with both exercise and depression over time? (It would be potentially important for the researchers to control for this variable if they hoped to support a causal claim.)

 

 

 

 

 

 

Swapping saliva

According to the study, even babies know that we don't share our food with just anybody... Photo: szefei/Shutterstock

When you see the photo here, you probably assume the kids are siblings, or at least very close friends. It's not the matching dresses--it's because they're taking turns tasting the same ice cream cone--an activity in which they'll undoubtedly exchange saliva. We humans tend to conclude that people willing to share spit with each other are probably more intimately related. 

Do babies realize the same thing? Apparently so! Here's an NPR story about a research study on whether babies also conclude that people who share saliva  also conclude that they are more closely related. They introduce the story like this: 

"From a really young age, without much experience at all with these things, infants are able to understand not only who is connected but how they are connected," says Ashley Thomas of MIT, who studies what babies and young children understand about the complexities of their social world. "They are able to distinguish between different kinds of cooperative relationships."

As is usually the case, researchers have to get creative to study what young kids think. The present set of studies tested both infants (aged 8-10 months) and young toddlers. They used a puppet show to experimentally manipulate different variables: 

One of their videos shows a woman rolling a ball back and forth with a blue fuzzy puppet. Then another woman shares an orange with that same puppet by putting a slice of orange in her mouth, then letting the puppet nibble on the slice, and then putting it back in her own mouth.

Then they used looking time to measure what infants were thinking:

...the video then shows the two women with the blue fuzzy puppet in between them. The puppet starts to cry and puts its head down, as if it is suddenly unhappy.

When the puppet cried, infants and toddlers looked first and looked longer at the woman who had shared bites of her orange.

"They're looking in that direction because they expect something to happen there," says Thomas. "They expect that woman to be the one to respond to the puppet's distress."

Questions:

a) What is the independent variable (IV) in this experiment? What were its levels? Is it manipulated as independent-groups or within groups? 

b) What is the dependent variable (DV)?

c) Which of the four simple experiments does this seem to be--pretest-posttest? posttest only? concurrent measures? or repeated measures?

d) Sketch a bar graph of the results, putting the IV on the x-axis and the DV on the y-axis. 

Here's another part of the study: 

When the two women were shown with a totally new puppet that started to cry, however, infants and young children looked at both women equally often. This suggested that they didn't see this particular food-sharing woman as especially helpful; instead, her relationship with the puppet was what really mattered.

e) Sketch a bar graph of these results, putting the IV on the x-axis and the DV on the y-axis. 

Like most modern research articles, this one includes multiple studies (three studies, in fact). Additional studies help provide replications of the observed effects, and can also help rule out alternative explanations.  Here is the journalist's description of one of the other studies in the empirical journal article:

To make sure it wasn't just sharing of food that seemed to make babies infer the existence of a close social connection, the researchers created another, similar video. This time, instead of sharing an orange slice, a woman simply put her finger in her own mouth and then put it in a purple puppet's mouth, before putting it back in her own mouth.

Then that same woman also interacted with a green puppet, touching its forehead and then touching her own forehead. After that, the video showed the woman seemingly in distress, with the purple and green puppets looking on.

Infants and toddlers gazed at the purple puppet that had the more intimate, finger-in-the-mouth interaction, as if expecting this puppet to be more affected by the woman's consternation, presumably because they seemed to have a closer relationship.

f) What are the IVs and DVs in this new study?

g) Sketch a bar graph of the results being described.

h) What type of replication is this: a direct replication, a conceptual replication, or a replication-plus-extension? 

i) One possible explanation for the infants' looking behavior in this study is that infants just prefer to look at purple puppets more than green puppets. How could the researchers rule out this alternative explanation? 

Want to read more? here's a link to the empirical journal article, which appeared in the journal Science.

Curbing hate speech with empathy

The study found that empathy-inducing responses like this were more likely to be followed by people deleting their original hateful posts. Photo courtesy of the author

Hate speech abounds on certain corners of social media. Some people post racist, sexist, anti-immigrant, or homophobic comments, and these contribute to a harmful and hateful discourse online. Some companies filter, censor, or censure accounts when they cross the line. Other companies have proved unwilling to ban such content, citing free speech concerns. Recently, some researchers asked if people might engage with such posters online in order to prevent them from posting hate speech. 

The website Optimist Daily ran a story about their work. Here's how they introduced it: 

Led by public policy professor Dominik Hangartner, the research team identified 1,350 English-speaking Twitter users who had posted racist or xenophobic content. The researchers then assigned these users randomly to a control group or one of three test groups. The test groups were then prescribed different counterspeech strategies: empathy-eliciting messages, humor, or warning of consequences.

For example, the empathy eliciting messages might have read "your post is very painful for Jewish people to read...", or "For African-Americans, it really hurts to see people use language like this". Another example can be seen in the comment about immigrants in the photo above, accompanying this post.  

The humorous messages included a meme of an animal with a quip such as "please sir, stop tweeting".

The warning-of-consequences messages read, for example, "Hey, remember that your friends and family can see this tweet too." 

What happened after one of these responses was posted? Here is how the journalist described the results: 

The researchers found that by far, the most effective strategy for getting these users to delete or amend hateful content was counterspeech messages that elicit empathy.

Questions

a) What words in the journalist's story convey that this was an experiment?  I would additionally classify it as a field experiment--can you see why?

b) What was the independent variable? What were its levels? Was the IV manipulated as independent-groups or within groups?

c) There are two dependent variables mentioned in the quotes above. What are they? If you were the researcher, what other DVs would you be interested in? (you can consult the original study to see some of the other DVs)

d) How were the participants in this study selected? Did they use a random sampling method? Or something else? 

e) This study was approved by the researchers' IRB, but the participants in this study apparently did not provide informed consent to participate in the research. Why do you think the IRB approved the study?

Here is a link to the original journal article in PNAS. You can use it to study the graphs of the study's outcome. 

Make a face

Should you smile, grimace, or neither when you get your COVID jab? Photo credit: Gregory Johnston/Shutterstock

Have you gotten a COVID vaccination yet? If not,  grab a chopstick and read this article before you go. Here's a link to a journalist's summary of a study on the effect of making a face (such as a smile or a grimace) on the pain of an injection. 

The background for the research is the "facial feedback hypothesis:" If you've read Chapter 14 already, you'll recognize this line of work. 

The idea that manipulating our facial expressions can affect our emotions has a long and storied history. There are many advocates of [this] hypothesis, and many critics, too. Indeed, one of the classic findings in the field — that people find cartoons funnier if they hold a pen between their teeth, inducing a smile — recently failed to replicate. This mixed research background was well known to Sarah D. Pressman and Amanda M. Acevedo at the University of California, Irvine, who led the new work, published in Emotion.

Here is some detail about the study: 

The team randomised 231 student participants to hold either a neutral expression, a regular smile (which involves the cheek muscles), a Duchenne smile (which also recruits muscles by the eyes, resulting in creasing around the eyes) or a grimace (which recruits both these sets of muscles, plus those which allow us to wrinkle our brows.

In the study they manipulated the facial expressions unobtrusively, by asking participants to hold a chopstick in different ways:

To induce these expressions, the participants (who thought they were taking part in an investigation of multi-tasking) were given chopsticks to hold in one of four positions between their teeth, and asked to imitate an example photo. Each participant then held their assigned facial position while they received an injection of saline solution into the upper arm, designed to mimic a flu jab. Their heart rates and skin conductance were measured throughout [before, during, and after the jab], as objective markers of their stress levels, and they also reported on levels of pain (anticipated and then actual) and other emotions, such as being jittery or relaxed.

This study was an experiment. Before we get to the results, let's identify some of its features.

a) What was the independent variable (IV) here? How many levels in this IV, and what were they? 

b) Was the IV manipulated as independent groups or within groups? 

c) The journalist writes, "The team randomised 231 student participants to hold either....". Is the word "randomized" here referring to random assignment or random sampling? 

d) There are several dependent variables (DVs) reported here. Name three of them. 

e) Which type of experiment was this? posttest only? pretest-posttest? repeated measures? or concurrent measures?

Now for the results. The researchers collected multiple DVs and the patterns were not the same across all DVs. I will summarize the results for only one DV, self-rated pain:

The Duchenne smiling and grimacing groups gave lower pain ratings than the neutral group while waiting for the injection, during the injection, and immediately after it. The regular smiling group fell somewhere in between.

f) Sketch a graph of this result (placing the DV on the y-axis, as usual).

g) Now a big challenge: Apply Chapter 11's twelve internal validity threats to this example. How does the study's design help rule these threats out? I've done a few of these for you. You'll find Table 11.1 helpful here!

Design confound
Selection effect
Order effect
Maturation	This was a pretest-posttest design that took place in a single session. The single session helps rule out maturation. We can also rule out maturation because all four groups had the same opportunity to "mature."
History
Regression to the mean
Attrition
Testing	All groups were tested multiple times, which means this potential threat was held constant in all groups, ruling it out as an internal validity threat.
Instrumentation
Observer bias
Demand characteristic	The independent-groups design and cover story about multitasking can probably help disguise the true nature of the experiment.
Placebo effect

 

Replication: "Don't ditch the laptop just yet"

A mini meta-analysis found that the average effect size comparing laptop to handwritten notes was negligible. Credit: insta_photos/Shutterstock

One of the major empirical examples in Chapter 10 (in the 3rd and 4th editions of the textbook) is a study that tested whether laptop or handwritten notes are better for people's learning from a video lecture. As you may recall, the study (Mueller & Oppenheimer, 2014) found that people who took notes by hand generally remembered more--they scored better on conceptual questions (though not better on factual questions) about a TED talk they had watched. 

Last month, a replication study was published in the journal Psychological Science--the same journal that published the original Mueller and Oppenheimer study. The replication was lead by Dr. Heather Urry of Tufts University. The coauthors on the paper are her entire research methods class, who conducted the work as a class project (How cool is that? Shout out to Dr. Urry's students!). 

The work is an excellent example of a replication study. It was covered in a few media outlets . Here is one summary, as reported by the APS website:

But although previous work suggested that students who write notes by hand may retain information better than those who type them out on a laptop, attempts to replicate this research indicate that taking notes longhand may offer no measurable benefit. 

“We found only small, statistically nonsignificant differences in quiz performance as a function of note-taking medium,” whether students used a laptop or pen, wrote APS Fellow Heather L. Urry (Tufts University) and colleagues in Psychological Science. 

Here's some coverage of the same study, in a blog post by Dr. Regan Gurung, on Psychology Today:

There was no difference in how well the students performed on the factual recall questions as a function of how they took notes. This was similar to the 2014 study. The difference, and failure to replicate, came when Urry et al. analyzed performance on the applied questions. Again, there was no difference in scores based on note-taking style.

In the APS summary, it is reported:

Urry and colleagues reached this conclusion through a direct replication of a 2014 study by Pam A. Mueller (Princeton University) and APS Fellow Daniel M. Oppenheimer (University of California, Los Angeles), in addition to a mini meta-analysis of eight related studies. 

a) As a review, what were the IV (and its levels) and DVs in the original study? (hint: There are two DVs)

b) What does the term "direct replication" mean in the previous quote?

c) How does direct replication compare to a conceptual replication? (You'll find these terms in Chapter 14)

 

Now let's talk about the term "mini meta-analysis" in the quote, "...in addition to a mini meta-analysis of eight related studies."   Here's a summary from the Psychology Today blog:

Urry et al. (2021) also found 8 similar studies, that also aimed to replicate the 2014 results.  The studies compared also used similar methodologies with slight variations. The effect of taking notes longhand was not statistically significant.

d) Do you think the eight studies  in the mini meta-analysis were direct replications or conceptual replications? How can you tell? 

e) Now explain, in your own words, how they might have done the mini meta-analysis? What would the authors have done, once they found the eight studies? (Why do you think it was called "mini"?)

f) Dr. Urry's team preregistered their replication study--what does preregistration mean in this context? 

g) While the original Mueller and Oppenheimer study was covered by multiple popular press outlets (such as here, here, and here), the replication study has, so far, not really broken through to the popular press. Why do you think that is? 

You should check out forest plots of the mini meta-analysis!  Here's the link to the journal article. If you're locked out, you can access a preprint of the replication study here. 

Meta-analyses: Strength training treats depression and anxiety

Strength training seems to improve both depression and anxiety, according to two meta-analyses. Photo credit: romanchazov27/Deposit Photos

Many of us start the new year resolved to improve our exercise habits. Exercise, of course, is great for our physical health, and we also know that aerobic exercise (i.e., "cardio") can improve mood. The benefits of aerobic exercise on depression and mood have been well-documented by several studies (here's an example of one meta-analysis). 

A recent article in the Washington Post describes the benefits of strength, or resistance, training for depression. The piece provides a good example of how a journalist covers meta-analysis for a popular audience.

The author himself suffers from depression, and starts his article by explaining his own workout strategy:

It’s strength training — the muscle-building, body-toning and sometimes grunt-inducing workout — that has helped transform my chronic mental illness into better mental health. In fact, once I began a regular regimen of resistance training, the positive shift in my mood, my confidence and my self-image was so pronounced that I switched careers.

Of course, while an anecdote about one person's experience can be compelling, it doesn't carry the same weight as a systematic observation. It's possible that the author was a biased observer of his own improvement. Therefore, we need experiments that randomly assign people to different exercise groups to test whether resistance training has a causal effect on depression and mood. 

It turns out that several researchers have, in fact, studied the effect of strength training, enough to collect and combine:  

[Dr. Jacob] Meyer was among the authors of a recent meta-analysis of the link between resistance training and depressive symptoms ... The study, published in the June 2018 issue of JAMA Psychiatry, analyzed 33 clinical trials for the effects of resistance exercise on depression. Results showed that resistance exercise “significantly reduced depressive symptoms” among research participants.

a) In your own words, what does a meta-analysis do?

b) In the meta-analysis described in the paragraph above, there were 33 studies, all of which were experiments. Given the context, what was the independent variable in each of these 33 studies? What was the dependent variable? 

c) Reread the description above, which states that strength training "significantly reduced depressive symptoms". This phrasing does not indicate the meta-analytic effect size (that is, the effect size they obtained by averaging the effect sizes of all 33 studies). To see that effect size, you'd have to turn to the original empirical article, here.  The article is open-access and the forest plot you will find there depicts  the mean, or overall, effect. What is the overall effect size? 

The meta-analysis above showed that strength training can help improve depression. What about a related mental health condition, anxiety? Turns out there's a separate meta-analysis on that:

Resistance exercise doesn’t affect depression alone; research shows that it can reduce symptoms of anxiety, too. For a 2017 study published in the journal Sports Medicine, Gordon and his colleagues conducted another meta-analysis of 16 studies involving 992 total participants; it concluded that resistance training “significantly improves anxiety symptoms among both healthy participants and participants with a physical or mental illness.”

d) How many studies were in this meta-analysis on strength training and anxiety? Estimate how many participants were in each study.

e) What would have been the independent variable in each of these studies? What would have been the dependent variable?

f) Once again, the journalist explains that resistance training "significantly improves" anxiety symptoms, but we don't hear about the effect size. The original empirical article  is behind a paywall. You might be able to access it from your own library. If you can't find it, I'll tell you that the mean effect size (d) of the studies was 0.31, 95% CI [0.17, 0.44]. What information do you need to evaluate this effect size? Is the effect strong or weak? 

As a bonus, the researchers are exploring why strength training might improve depression and anxiety. That is, they are exploring potential mediators of the relationship between strength training and mental health. Here is the first of two mediator explanations, as summarized by the journalist:

One notable finding was that participants showed an improvement in their mood regardless of whether they grew physically stronger from the exercises. ... “Perhaps it might be the sense of accomplishment and confidence that comes from exercising, rather than the achievement of actual strength gains, to explain why study subjects felt better,” Meyer says.

And the second:

Resistance training, like other exercise, induces the release of a protein called BDNF, or brain-derived neurotrophic factor, into the hippocampus region of the brain. Among other functions, the hippocampus is responsible for mood regulation, and in people who are depressed, it shrinks up to 25 percent of its normal volume. The release of BDNF triggers the growth of new brain cells in the hippocampus, restoring it to its full size and improving communication between cells.

“That’s the physiological theory behind why you feel so good after working out,” says Jason Sawyer, an assistant professor of wellness and exercise science at Rhode Island College in Providence.... 

g) Choose one of the mediator descriptions above and sketch a figure depicting the mediator argument. Use a box for each variable, and arrange the boxes in a triangle shape, using the figures at the end of Chapter 9 as a model. 

 

Selected answers

c) The overall (meta-analytic) effect size was d = 0.66 95% CI [0.48, 0.83]. Therefore, strength training groups scored about 2/3 of a standard deviation better than control groups, on average. To evaluate how strong this effect is, it might be helpful to compare the effect of strength training to some other treatments for depression, including cognitive-behavioral therapy or medication. (A 2013 meta-analysis found that CBT's effect on depression is at least d = 0.53, 95% CI [0.43, 0.62].)

d) There were 16 studies and a total of 992 participants, so that's about 165 participants per study. 

e) Each study would have used exercise condition as an IV and anxiety level as a DV. 

f) As you can see, the overall (meta-analytic) effect size of resistance training on anxiety was not as strong as the overall effect size of resistance training on depression (the two CIs do not overlap). This pattern suggests that strength training does more for depression than for anxiety. 

g) Here's how you might sketch one of these mediation proposals:

[strength training condition] -->  [sense of accomplishment] --> [improved mood]