Everyday Research Methods

Replication: "Don't ditch the laptop just yet"

A mini meta-analysis found that the average effect size comparing laptop to handwritten notes was negligible. Credit: insta_photos/Shutterstock

One of the major empirical examples in Chapter 10 (in the 3rd and 4th editions of the textbook) is a study that tested whether laptop or handwritten notes are better for people's learning from a video lecture. As you may recall, the study (Mueller & Oppenheimer, 2014) found that people who took notes by hand generally remembered more--they scored better on conceptual questions (though not better on factual questions) about a TED talk they had watched. 

Last month, a replication study was published in the journal Psychological Science--the same journal that published the original Mueller and Oppenheimer study. The replication was lead by Dr. Heather Urry of Tufts University. The coauthors on the paper are her entire research methods class, who conducted the work as a class project (How cool is that? Shout out to Dr. Urry's students!). 

The work is an excellent example of a replication study. It was covered in a few media outlets . Here is one summary, as reported by the APS website:

But although previous work suggested that students who write notes by hand may retain information better than those who type them out on a laptop, attempts to replicate this research indicate that taking notes longhand may offer no measurable benefit. 

“We found only small, statistically nonsignificant differences in quiz performance as a function of note-taking medium,” whether students used a laptop or pen, wrote APS Fellow Heather L. Urry (Tufts University) and colleagues in Psychological Science. 

Here's some coverage of the same study, in a blog post by Dr. Regan Gurung, on Psychology Today:

There was no difference in how well the students performed on the factual recall questions as a function of how they took notes. This was similar to the 2014 study. The difference, and failure to replicate, came when Urry et al. analyzed performance on the applied questions. Again, there was no difference in scores based on note-taking style.

In the APS summary, it is reported:

Urry and colleagues reached this conclusion through a direct replication of a 2014 study by Pam A. Mueller (Princeton University) and APS Fellow Daniel M. Oppenheimer (University of California, Los Angeles), in addition to a mini meta-analysis of eight related studies. 

a) As a review, what were the IV (and its levels) and DVs in the original study? (hint: There are two DVs)

b) What does the term "direct replication" mean in the previous quote?

c) How does direct replication compare to a conceptual replication? (You'll find these terms in Chapter 14)

 

Now let's talk about the term "mini meta-analysis" in the quote, "...in addition to a mini meta-analysis of eight related studies."   Here's a summary from the Psychology Today blog:

Urry et al. (2021) also found 8 similar studies, that also aimed to replicate the 2014 results.  The studies compared also used similar methodologies with slight variations. The effect of taking notes longhand was not statistically significant.

d) Do you think the eight studies  in the mini meta-analysis were direct replications or conceptual replications? How can you tell? 

e) Now explain, in your own words, how they might have done the mini meta-analysis? What would the authors have done, once they found the eight studies? (Why do you think it was called "mini"?)

f) Dr. Urry's team preregistered their replication study--what does preregistration mean in this context? 

g) While the original Mueller and Oppenheimer study was covered by multiple popular press outlets (such as here, here, and here), the replication study has, so far, not really broken through to the popular press. Why do you think that is? 

You should check out forest plots of the mini meta-analysis!  Here's the link to the journal article. If you're locked out, you can access a preprint of the replication study here. 

People believe claims of sexual harassment more when women are attractive

The coded stereotypicality of the drawing was a dependent variable in one of the studies. Photo: Courtesy of Dr. Jin Goh. Citation: Goh, J. X., Bandt-Law, B., Cheek, N. N., Sinclair, S., & Kaiser, C. R. (2021). Narrow prototypes and neglected victims: Understanding perceptions of sexual harassment. Journal of Personality and Social Psychology. Advance online publication. https://doi.org/10.1037/pspi0000260

Sexual harassment (unwelcome and inappropriate sexual remarks or physical contact) harms people in schools and workplaces. One potential problem with sexual harassment is whether people who report it are taken seriously. Unfortunately, when people report sexual harassment, their claims are not always believed, which means perpetrators can go uninvestigated and victims' harassment might continue. 

The press recently covered an important series of studies that investigated the role that physical appearance plays in the reporting (and believing) process. One of the summaries appeared in the Seattle Times; another  appeared on the APA website.  The Seattle Times journalist summed up the studies like this: 

The study found people are more apt to believe sexual harassment claims by women who are young, “conventionally attractive,” and appear and act feminine. Women who don’t fit that prototype not only are less likely to be believed, but also are presumed to be unharmed by harassing behavior, the study said. 

The APA website does a better job detailing the methods:

The researchers conducted a series of 11 multimethod experiments, involving more than 4,000 total participants, designed to investigate the effect a victim’s fit to the concept of a typical woman had on participants’ view of sexual harassment and the consequences of that mental association.

In five of the experiments, participants read scenarios in which women either did or did not experience sexual harassment. Participants then assessed the extent to which these women fit with the idealized image of women, either by drawing what they thought the woman might look like or selecting from a series of photos. Across all the experiments, participants perceived the targets of sexual harassment as more stereotypical than those who did not experience harassment.

The details above describe a set of experiments (do you know why they are experiments?)

a) What was the main independent variable (IV) in the five studies above? What were its levels? 

b) What was the main dependent variable (DV) in the five studies? What do you think its levels were? 

c) The image (with the two drawings, above) that goes with this post came from the first study (Study 1), in which participants read a scenario in which " women either did or did not experience sexual harassment" and made a "drawing what they thought the woman might look like". Which drawing do you think came from each condition? 

d) The drawings from Study 1 had to be coded for stereotypical femininity. Since this variable required coding, inter-rater reliability would have been important. How do you think the researchers established inter-rater reliability for their coders? (Hint: you can find relevant information in Chapter 5.)

e) Sketch a small bar graph depicting the results of these studies, based on the journalist's description. 

Now here's some more description from the APA website:

In the next four experiments, participants [read] ambiguous sexual harassment scenarios, such as a boss inquiring about a woman’s dating life. These scenarios were paired with descriptions or photos of women who were either stereotypical or not. The participants then rated the likelihood that the incident constituted sexual harassment.

f) What was the main independent variable (IV) in these four studies? What were its levels? 

g) What was the main dependent variable (DV) in the four studies? What do its levels seem to have been? 

h) Sketch a small bar graph depicting the results based on the journalist's description. 

i) Assess the plausibility of this claim: "Being more stereotypically feminine causes people to think a woman's sexual harassment claim is true"  Apply the three causal criteria (covariance, temporal precedence, and internal validity). 

Instructors: Students are likely to enjoy reading the studies in the original journal article, which appears on my list of journal articles that are student-friendly.

Interrogating Buzzfeed: Does Instagram really have "huge effects" on our psychology?

Do the studies really support cause and effect? And are those effects "huge"? Photo: Julia_Tim/Deposit Photos

This Buzzfeed headline is eyecatching and dramatic: Instagram Has Some Pretty Huge Effects On Our Psychology. Here's Everything We Know So Far.

I see two important issues to interrogate in this headline. First, it's a causal claim ("Instagram has some...effects on our psychology"). Second, it's making an argument about effect size ("pretty huge effects"). Let's see if they pan out. 

The article focuses on body image and mental health. Here's an example:

The research into how Instagram affects our psychology is relatively new, but there is a growing body of evidence to say that it has a profound effect on the way we perceive ourselves and the world around us.

Does the research support such a causal claim? Here's one study that the journalist summarized: 

Earlier this year, the UK's Royal Society for Public Health (RSPH) and the Young Health Movement published a report that looked at the impact of different social media platforms on mental health.

The report was based on a survey of over 1,400 British 14-to-24-year-olds that asked about the positive and negative implications of each social media platform for health and wellbeing.Instagram scored well on its ability for self-expression, but it was linked with some of the worst scores for sleep loss, body image concerns, and FOMO.

It's not completely clear from this summary what questions were included on this survey. However, a survey is unlikely to have manipulated exposure to Instagram, so it can't support the claim that "Instagram has implications for how people perceive themselves and the world around them."

What about the effect size of the associations? The journalist didn't report the data from the study, so we can't evaluate whether or not the effect was really "huge" or "profound." We'd have to search out the original paper for that. 

How about this other study, summarized later in the article: 

In a study published last year, Cohen and her co-authors found that it wasn’t so much social media use itself that impacted how people perceived their bodies but rather how they were using it.

Following body- and appearance-focused accounts on Instagram — as opposed to neutral accounts dedicated to things such as travel — was associated with idealizing thin bodies, obsessive surveillance of one’s own body, and striving for thinness.

a) Four variables are listed in the study above (Following body-....) What are they?

b) Were these variables measured or manipulated, do you think?  (Check your hunches here, by reading the abstract of the Cohen study.)

c) Why can't this study support a claim such as "following body-focused accounts impacts how people perceive their bodies"? Apply the three criteria for causation. 

d) Does the journalist provide any indication of the effect size of this association? If not, what data might have helped you decide if the link was "huge"?

Let's read a bit further in the journalist's article to see if we can find some experimental evidence. How about this one:

However, Cohen is hopeful that the positive body movements emerging on the platform will have far-reaching benefits for how women reflect on their own physiques.

She has recently submitted a paper for publication demonstrating that among 195 young women exposed to “body-positive” images, mood, body satisfaction, and body appreciation were all improved significantly.

The wording in the example above suggests that there was a manipulated variable, because it reads "among 195 young women exposed to 'body-positive' images'". When I read that, I picture that some people were exposed to these images, and some were not. 

e) Imagine you were going to conduct a conceptual replication of this experiment. What was the independent variable? What would its levels be? 

f) What would the dependent variable(s) be? (Hint: The journalist mentions three).

g Sketch a little bar graph of the results of your experiment--picking one of the DVs to graph. In your view, how far apart would the means and error bars have to be for you to be convinced that the effect is "huge"?

h) Would this experiment have the potential to support a causal claim? What confounds would you need to control for? 

Suggested answers to selected questions

c) Covariance: Yes--apparently this questionnaire study found a link between following more body-focused accounts and idealizing thin bodies more. 

Temporal precedence: Nope--this was a questionnaire study that measured both variables at the same time. We can't know if the IG accounts came first, or if people who idealize thin bodies more are more likely to want to follow body-focused accounts.

Internal validity: Also nope. In this correlational study we can't rule out every possible third variable. For example, maybe people with a history of eating disorders are both more likely to follow body-focused accounts and are more likely to idealize thin bodies.

d) No correlation was reported here--too bad! It would be nice if the journalist had given us some benchmarks indicating how much more likely people were to idealize thin bodies....perhaps a percentage change or a correlation coefficient, and a 95% CI for that change or coefficient. 

 

Thanks to UD student Emily Hogan for suggesting this article for the blog!

 

 

Don't believe it?

Macadamia nuts are in the same evolutionary family as peaches. (or are they?) Credit: gresey/Deposit photos

People's tendency to believe, share, and act upon disinformation (fake news) is a serious threat to civic engagement. Psychologists are studying what makes people more or less likely to believe false claims. Here are some excerpts from a brief article published in Buzzfeed, which summarized a few studies about people's willingness to believe what they read.

"What we find in our research really robustly is that we can push around people's feelings of truth using things like photographs and audio quality, and even the ease of pronouncing someone's name," said [researcher] Newman.

Here's an example of a study on how perceptions of "truthiness" can be influenced:

In 2012 Newman co-authored another paper with studies that found photographs can inflate the sense of "truthiness" that people get from reading a statement.

The sort of photographs that the researchers used for this research were "non-probative", meaning that they didn't actually contain any information about the claims being made.

One study ...found that if participants were given a trivia statement such as "Macadamia nuts are in the same evolutionary family as peaches", they were more likely to believe the statement was true if a photograph of macadamias simply appeared alongside the sentence [compared to no photo].

Bonus: Stephen Colbert (who coined the term "truthiness") also summarizes it in this clip. 

Questions about the first example:

a) What are the two variables in the macadamia nuts study? Name each variable, its apparent levels, and indicate if it is measured or manipulated.

b) Do you think the "photo appearance" variable was manipulated as between-groups or within-groups? Explain your reasoning.

c) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

d) Ask three questions about the statistical validity of the macadamia nut study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

 Here's a second example:

A study published in March found that people were less likely to believe an academic researcher on the radio if the audio quality of the recording was poor or glitchy.

The study involved 97 participants, who were instructed to listen to recordings of different pieces of science news and assess the quality of the research and the researcher who was reporting it.

Half of the audio recordings were edited to sound as though they were recorded on a bad phone line. The listeners who heard these recordings rated the credibility of the researcher and the studies they were discussing as significantly less favourable than recordings with higher audio quality.

Questions about the second example:

e) What are the two variables in the audio quality study? Name each variable, and indicate if it is measured or manipulated.

f) Do you think the "audio quality" variable was manipulated as between-groups or within-groups? Explain your reasoning.

g) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

h) Ask three questions about the statistical validity of the audio quality study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

i) Now consider the internal validity of the audio quality study: Why would random assignment be important in this study? 

Selected answers (use these answers as models to help answer the audio quality questions, if needed)

a) One variable is the appearance of a photo; its levels are "photo or no photo." It is manipulated.
A second variable is "level of belief in the trivia statement" or "rated truthiness." Its levels probably range from "low belief" to "high belief". It is measured. 

b) For a single fact, the presence of a photo would have to be manipulated as between groups--people would have to see the fact either with or without the photo. If they were exposed to both levels of the variable (a within-groups design), people might wonder why they are being asked to rate the same fact two times.  However, in the study itself, participants were exposed to multiple facts, some of which were accompanied by a photo and some of which were not....so that makes the variable within-subjects when you consider all of the facts together. 

c) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

d) Ask three questions about the statistical validity of the macadamia nut study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

For effect size, you could ask questions such as, "how far apart were the truthiness ratings?" "How big is the difference between the photo and no-photo group?" or "What is the d? (or other effect size)"
For precision you could ask, "what is the 95% CI of the effect size estimate?"
For replication you could ask, "has this effect been replicated in other studies? What do those other studies say?"

I would like to thank my student Emily Hogan for sharing this example and Buzzfeed story with me!

 

Will an "awe walk" do wonders for your well-being?

The study asked all volunteers to go on a daily walk, but randomly assigned only some of them to seek out experiences of awe.  Credit: Studio Grand Web/Shutterstock

A recent headline reads, "An 'awe walk' might do wonders for your well-being."  Here is the journalist's summary of an empirical journal article.

The journalist starts by explaining that psychological researchers already know that exercise can improve mood. And we also already know that a feeling of awe can improve mental health. The journalist introduces the concept:

A somewhat nebulous emotion, awe generally is defined as the sense that you are in the presence of something larger and more consequential than yourself and that this something is mysterious and ineffable. 

In the study,

They concentrated on people in their 60s, 70s and 80s, an age when some people can face heightened risks for declining mental health.

Read the journalist's description of the study, identifying the study's independent variable, dependent variables (there are more than one DV), and control variable(s):

The scientists asked 52 of the study volunteers if they would mind adding a weekly 15-minute walk to their normal schedules. ...Fresh, baseline studies of their mental health showed they were psychologically well-adjusted as well, with little anxiety or depression.

The scientists randomly divided these volunteers into two groups. One, as a control group, was asked to start walking, at least once a week, for 15 minutes, preferably outside, but given few other mandates.

The members of the other group likewise were asked to walk once a week, but also were instructed in how to cultivate awe as they walked.

Not surprisingly, they found that the awe walkers seemed to have become adept at discovering and amplifying awe. One volunteer reported focusing now on “the beautiful fall colors and the absence of them among the evergreen forest.” A control walker, in contrast, said she spent much of a recent walk fretting about an upcoming vacation and “all the things I had to do before we leave.”

The researchers also found small but significant differences in the groups’ sense of well-being. Over all, the awe walkers felt happier, less upset and more socially connected than the men and women in the control group. The volunteers in the control group reported some improvements in mood, but their gains were slighter.

Questions

a) What was this study's IV? What were its levels? Was the IV manipulated as independent groups or within groups? 

b) Name three distinct DVs from this study, based on the journalist's description.

c) Which type of experiment is this? Posttest only? prettest-posttest? repeated measures? concurrent measures?

d)  Pick one of the DVs you listed in (b). Sketch a bar graph of this DVs. (Hint: you are going to need to plot four data points--do you see why?).

e) Construct validity question: What's going on in the paragraph that starts, "Not surprisingly, they found that the awe walkers seemed to have become adept at discovering and amplifying awe."? Use the term "manipulation check" in your response, and explain why a manipulation check is relevant to construct validity.

f) Statistical validity question: The journalist describes "small but significant differences". What does this phrase mean? Use the term "effect size" in your answer. 

g) External validity question: Can we generalize from this sample, or is generalization unknown? defend your answer (but please don't refer to sample size!)

h) Internal validity question: What role did random assignment play in this study? What if somebody says, "some people just like going for walks"--would you consider this to be a threat to internal validity? Why or why not? 

The journalist also mentions another DV. The researchers asked people to take a couple of selfies during their walks:

Over the course of the eight weeks, the size of awe walkers’ countenances shrank in relation to the scenery around them. Their faces grew smaller, the world larger. Nothing similar occurred in the photos from the control group.

i) It sounds like the researchers did not preregister prediction about the selfie content. Is lack of preregistration a potential problem?

 

 

 

Podcast: WOOP, WOOP!

 

Wish, Outcome, Obstacle, Plan: An evidence-based app. Photo courtesy of Dr. Oettingen/Woopmylife.

Here's another podcast-based post. NPR's Hidden Brain podcast frequently uses psychological research to illuminate a variety of psychological topics. Today's post is based on a Hidden Brain episode called You 2.0: WOOP, WOOP!

 

The episode asks,"is there a methodical way to achieve your dreams?"

The episode starts off with the suggestion that maybe positive thinking alone isn't the best way to achieve your dreams. Of course, in the United States, we are encouraged to "dream big" and shoot for the stars. But as host Shankar Vedantam interviews researcher Dr. Gabriele Oettingen, we learn that positive thinking--by itself--may not be very helpful. 

At minute 1:40,  you hear how Dr. Oettingen first started getting interested in positive thinking--it stems from a cultural difference she observed between Germany (her home country) and the U.S.  Then Dr. Oettingen lays out the theory. She explains that positive thinking feels good, but...

(2:45)...these positive fantasies--they seduce us to feel already accomplished, and they take our energy away.

Dr. Oettingen describes a series of correlational findings:

(3:00) We found that, for example, the more positively women enrolled in a weight reduction program fantasized about their success in the program, the fewer pounds they lost--three months later, one year later, two years later.

Or the more positively university graduates fantasized about an easy transition into work life, the fewer dollars they earned two years later, the fewer job offers they had gotten, and the fewer job applications they had sent out.

a) Consider the dieting study above. Sketch a well-labeled scatterplot of the correlation she describes (one axis should have "level of fantasizing" on it, and pick one of the weight loss variables for the other axis).

b) Does this study, by itself, show that fantasizing caused these women to lose less weight? (Apply the three causal criteria).

c) Now consider the work life study above. Sketch a well-labeled scatterplot of the correlation she describes (one axis should have "level of fantasizing" on it, and you should pick only one of the job-related outcomes for the other axis).

d) Does this study, by itself, show that fantasizing caused these graduates to do worse on the job market? (Apply the three causal criteria).

In the podcast, Dr. Oettingen shared two other examples: One study on surgery patients and another study on people who had a crush on another person. [Instructors, you could consider having four different groups of students graph each of the four studies.]

e) What are the advantages of doing this same correlational study in four samples (dieters, graduates on the job market, surgery patients, and people with a crush?) Use the term "external validity" in your answer. 

Now let's move on to the rest of the theory. Dr. Oettingen's theory proposes that fantasizing by itself makes people relax--"because mentally, they're already there"--they've achieved the goal. It's better, she argues, to pair your fantasies with thoughts about obstacles. Starting at minute 9:00, Oettingen introduces the term mental contrasting, in which people follow their fantasies with thoughts about the potential obstacles, asking the question: "What is it in you that stands in the way?" 

Listen from minutes 9:00 to 15:30, as Oettingen walks through the WOOP process of activating wishes, fantasizing about the outcome, anticipating obstacles, and making a plan. (WOOP stands for Wish, Outcome, Obstacle, Plan.) 

Oettingen and her team are well aware of the limitations of correlational evidence like that introduced above.  So, although it's not described in the podcast, her team has also conducted experimental studies. In her experimental work, Oettingen and her team have randomly assigned people to engage in WOOP and others to engage in the activities in a different order (perhaps Obstacle, Wish, Outcome). They find that people in the WOOP conditions are more likely to achieve their goals.

f) If you were designing a study for people in a weight loss program, how would you design an experimental study to test if WOOP thinking would lead to more successful weight loss? 

i) What would the IV be (What would its levels be)? What would the DV be?
ii) How would you assign people to the conditions? 
iii) How would you eliminate internal validity threats? 
iv) Sketch a bar graph of your predicted outcome. 

Are you interested in trying WOOP out on your own goals and wishes? The phone app is free, and you can also access it online: www.woopmylife.org.

 

 

 

Too sleepy to choose that major

These students might misattribute their early-morning fatigue to their interest in the subject. Photo credit: Gorodenkoff/Shutterstock

Podcasts have been gaining in popularity in the last few years, and several feature psychological research. I'm going to be devoting my next few posts to some podcasts. We'll start with this very short, 3-minute episode/ interview from the NPR podcast, Hidden Brain. The episode is called "How does the way you feel shape your life?" and it's about how students might be influenced by situational factors (such as how early a class is) when they select their majors. 

The podcast opens with a story:

...behavioral economist Kareem Haggag at Carnegie Mellon University... sometimes teaches early morning classes. And he's worried that when his students are tired or sleepy, they might draw the wrong conclusion about him and about the class.

The researcher himself states (around minute 1)

DR. KAREEM HAGGAG: Our hypothesis was that students who are assigned to an early morning section of the class, or to multiple back-to-back classes before a class, might mix up how tired they are in that class with how much they like the subject, thus leading them to be less likely to choose the subject as their major.

The host, Shankar Vedantam, reports:

This is a psychological phenomenon called misattribution. You know, you go to an amusement park on a sunny day and you think the park is great. You go when it's raining - you think the park is terrible. You're unconsciously confusing your feelings about the weather for the quality of the park.

The researchers used thousands of data points from students at the U.S. Military Academy at West Point to test their hypothesis. As Vedantam points out:

... students at West Point are randomly assigned classes that are scheduled at different times of the day, which means that some students are taking this introductory class first thing in the morning. Some students are taking the very same class later in the day. The researchers then measure the chance that students would later on choose that subject as their college major.

The researchers, led by Dr. Haggag, found that students were more likely to major in the topic of that introductory class if they'd taken it at a later time of the day: 

HAGGAG: We find that students who are randomly assigned to the first period, 7:30 a.m. section, are about 10% less likely to choose the corresponding major compared to a student who takes that class later in the day. 

Questions

a) What are the two main variables of interest in this study (the independent and dependent variables)? What are the levels of each variable?

b) Is this a true experiment or a quasi-experiment? If it's a true experiment, what kind is it (prettest-posttest, posttest-only, repeated measures, concurrent measures)?

If it's a quasi-experiment, which of the four types is it (nonequivalent control groups, nonequivalent control group prettest-posttest, nonequivalent control groups interrupted time series, nonequivalent control groups interrupted time series)?

c) Which is the IV and which is the DV here?

d) Sketch a small bar graph of the results they describe.

e) Can these researchers support the causal claim that "having an early morning introductory class makes you less likely to want to major in that discipline?" Apply the three rules of causation: covariance, temporal precedence, and internal validity in your response.

f) Ask a question about this study's internal, external, statistical, and construct validity.

Bonus content:

In the study above, the main IV was "time of day of the introductory class". As a bonus, here is another independent variable they studied--"number of breaks before the introductory class":

DR. KAREEM HAGGAG: We also compared two students who are sitting in the exact same classroom, but one of whom just had a free period as a break before, and the other came from one or more back-to-back classes. We find that each additional back-to-back class reduces the likelihood that that student enrolls in the major by about 12%.

g) Which is the IV and which is the DV here?

h) Sketch a small bar graph of the results they describe.

i) Can these researchers support the causal claim that "having breaks before your introductory class makes you more likely to want to major in that discipline?" Apply the three rules of causation: covariance, temporal precedence, and internal validity.

 

 

Lion attacks on cattle: A field experiment

Researchers painted eyes like these on some of the cattle in each herd. Credit: Cameron Radford

Here's an example of a field experiment that took place in actual fields. The empirical article appeared in Nature Communications, and the authors of the research wrote their own summary for a popular audience in The Conversation. The quotes below come from their popular audience summary. 

In this study, the authors worked with cattle ranchers in Botswana who were trying to reduce the number of lions and leopards who attacked their herds.

The introduction outlines the theory the researchers were testing:

Many big cats – including lions, leopards, and tigers – are ambush predators. This means that they rely on stalking their prey and retaining the element of surprise. In some cases, being seen by their prey can lead them to abandon the hunt. We tested whether we could hack into this response to reduce livestock losses to lions and leopards in Botswana’s Okavango delta region.

Below are some details of the method of their study. It sounds like an experiment: Can you see why?

Working with Botswana Predator Conservation and local herders, we painted cattle from 14 herds that had recently suffered lion attacks. Over four years, a total of 2,061 cattle were involved in the study.

Before release from their overnight enclosure, we painted about one-third of each herd with an artificial eye-spot design on the rump, one-third with simple cross-marks, and left the remaining third of the herd unmarked. We carried out 49 painting sessions and each of these lasted for 24 days.

The cattle were also collared and all foraged in the same area and moved similarly, suggesting they were exposed to similar risk. However the individuals painted with artificial eye-spots were significantly more likely to survive than unpainted or cross-painted control cattle within the same herd.

In fact, none of the 683 painted “eye-cows” were killed by ambush predators during the four-year study, while 15 (of 835) unpainted, and 4 (of 543) cross-painted cattle were killed.

Questions

a) What were the independent and dependent variables in this study? What were the levels of each variable?

b) What kind of experiment was this: Posttest-only? Prettest-posttest? Repeated measures? Or concurrent measures? (Challenge question: How might your answer change if you take the predator's perspective, as compared to the cattle's perspective?)

c) Sketch a bar graph of the results of the study. (Later you can compare your sketch to the figures in the Nature article).

d) The authors don't seem to specify here whether or not they used random assignment to the three painting conditions. Why would random assignment be important?

e) Ask a question about the external validity of this study. (Which term will your answer involve: random assignment or random sampling?) 

At the end of their popular press summary, the authors raise two important limitations:

First, it is important to realise that, in our experimental design, there were always unmarked cattle in the herd. Consequently, it is unclear whether painting would still be effective if these proverbial “sacrificial lambs” were not still on the menu. Further research could uncover this, but in the meantime applying artificial marks to the highest-value individuals within the herd may be most pragmatic.

Second, it is important to consider habituation, meaning that predators may get used to and eventually ignore the deterrent. This is a fundamental issue for nearly all non-lethal approaches. Whether the technique remains effective in the longer term is not yet known in this case.

f) How might you design a study to test the first limitation? That is, how could you manipulate the eye-paintings variable in order to test if this intervention still works if there are no "sacrificial lambs" in the herd? What would your different conditions look like?

g) How might the authors test the second limitation? What can they change about, or add to, their study to test whether "the technique remains effective in the longer term"?

 

 

 

Does "virtue signaling" change minds?

If a lot of us send virtue signals, it can communicate to others that norms are changing. Credit: SEEphotos7171/Shutterstock"

Among people fighting for racial justice, "virtue signaling" may sometimes be derided as empty action. "Virtue signals" are public messages of support for a social justice cause. They could take the form of a yard sign, a t-shirt, or a social media post. Critics sometimes argue that such messages are easy and cheap. It's easy for a company to put a message on their home page that supports Black Lives Matter. It's not hard to wear a t-shirt that supports a social justice cause. But are these expressions of support all talk and no action?

An article in Time magazine argues that virtue signaling can be more powerful than we think. The authors, psychological scientists Jamil Zaki and Mina Cikara, integrate several studies to make their argument. Here's part of their introduction: 

It’s critical to question people whose most radical act is temporarily changing their profile picture. It’s critical they follow their words with action; the time for talk has passed. But focusing solely on virtue signalers obscures their most important role. Like radio waves, the signals they send are received by someone, and those receivers matter, too.

Zaki and Cikara argue that virtue signaling is an important way of influencing others. If a lot of people change their profile pictures to support BLM, for example, more people will start to believe that this movement is popular and they may adopt it, too:

Opinions can work this way: when a particular viewpoint gets a lot of attention, people assume it’s popular, and shift towards it. We receive signals, and are changed by them.

What studies support their argument? Here's one that the authors summarize: 

As social norms shift, individuals shift with them: adopting popular opinions and behaviors, and dropping ones that fall out of style. Norms are especially powerful as they gain steam. In one set of studies, Gregg Sparkman and Greg Walton presented diners in a cafeteria [either] with evidence that 30% of the U.S. population was vegetarian, or that 30% of the population was newly vegetarian. People who learned about this second, “dynamic social norm” were twice as likely to order a meatless lunch themselves. They saw not just where the crowd was, but where it was going, and didn’t want to be left behind.

Here are some questions about the study summarized above:

a) Was this an experiment or a correlational study? How do you know? (If it's an experiment, what kind of study is it?)

b) What were the key variables in the study? Were they manipulated or measured? 

c) Sketch a small graph of the results being described. Would a bar graph or a scatterplot make more sense?

Here's another study Zaki and Cikara introduce:

Signaling and conformity are not inherently positive. In 2016, the U.S. elected a president whose campaign was jammed with racist talking points. Prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants). 

d) This summary doesn't really help you figure out how the study was designed. But here's a chance to think carefully: What kind of study would we need to conduct in order to test the claim that "prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants)."?  There are three variables you would need to include in your study. What are they? 

 

Suggested answers to selected questions

a) This was probably a posttest-only experiment, because it sounds like they manipulated the message about vegetarianism and measured what foods people took in the cafeteria. Here is a link to the original empirical article. 

d) The key dependent variable in this study is prejudice levels, so that's the first variable. Then, in order to establish that "prejudice increased", you'd need to measure prejudice both before and after the election, so that's another variable. Finally, in order to show that it increased specifically toward groups he targeted, you'd need to measure different types of groups (Muslims, immigrants, as well as other potential groups that Trump did not deride in his campaign). You can review the Crandall article here. Do their results really support what Zaki and Cikara are saying about it? (Pay attention to the difference between people's actual prejudice and the normativity of prejudice.) 

 

 

 

 

Saturated fat can impair concentration

Breakfast of (concentration) champions? Maybe not....Photo credit: Kyndall Ramirez/Unsplash

A headline at Psych Central reads, "High fat meal may impair concentration." Does the study behind the headline support that causal claim?

At first, you might assume that this came from a correlational study on diet and concentration. However, it turns out the research summarized in the article was experimental: It manipulated the saturated fat content of one specific meal. In the study, researchers created two versions of a breakfast--one made with saturated fat and the other made with sunflower oil, which is also high in dietary fat but is not the saturated type. 

The journalist describes the method of the study this way: 

Women in the study [n = 52] completed a baseline assessment of their attention during a morning visit to the lab. The tool, called a continuous performance test, is a measure of sustained attention, concentration and reaction time based on 10 minutes of computer-based activities.

The high-fat meal followed: eggs, biscuits, turkey sausage and gravy containing 60 grams of fat, either a palmitic acid-based oil high in saturated fat or the lower-saturated-fat sunflower oil. Both meals totaled 930 calories and were designed to mimic the contents of various fast-food meals such as a Burger King double whopper with cheese or a McDonald’s Big Mac and medium fries.

Five hours later, the women took the continuous performance test again.

Between one and four weeks later, they repeated these steps, eating the opposite meal of what they had eaten on the first visit.

After eating the meal high in saturated fat, all of the participating women were, on average, 11 percent less able to detect target stimuli in the attention assessment.

(The original empirical journal article is here; it might be paywalled)

Questions:

a) What was the independent variable (IV) in this study and what were its levels?

b) What was the dependent variable?

c) Was the IV manipulated as independent groups or within-groups?

d) This design is difficult to classify because it has a pretest and posttest, but the women also experienced both levels of the IV. Therefore it shares features with two of the four designs in Chapter 10. What are those two designs it shares features with? (Note to students: As you can see in this example, the four designs you learned about in Chapter 10 are not the only types of experiments there are, and this study provides an example of alternative designs.) 

e) Sketch the results of this study--using "reduction in concentration" (i.e., a difference score) as the dependent variable. 

Next let's work through the four big validities. 

f) Here's some text related to external validity of this study:

For this work, Madison [the lead author] conducted a secondary analysis of data from Kiecolt-Glaser’s study assessing whether high-fat meals increased fatigue and inflammation among cancer survivors.

and

[the experimental meals]  were designed to mimic the contents of various fast-food meals such as a Burger King double whopper with cheese or a McDonald’s Big Mac and medium fries.

Given the information above, to what kinds of people and what kinds of situations might this research be generalized? How do you know? 

g) The researchers found an 11 percent drop in concentration after the meal high in saturated-fat. What further information would you like to know in order to evaluate the study's statistical conclusions? 

h) Can this study support the causal claim that "A high-fat meal may impair concentration"? Apply the three criteria to the study: Covariance (what did the results show?), temporal precedence (did the method ensure that the meal came before the drop in concentration?), and internal validity (What potential confounds did they control for in the study?) 

i) Here's some information about the study's construct validity: 

Women in the study completed a baseline assessment of their attention during a morning visit to the lab. The tool, called a continuous performance test, is a measure of sustained attention, concentration and reaction time based on 10 minutes of computer-based activities.

What questions would you need to ask to determine if this test is a good measure of people's ability to concentrate? 

Instructors: If you teach Chapter 12, you can also use this example as a mixed factorial design, because the study also included a participant variable of endotoxemia, a condition called "leaky gut." From the journalist's summary: 

After eating the meal high in saturated fat, all of the participating women were, on average, 11 percent less able to detect target stimuli in the attention assessment. Concentration lapses were also apparent in the women with signs of leaky gut: Their response times were more erratic and they were less able to sustain their attention during the 10-minute test.

“If the women had high levels of endotoxemia, it also wiped out the between-meal differences. They were performing poorly no matter what type of fat they ate,” Madison said.

Students could identify the IV, PV, and DV and practice graphing the results described above. 

Thanks to my colleague Stephen Chew, who shared this example.