Everyday Research Methods

Evidence in favor of U.S. stimulus checks

Which type of quasi-experimental design does this study seem to fit best? The dark line on top represents respondents with children. The red line represents all respondents. The lighter blue line at the bottom represents respondents without children. Credit: Courtesy Patrick Cooney and H. Luke Shaefer. From Poverty Solutions, May 2021, University of Michigan,

Since the COVID-19 pandemic began, several governments, including the U.S. federal government, have distributed cash payments to citizens in an attempt to alleviate the  financial hardship caused by the global shutdown. 

As the New York Times reports here, policymakers wonder if such payments are useful. Now we have data to help address that research question. 

In offering most Americans two more rounds of stimulus checks in the past six months, totaling $2,000 a person, the federal government effectively conducted a huge experiment in safety net policy. Supporters said a quick, broad outpouring of cash would ease the economic hardships caused by the coronavirus pandemic. Skeptics called the policy wasteful and expensive.

[...]

A new analysis of Census Bureau surveys argues that the two latest rounds of aid significantly improved Americans’ ability to buy food and pay household bills and reduced anxiety and depression, with the largest benefits going to the poorest households and those with children. The analysis offers the fullest look at hardship reduction under the stimulus aid.

The full report is available here, and provides some excellent graphed examples of quasi-experimental data. We'll focus on Figure 1 from the report, reprinted above with permission from the authors.

The y-axis on the figure represents the percentage of Americans who "sometimes or often had not enough food to eat in the last seven days." The question is asked by U.S. Census' Household Pulse survey about once per month to a random sample of Americans. The dates of the two U.S. stimulus checks are indicated in the laddered, light-blue vertical lines. Payments occurred in December, 2020 and March, 2021.

Questions

a) The journalist writes that "the federal government effectively conducted a huge experiment in safety net policy".  What is the independent variable in this "experiment?"  What is the dependent variable (i.e., the one depicted in Figure 1)?

b) The independent variable here is not a true IV--it is a quasi-independent variable. What makes this variable quasi-independent?

c) The authors claim that the results from Figure 1 are consistent with the argument that the stimulus checks helped people. The authors wrote in their report: "Based on the speed with which we see hardship fall, we suspect much of this drop was the result of EIP checks, which the federal government was able to quickly deliver to bank accounts for most U.S. households following passage of both bills." Locate the areas on the figure that the authors are referring to. 

d) This quasi-experimental study is probably best described as an interrupted time-series design. What is the "interruption" here? What is the time series?

e) A history threat is one potential internal validity problem in an interrupted time-series design. Explain what a history threat is and also why a history threat might be a problem here. What do you think--to what extent can we be sure it was the stimulus checks, and not some other pair of events, that is responsible for the increased food security depicted here? 

You can see results for other variables in the report here, including figures that tracked people's feelings of anxiety and their feelings of depression. 

 

 

 

 

 

Estimating how many U.S. adults identify as nonbinary

 

Photo credit: nito103/Deposit Photos

Press outlets have been covering a report by the Williams Institute (based at UCLA), a research organization collecting data on demographic information on sexual orientation and gender identity. The quotes in this post come from this summary from PBS.

The new study has estimated the number of LGBTQ adults in the U.S. who identify as nonbinary. (Nonbinary is a term that is often used to describe people whose gender identities are more than the man/woman category; some nonbinary people use they/them pronouns.)  It's important to estimate how many U.S. adults identify as nonbinary (as well as other sexual- and gender-minority identities) because anti-discrimination laws potentially affect them. PBS reports that:

Nonbinary people are part of a “growing subgroup” of the LGBTQ population, Wilson said, adding there’s been greater visibility for nonbinary people. More and more celebrities, like singers Demi Lovato and Sam Smith, and actors Indya Moore and Amandla Stenberg, have talked about identifying themselves as nonbinary or in related terms.

Here's a portion of the PBS summary: 

According to the study, which focused on people ages 18 to 60, nonbinary people make up about 11 percent of the adult LGBTQ population in the country. The study builds on the work of two prior surveys — one on trans adults, [TransPop] the other on cisgender LGBTQ adults [Generations]— to provide more comprehensive data on nonbinary people, a historically underresearched subgroup.

When the researchers extrapolated out form this subsample, they concluded that

There are about 1.2 million LGBTQ adults in the U.S. who are nonbinary.

The data in this report constitute a frequency claim, because they are estimating the level of one variable (that is, nonbinary identity).

First let's talk about the precision of these estimates. The report from the Williams Institute (here) includes 95% CIs for the estimates. For example, the two estimates from the PBS story were presented this way:

Number of nonbinary LGBTQ adults in the U.S., including transgender and cisgender people N [95% CI]	1,219,000 [1,000,000 - 1,481,000]
% [95% CI] of LGBTQ adult population	11.1 [9.3, 13.2]

a) In your own words, describe what the first CI means. Use this definition and template to help you: The 95% CI is meant to capture the true value in the population 95% of the time. The true value in the population might be as low as ____ or as high as ____. 

b) Sometimes the 95% CI misses the true value in the population. What percentage of the time will the CI miss?

Next lets' talk about external validity (generalizability) and sampling. As you consider these estimates keep in mind that there is a distinction between estimating the number of all U.S. adults who identify as non-binary and the number of U.S. LGBTQ adults who identify as non-binary. These values may be different. 

c) Given what you've read so far, what seems to be the population of interest of this study? 

To know if the sample is generalizable, you need to know how they obtained the sample. The PBS summary doesn't say anything about the sampling method used in this study, and neither does the Williams Institute report. To locate information about the sampling, I had to go to the methods sections of the two studies on which the nonbinary study is based: The TransPop study (which studied a sample of transgender U.S. adults) and the Generations study (which studied a sample of LGBTQ U.S. adults). 

In these methods sections, they say that these two studies (TransPop and Generations) worked with the Gallup organization to recruit nationally representative samples of transgender and LGBTQ adults. Gallup performs a Daily Tracking Survey in which they call a national probability sample of 1000 adults, almost every day of the year. These adults are sampled using random-digit dialing, including both landline and cell phone numbers. Gallup asks a variety of questions about well-being, political attitudes, and economic situations. At the end of each  telephone survey, the interviewer asks, “I have one final question we are asking only for statistical purposes. Do you, personally, identify as lesbian, gay, bisexual, or transgender?” Gallup retained the records for people who said yes to this question over one year, and the Generations and TransPop studies contacted these folks later. At this second contact, researchers checked to be sure each person was eligible for the Generations or TransPop study, and if they were, the researchers sent them a survey by mail or email. To be eligible, people had to identify as sexual minority or transgender, be within certain age ranges, and be able to complete surveys in English. 

After people were recruited into the TransPop and Generations studies in this way, they were asked multiple survey questions, including the target question about whether they identified as nonbinary. 

d) In your own words, explain why the procedure used above is likely to lead to a probability (in other words, a generalizable, or externally-valid) sample of LGBTQ adults in the United States. 

e) Reading the description above, which type of random was used here--random sampling or random assignment? 

f) To reiterate, the study concluded that about 11% of LGBTQ people in the United States identify as nonbinary. What method could we use to estimate the percentage of all people in the United States identify as nonbinary? 

g) What do you predict might happen to the estimate of nonbinary people in the U.S. if they redid this study 20 years in the future? 

 

Suggested Answers

a) The 95% CI is meant to capture the true value in the population 95% of the time. The true value in the population might be as low as 1 million or as high as 1.48 million. Importantly, our CI may have missed the true population mean. We don't know if it missed it or not, but we know the probability that our CI missed it--see question b)!

b) the 95% CI will miss the population mean in 5% of cases.  

More information: If we replicated the study, the new estimate and 95% CI will be a little bit different from the first one. But we'd still expect that the replication estimate of the number of nonbinary adults will be between 1 million and 1.48 million. We'd be surprised if our new estimate was a lot lower or a lot higher than this range. 

c) The population of interest is LGBTQ adults in the United States (who have access to a cell phone or landline)

d) The Gallup sample is generalizable because their Daily Tracking Survey goes out to a random sample U.S. adults. Any LGBTQ folks that are called in that survey are therefore also a random sample, and random samples are generalizable.  

e) They used random sampling. (As a reminder, random assignment is used in experiments, to assign participants to different groups.)  

f) One method could be to ask directly of a random sample of US adults. For example, Gallup could potentially add a nonbinary identity question to their Daily Tracking Survey for one year.

Another method was the one used in the present study. Gallup learned that 3.5% of their Daily Tracking Survey sample answered yes to the LGBTQ question, and then, of that subsample, a certain percentage (about 11%) identified as nonbinary. After adjusting weights, they got a percentage, (maybe around .05%) which they multiplied by the entire U.S. adult population, to produce the estimate that about 1,219,000 adults (or between 1 and 1.48 million) identify as nonbinary. To see some of the statistics I quoted here, visit this page)

g) Answers will vary

Claim: "Drinking any amount of alcohol causes damage to the brain"

Photo credit: Dangubic/Deposit Photos

Here's a link to CNN's coverage of a study on alcohol use and gray matter in the brain. The headline reads, "Drinking any amount of alcohol causes damage to the brain, study finds." Let's consider how well the study supports the causal claim the journalist (or the editor) attached to it.  Here's one of CNN's introductory statements: 

In an observational study, which has not yet been peer-reviewed, researchers from the University of Oxford studied the relationship between the self-reported alcohol intake of some 25,000 people in the UK, and their brain scans.

a) The term, "observational study" in this context is synonymous with "correlational study." What are the key variables in the study? What does the quote above tell you about whether the variables were manipulated or measured? 

Here's the next paragraph in the CNN summary: 

The researchers noted that drinking had an effect on the brain's gray matter -- regions in the brain that make up "important bits where information is processed," according to lead author Anya Topiwala, a senior clinical researcher at Oxford.

"The more people drank, the less the volume of their gray matter," Topiwala said via email.

b) The CNN journalist erroneously wrote, "drinking had an effect on the brain's gray matter".  Let's walk through whether this correlational study here can support the journalist's causal claim. The first criterion is covariance, which is about the study's results. Does the study show covariance?

c) The second criterion is temporal precedence. The journalist doesn't provide information about whether the drinking behavior (variable A) was measured first in time, before brain scans measured people's gray matter (variable B). What are the two potential directions this relationship might go?

d) The third criterion is internal validity. There could be some third variable, C, which is associated with both A and B. See if you can come up with a potential C variable that could reasonably correlate with both amount of drinking and reduction in gray matter. 

You might have noticed that the opening to the CNN summary mentions that the study "has not yet been peer-reviewed". 

e) What happens during the peer review process? Do you think journalists should summarize, for the general public, studies that have not yet been peer-reviewed?

f) The empirical report has not yet been published, but you can read more about the study where it is posted on the preprint site, medarxiv. What are some pros and cons of putting one's research on "arxiv" sites like these? Consider issues such as openness of science, peer review, and public interest. 

 

Thanks again, Stephen Chew, for sharing the example!

Cat-in-the-box

Pancetta the cat sitting in Kanizsa square. Photo and research by Gabriella Smith, M.A.

I've shared several examples of research on pet dogs on this blog, so it's about time I present an example of research on pet cats.

If you own a cat yourself, you've probably noticed that they enjoy climbing into 3-dimensional boxes, baskets, and other containers. You might also know that cats also seem to prefer "climbing" into 2-dimensional boxes--that is, they are likely to sit in a simple square that has been masking-taped onto the floor. A recent study took a closer look at this adorable behavior as part of a test of whether cats are susceptible to a phenomenon known as the Kanisza contour illusion.

As illustrated in the photo of Pancetta that accompanies this post, the Kanisza contour illusion consists of four Pac-Man-like shapes that, when arranged just so, are usually interpreted by humans as a square. Researcher Gabriella Smith wondered if cats, too, would fall for the illusion, so she created an experiment to test it. 

An interesting aspect of Smith's study was that the cats participated in their own homes, an approach called "citizen science." In other words, the researchers  "let volunteers test their own cats at home in a standardized way and then send in the results".

Here's a description of the research study, as covered by NPR's All Things Considered.(The original empirical study was published in a journal published  by Elsevier, which means it's very difficult to obtain, even through universities.):

Participants received booklets with instructions on how to print out and cut up paper shapes that could be taped to the floor to create three options: the Kanizsa illusion, the actual outline of a square, and a control that had the Pac-Man shapes facing outward. The volunteers were instructed to videotape their cats as they encountered these stimuli in short tests over six days, while the volunteers wore sunglasses to avoid inadvertently giving the cats any signals with their eyes.

Their results? That the cats chose to sit in the illusory square as often as they did the actual outline of a square — and more than the non-square arrangement.

Here are some questions about the study.

a) This was an experiment. What was the independent variable (IV) and what were its levels?

b) What was the dependent variable (DV)?

c) Was the IV manipulated as independent-groups or within-groups?

d) Which of the four basic experiments does this study seem to have used--Posttest only? Prettest-posttest? Repeated measures? or Concurrent measures?

e) Sketch a graph of the results that were described (hint--put the three IV groups on the x-axis).

f) In the empirical article, the researchers mention that the presentation of the different IV levels was fully counterbalanced, which means that cats in the study were exposed to multiple trials. Why was counterbalancing necessary, and what might counterbalancing have looked like? 

g) The story mentions that "the volunteers wore sunglasses to avoid inadvertently giving the cats any signals with their eyes.". Which internal validity threat was this designed to prevent?

Now here's a detail about the citizen science aspect of the study:

While hundreds of people signed up for the experiment, in the end the researchers got complete data on only 30 cats..... [Smith] notes the number of people who dropped out of this study and says future efforts along these lines may need to be even shorter and easier to ensure that volunteers are able to complete all the required tasks....This latest study, [in contrast to past work on cats] has the advantage of testing untrained cats in a.... natural setting, 

h) Given this outcome, summarize the pros and cons of the citizen science approach to animal research, perhaps adding some of your own.

i) You might be alarmed that the sample size for this study dropped from hundreds of people to only 30, and you might be tempted to think this study has an attrition threat to internal validity. But attrition isn't an issue for internal validity here--can you explain why? 

 

 

An empirical reason to say "Thanks"

Senders of gratitude letters benefit from writing them, and recipients "feel great" when receiving them. Photo credit: Rawpixel/Deposit Photos

This example comes from the subfield of positive psychology, and the study of gratitude. One technique studied by positive psychologists is the practice of writing a gratitude letter to a person who has influenced you.  According to this summary, "There are two excellent reasons for writing a gratitude letter: It will make you feel really good, and it will make the recipient feel great"

The concept is simple: Write a letter to a person whom you'd like to thank. Gratitude letters can be handwritten or typed, emailed or snail-mailed. The important thing is to send them. Gratitude letters can help both the writers and the recipients. 

First, here's the journalist's description of a study about the benefits of writing these letters:

Scientists randomly assigned the 293 participants to three groups: Those receiving psychotherapy, those receiving psychotherapy and participating in expressive writing, or those receiving psychotherapy and participating in gratitude-letter writing. Even in the small study, participants in the gratitude group reported significantly better mental health than the other two groups, even three months after the trial ended.

a) What is the independent variable (IV) in the study above?  What are its levels?

b) What is the dependent variable? 

c) Was the IV manipulated as independent groups or within-groups? 

d) What kind of experiment was this? Posttest only? Pretest-posttest? Repeated measures? or Concurrent measures? 

e) What role do you think the "expressive writing" group played in this study? Use the term "design confound" or "control variable" in your response. 

f) Can this study support the claim that "writing gratitude letters (in addition to psychotherapy) improves mental health"? Apply the three causal criteria. 

g) Ask two questions about the construct validity of the study--one about the IV and one about the DV.

h) Ask a question about the statistical validity of the study. To what populations might we be able to generalize? 

 

What about the recipients? It turns out that people aren't really sure if they should send their gratitude letters. They might feel awkward about it, perhaps surmising that the recipient may not really care about such a thing anyway. But here's a quote about research on receiving these letters: 

On the receiving end, opening a gratitude letter feels even better than you might imagine. Amit Kumar, a social scientist at the University of Texas at Austin, studies the reactions of gratitude-letter recipients.

“It’s not like it makes some people feel great and some people feel just OK and some people feel kind of weird,” he said of his research findings. “Almost everybody is saying that they feel really, really great.”

i) Most psychology studies have at least two variables (because they are testing association or causal claims). But, at least according to the journalist's description above, the study by Dr. Kumar had only one variable. What was it? 

j When Dr. Kumar said “Almost everybody is saying that they feel really, really great.”, he was saying something about the mean of this variable as well as its standard error. What can you infer about these two statistics, based on his quote? 

k) This study can support a frequency claim, because it's about the level of one variable. In your own words, write the frequency claim it can support. 

(Dr. Kumar's study on gratitude letter recipients can be previewed here. You'll see that in addition to asking recipients how they felt after receiving a gratitude letter, they also asked senders to predict how the recipients would feel. The main finding was that recipients felt much better than senders predicted they would feel.)

Make a face

Should you smile, grimace, or neither when you get your COVID jab? Photo credit: Gregory Johnston/Shutterstock

Have you gotten a COVID vaccination yet? If not,  grab a chopstick and read this article before you go. Here's a link to a journalist's summary of a study on the effect of making a face (such as a smile or a grimace) on the pain of an injection. 

The background for the research is the "facial feedback hypothesis:" If you've read Chapter 14 already, you'll recognize this line of work. 

The idea that manipulating our facial expressions can affect our emotions has a long and storied history. There are many advocates of [this] hypothesis, and many critics, too. Indeed, one of the classic findings in the field — that people find cartoons funnier if they hold a pen between their teeth, inducing a smile — recently failed to replicate. This mixed research background was well known to Sarah D. Pressman and Amanda M. Acevedo at the University of California, Irvine, who led the new work, published in Emotion.

Here is some detail about the study: 

The team randomised 231 student participants to hold either a neutral expression, a regular smile (which involves the cheek muscles), a Duchenne smile (which also recruits muscles by the eyes, resulting in creasing around the eyes) or a grimace (which recruits both these sets of muscles, plus those which allow us to wrinkle our brows.

In the study they manipulated the facial expressions unobtrusively, by asking participants to hold a chopstick in different ways:

To induce these expressions, the participants (who thought they were taking part in an investigation of multi-tasking) were given chopsticks to hold in one of four positions between their teeth, and asked to imitate an example photo. Each participant then held their assigned facial position while they received an injection of saline solution into the upper arm, designed to mimic a flu jab. Their heart rates and skin conductance were measured throughout [before, during, and after the jab], as objective markers of their stress levels, and they also reported on levels of pain (anticipated and then actual) and other emotions, such as being jittery or relaxed.

This study was an experiment. Before we get to the results, let's identify some of its features.

a) What was the independent variable (IV) here? How many levels in this IV, and what were they? 

b) Was the IV manipulated as independent groups or within groups? 

c) The journalist writes, "The team randomised 231 student participants to hold either....". Is the word "randomized" here referring to random assignment or random sampling? 

d) There are several dependent variables (DVs) reported here. Name three of them. 

e) Which type of experiment was this? posttest only? pretest-posttest? repeated measures? or concurrent measures?

Now for the results. The researchers collected multiple DVs and the patterns were not the same across all DVs. I will summarize the results for only one DV, self-rated pain:

The Duchenne smiling and grimacing groups gave lower pain ratings than the neutral group while waiting for the injection, during the injection, and immediately after it. The regular smiling group fell somewhere in between.

f) Sketch a graph of this result (placing the DV on the y-axis, as usual).

g) Now a big challenge: Apply Chapter 11's twelve internal validity threats to this example. How does the study's design help rule these threats out? I've done a few of these for you. You'll find Table 11.1 helpful here!

Design confound
Selection effect
Order effect
Maturation	This was a pretest-posttest design that took place in a single session. The single session helps rule out maturation. We can also rule out maturation because all four groups had the same opportunity to "mature."
History
Regression to the mean
Attrition
Testing	All groups were tested multiple times, which means this potential threat was held constant in all groups, ruling it out as an internal validity threat.
Instrumentation
Observer bias
Demand characteristic	The independent-groups design and cover story about multitasking can probably help disguise the true nature of the experiment.
Placebo effect

 

Replication: "Don't ditch the laptop just yet"

A mini meta-analysis found that the average effect size comparing laptop to handwritten notes was negligible. Credit: insta_photos/Shutterstock

One of the major empirical examples in Chapter 10 (in the 3rd and 4th editions of the textbook) is a study that tested whether laptop or handwritten notes are better for people's learning from a video lecture. As you may recall, the study (Mueller & Oppenheimer, 2014) found that people who took notes by hand generally remembered more--they scored better on conceptual questions (though not better on factual questions) about a TED talk they had watched. 

Last month, a replication study was published in the journal Psychological Science--the same journal that published the original Mueller and Oppenheimer study. The replication was lead by Dr. Heather Urry of Tufts University. The coauthors on the paper are her entire research methods class, who conducted the work as a class project (How cool is that? Shout out to Dr. Urry's students!). 

The work is an excellent example of a replication study. It was covered in a few media outlets . Here is one summary, as reported by the APS website:

But although previous work suggested that students who write notes by hand may retain information better than those who type them out on a laptop, attempts to replicate this research indicate that taking notes longhand may offer no measurable benefit. 

“We found only small, statistically nonsignificant differences in quiz performance as a function of note-taking medium,” whether students used a laptop or pen, wrote APS Fellow Heather L. Urry (Tufts University) and colleagues in Psychological Science. 

Here's some coverage of the same study, in a blog post by Dr. Regan Gurung, on Psychology Today:

There was no difference in how well the students performed on the factual recall questions as a function of how they took notes. This was similar to the 2014 study. The difference, and failure to replicate, came when Urry et al. analyzed performance on the applied questions. Again, there was no difference in scores based on note-taking style.

In the APS summary, it is reported:

Urry and colleagues reached this conclusion through a direct replication of a 2014 study by Pam A. Mueller (Princeton University) and APS Fellow Daniel M. Oppenheimer (University of California, Los Angeles), in addition to a mini meta-analysis of eight related studies. 

a) As a review, what were the IV (and its levels) and DVs in the original study? (hint: There are two DVs)

b) What does the term "direct replication" mean in the previous quote?

c) How does direct replication compare to a conceptual replication? (You'll find these terms in Chapter 14)

 

Now let's talk about the term "mini meta-analysis" in the quote, "...in addition to a mini meta-analysis of eight related studies."   Here's a summary from the Psychology Today blog:

Urry et al. (2021) also found 8 similar studies, that also aimed to replicate the 2014 results.  The studies compared also used similar methodologies with slight variations. The effect of taking notes longhand was not statistically significant.

d) Do you think the eight studies  in the mini meta-analysis were direct replications or conceptual replications? How can you tell? 

e) Now explain, in your own words, how they might have done the mini meta-analysis? What would the authors have done, once they found the eight studies? (Why do you think it was called "mini"?)

f) Dr. Urry's team preregistered their replication study--what does preregistration mean in this context? 

g) While the original Mueller and Oppenheimer study was covered by multiple popular press outlets (such as here, here, and here), the replication study has, so far, not really broken through to the popular press. Why do you think that is? 

You should check out forest plots of the mini meta-analysis!  Here's the link to the journal article. If you're locked out, you can access a preprint of the replication study here. 

People believe claims of sexual harassment more when women are attractive

The coded stereotypicality of the drawing was a dependent variable in one of the studies. Photo: Courtesy of Dr. Jin Goh. Citation: Goh, J. X., Bandt-Law, B., Cheek, N. N., Sinclair, S., & Kaiser, C. R. (2021). Narrow prototypes and neglected victims: Understanding perceptions of sexual harassment. Journal of Personality and Social Psychology. Advance online publication. https://doi.org/10.1037/pspi0000260

Sexual harassment (unwelcome and inappropriate sexual remarks or physical contact) harms people in schools and workplaces. One potential problem with sexual harassment is whether people who report it are taken seriously. Unfortunately, when people report sexual harassment, their claims are not always believed, which means perpetrators can go uninvestigated and victims' harassment might continue. 

The press recently covered an important series of studies that investigated the role that physical appearance plays in the reporting (and believing) process. One of the summaries appeared in the Seattle Times; another  appeared on the APA website.  The Seattle Times journalist summed up the studies like this: 

The study found people are more apt to believe sexual harassment claims by women who are young, “conventionally attractive,” and appear and act feminine. Women who don’t fit that prototype not only are less likely to be believed, but also are presumed to be unharmed by harassing behavior, the study said. 

The APA website does a better job detailing the methods:

The researchers conducted a series of 11 multimethod experiments, involving more than 4,000 total participants, designed to investigate the effect a victim’s fit to the concept of a typical woman had on participants’ view of sexual harassment and the consequences of that mental association.

In five of the experiments, participants read scenarios in which women either did or did not experience sexual harassment. Participants then assessed the extent to which these women fit with the idealized image of women, either by drawing what they thought the woman might look like or selecting from a series of photos. Across all the experiments, participants perceived the targets of sexual harassment as more stereotypical than those who did not experience harassment.

The details above describe a set of experiments (do you know why they are experiments?)

a) What was the main independent variable (IV) in the five studies above? What were its levels? 

b) What was the main dependent variable (DV) in the five studies? What do you think its levels were? 

c) The image (with the two drawings, above) that goes with this post came from the first study (Study 1), in which participants read a scenario in which " women either did or did not experience sexual harassment" and made a "drawing what they thought the woman might look like". Which drawing do you think came from each condition? 

d) The drawings from Study 1 had to be coded for stereotypical femininity. Since this variable required coding, inter-rater reliability would have been important. How do you think the researchers established inter-rater reliability for their coders? (Hint: you can find relevant information in Chapter 5.)

e) Sketch a small bar graph depicting the results of these studies, based on the journalist's description. 

Now here's some more description from the APA website:

In the next four experiments, participants [read] ambiguous sexual harassment scenarios, such as a boss inquiring about a woman’s dating life. These scenarios were paired with descriptions or photos of women who were either stereotypical or not. The participants then rated the likelihood that the incident constituted sexual harassment.

f) What was the main independent variable (IV) in these four studies? What were its levels? 

g) What was the main dependent variable (DV) in the four studies? What do its levels seem to have been? 

h) Sketch a small bar graph depicting the results based on the journalist's description. 

i) Assess the plausibility of this claim: "Being more stereotypically feminine causes people to think a woman's sexual harassment claim is true"  Apply the three causal criteria (covariance, temporal precedence, and internal validity). 

Instructors: Students are likely to enjoy reading the studies in the original journal article, which appears on my list of journal articles that are student-friendly.

Interrogating Buzzfeed: Does Instagram really have "huge effects" on our psychology?

Do the studies really support cause and effect? And are those effects "huge"? Photo: Julia_Tim/Deposit Photos

This Buzzfeed headline is eyecatching and dramatic: Instagram Has Some Pretty Huge Effects On Our Psychology. Here's Everything We Know So Far.

I see two important issues to interrogate in this headline. First, it's a causal claim ("Instagram has some...effects on our psychology"). Second, it's making an argument about effect size ("pretty huge effects"). Let's see if they pan out. 

The article focuses on body image and mental health. Here's an example:

The research into how Instagram affects our psychology is relatively new, but there is a growing body of evidence to say that it has a profound effect on the way we perceive ourselves and the world around us.

Does the research support such a causal claim? Here's one study that the journalist summarized: 

Earlier this year, the UK's Royal Society for Public Health (RSPH) and the Young Health Movement published a report that looked at the impact of different social media platforms on mental health.

The report was based on a survey of over 1,400 British 14-to-24-year-olds that asked about the positive and negative implications of each social media platform for health and wellbeing.Instagram scored well on its ability for self-expression, but it was linked with some of the worst scores for sleep loss, body image concerns, and FOMO.

It's not completely clear from this summary what questions were included on this survey. However, a survey is unlikely to have manipulated exposure to Instagram, so it can't support the claim that "Instagram has implications for how people perceive themselves and the world around them."

What about the effect size of the associations? The journalist didn't report the data from the study, so we can't evaluate whether or not the effect was really "huge" or "profound." We'd have to search out the original paper for that. 

How about this other study, summarized later in the article: 

In a study published last year, Cohen and her co-authors found that it wasn’t so much social media use itself that impacted how people perceived their bodies but rather how they were using it.

Following body- and appearance-focused accounts on Instagram — as opposed to neutral accounts dedicated to things such as travel — was associated with idealizing thin bodies, obsessive surveillance of one’s own body, and striving for thinness.

a) Four variables are listed in the study above (Following body-....) What are they?

b) Were these variables measured or manipulated, do you think?  (Check your hunches here, by reading the abstract of the Cohen study.)

c) Why can't this study support a claim such as "following body-focused accounts impacts how people perceive their bodies"? Apply the three criteria for causation. 

d) Does the journalist provide any indication of the effect size of this association? If not, what data might have helped you decide if the link was "huge"?

Let's read a bit further in the journalist's article to see if we can find some experimental evidence. How about this one:

However, Cohen is hopeful that the positive body movements emerging on the platform will have far-reaching benefits for how women reflect on their own physiques.

She has recently submitted a paper for publication demonstrating that among 195 young women exposed to “body-positive” images, mood, body satisfaction, and body appreciation were all improved significantly.

The wording in the example above suggests that there was a manipulated variable, because it reads "among 195 young women exposed to 'body-positive' images'". When I read that, I picture that some people were exposed to these images, and some were not. 

e) Imagine you were going to conduct a conceptual replication of this experiment. What was the independent variable? What would its levels be? 

f) What would the dependent variable(s) be? (Hint: The journalist mentions three).

g Sketch a little bar graph of the results of your experiment--picking one of the DVs to graph. In your view, how far apart would the means and error bars have to be for you to be convinced that the effect is "huge"?

h) Would this experiment have the potential to support a causal claim? What confounds would you need to control for? 

Suggested answers to selected questions

c) Covariance: Yes--apparently this questionnaire study found a link between following more body-focused accounts and idealizing thin bodies more. 

Temporal precedence: Nope--this was a questionnaire study that measured both variables at the same time. We can't know if the IG accounts came first, or if people who idealize thin bodies more are more likely to want to follow body-focused accounts.

Internal validity: Also nope. In this correlational study we can't rule out every possible third variable. For example, maybe people with a history of eating disorders are both more likely to follow body-focused accounts and are more likely to idealize thin bodies.

d) No correlation was reported here--too bad! It would be nice if the journalist had given us some benchmarks indicating how much more likely people were to idealize thin bodies....perhaps a percentage change or a correlation coefficient, and a 95% CI for that change or coefficient. 

 

Thanks to UD student Emily Hogan for suggesting this article for the blog!

 

 

Don't believe it?

Macadamia nuts are in the same evolutionary family as peaches. (or are they?) Credit: gresey/Deposit photos

People's tendency to believe, share, and act upon disinformation (fake news) is a serious threat to civic engagement. Psychologists are studying what makes people more or less likely to believe false claims. Here are some excerpts from a brief article published in Buzzfeed, which summarized a few studies about people's willingness to believe what they read.

"What we find in our research really robustly is that we can push around people's feelings of truth using things like photographs and audio quality, and even the ease of pronouncing someone's name," said [researcher] Newman.

Here's an example of a study on how perceptions of "truthiness" can be influenced:

In 2012 Newman co-authored another paper with studies that found photographs can inflate the sense of "truthiness" that people get from reading a statement.

The sort of photographs that the researchers used for this research were "non-probative", meaning that they didn't actually contain any information about the claims being made.

One study ...found that if participants were given a trivia statement such as "Macadamia nuts are in the same evolutionary family as peaches", they were more likely to believe the statement was true if a photograph of macadamias simply appeared alongside the sentence [compared to no photo].

Bonus: Stephen Colbert (who coined the term "truthiness") also summarizes it in this clip. 

Questions about the first example:

a) What are the two variables in the macadamia nuts study? Name each variable, its apparent levels, and indicate if it is measured or manipulated.

b) Do you think the "photo appearance" variable was manipulated as between-groups or within-groups? Explain your reasoning.

c) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

d) Ask three questions about the statistical validity of the macadamia nut study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

 Here's a second example:

A study published in March found that people were less likely to believe an academic researcher on the radio if the audio quality of the recording was poor or glitchy.

The study involved 97 participants, who were instructed to listen to recordings of different pieces of science news and assess the quality of the research and the researcher who was reporting it.

Half of the audio recordings were edited to sound as though they were recorded on a bad phone line. The listeners who heard these recordings rated the credibility of the researcher and the studies they were discussing as significantly less favourable than recordings with higher audio quality.

Questions about the second example:

e) What are the two variables in the audio quality study? Name each variable, and indicate if it is measured or manipulated.

f) Do you think the "audio quality" variable was manipulated as between-groups or within-groups? Explain your reasoning.

g) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

h) Ask three questions about the statistical validity of the audio quality study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

i) Now consider the internal validity of the audio quality study: Why would random assignment be important in this study? 

Selected answers (use these answers as models to help answer the audio quality questions, if needed)

a) One variable is the appearance of a photo; its levels are "photo or no photo." It is manipulated.
A second variable is "level of belief in the trivia statement" or "rated truthiness." Its levels probably range from "low belief" to "high belief". It is measured. 

b) For a single fact, the presence of a photo would have to be manipulated as between groups--people would have to see the fact either with or without the photo. If they were exposed to both levels of the variable (a within-groups design), people might wonder why they are being asked to rate the same fact two times.  However, in the study itself, participants were exposed to multiple facts, some of which were accompanied by a photo and some of which were not....so that makes the variable within-subjects when you consider all of the facts together. 

c) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

d) Ask three questions about the statistical validity of the macadamia nut study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

For effect size, you could ask questions such as, "how far apart were the truthiness ratings?" "How big is the difference between the photo and no-photo group?" or "What is the d? (or other effect size)"
For precision you could ask, "what is the 95% CI of the effect size estimate?"
For replication you could ask, "has this effect been replicated in other studies? What do those other studies say?"

I would like to thank my student Emily Hogan for sharing this example and Buzzfeed story with me!