Everyday Research Methods

Music festivals "make people more generous, socially connected"

Credit: halfpoint/Deposit Photos

Summer is the time for music festivals like Burning Man, Burning Nest, Firefly, and Bonnaroo. If you are planning to attend one this year, this post might interest you.

The science journalism clearinghouse called Study Finds recently reported on a study conducted at Yale University. The journalists' report included the causal headline that Music festivals like Burning Man make people more generous, socially connected to others". 

a) What are the variables in this causal headline, and what makes it a causal claim?

b) In general, researchers need to conduct an experiment in order to support a causal claim. Do you see any barriers to conducting an experiment on the causal effects of music festivals?

It will be interesting to read about whether the researchers' study can feasibly support the causal claim that the journalist attached to it. Here's how Study Finds described the study's method: 

The Yale researchers studied the behavior and experience of over 1,200 people attending festivals such as Burning Man, Burning Nest, Lightning in a Bottle, Dirty Bird, and Latitude. All of the festivals took place in the United States or the United Kingdom and emphasized art, music, and self-expression. The team set up booths at all the events, which they called “Play Games for Science.”

People ... self-reported their experiences at the event and whether they were willing to share their resources with friends and strangers. For example, week-long festivals such as Burning Man do not sell food, and people who find themselves short must rely on the generosity of others.

[...] Overall, 63.2 percent of people said they had a transformative experience and felt themselves changing. ... One of the takeaways from the transformative experience was feeling more connected to other humans, which made them more likely to share food and resources with other attendees.

c) So far, do the method and results of this study (as reported by the journalist) support the conclusion that attending the festivals caused a transformative experience? Why or why not? (Hint: Think about the need for a comparison group.)

d) Based on what you've read, was the variable of "transformative experience" operationalized as a physiological, observational, or self-report measure?

In this case, the journalist's coverage of the study doesn't do the original study justice, as I discovered when I read the original empirical article, which appeared in the journal Nature Communication (you can access it here).There are several important omissions in the journalist's report. One is that the researchers themselves did not use causal language in their work. Instead, their article is titled "Prosocial correlates of transformative experiences at secular multi-day mass gatherings" (my emphasis).

The Study Finds journalist also omitted key methods and variables. For example, one important variable in the original study was that the researchers graphed people's self-reports of having a transformative experience on each day of a 7-day festival (see Figure 2,  here). Using this method, the researchers found that the later the day (going from Day 1 to Day 7), the more likely respondents were to report having a transformative experience. 

e) Why is the above method and result important, in terms of supporting the claim that the music festivals can increase people's tendency to have a transformative experience? (Hint: think about temporal precedence.)

Another important detail omitted by the Study Finds story, is that in addition to asking people to self-report their transformative experiences, researchers also asked participants to imagine playing a "dictator game". In this game, participants are given 10 valuable tickets. Participants can then donate some number of their tickets to randomly selected stranger. The dictator game is an observational measure of generosity to strangers. The researchers found that festival participants chose to donate between 50 and 75% of their tickets to the stranger. This was much higher than the level of donations typically found, as reported in a meta-analysis of all previous studies using the dictator game (which found an average of 26%). 

f)  This dictator game result is a type of quasi-experiment--we might call it a non-equivalent control group, posttest-only design. What is the quasi-independent variable--that is, what are the two levels of the non-equivalent control group (and why are they "non-equivalent")? What is the dependent variable? 

g) The result suggests that attending the music festival might cause people to be more generous in the dictator game, but there might be a selection effect. Why? 

 

How much weekly exercise can prevent depression?

The weekly recommendation for exercise is 150 minutes.
Photo credit: blasbike/Deposit Photos

A journalist wrote about a recent meta-analysis in JAMA Psychiatry which evaluated the connection between weekly time spent exercising and levels of depression. The headline of the Washington Post summary read, "Exercising even half the recommended amount can help prevent depression". Let's delve into this research, as covered by the journalist in their brief report. You can access the original empirical work here. 

Here's the opening message:

Already known to help ease depression, regular exercise may also help prevent it, with people who exercised just half the recommended weekly amount lowering their risk for depression by 18 percent.... However, those who were more active, meeting at least the minimum recommended physical activity level, reduced their risk for depression by 25 percent, compared with inactive people.

The empirical research was a meta analysis, and it seems that all of the studies involved longitudinal designs: 

The findings stem from the analysis of data from 15 studies, involving 191,130 adults who were tracked for at least three years.

a) What are the steps you'd typically follow if you were conducting a meta-analysis? 

According to the journalist, the recommended amount of weekly exercise is 150 minutes. Therefore, it seems that working out only 75 minute per week (by walking, for example) does predict lower rates of depression over time. 

Let's apply the four big validities to this study. 

b) Construct validity: We'd need to address the construct validity of each major variable--exercise and depression. It's likely you can measure exercise in multiple ways (self-report, observational, physiological). Can you tell from the journalist's report how the studies measured exercise? 

What about depression--can you tell from the report how the studies measured depression?

I would assume that across the 15 studies, there were a variety of ways that depression and exercise were measured. Is this variability of operationalizations a strength of the meta-analysis, or a weakness? 

c) Statistical validity. The journalist merely reports a single estimate:  "people who exercised just half the recommended weekly amount lowering their risk for depression by 18 percent." But the empirical article gives us a 95% CI for this estimate: [13%, 23%].  In addition the journalist writes, "meeting at least the minimum recommended physical activity level, reduced their risk for depression by 25 percent, compared with inactive people." Again the empirical article gives the 95% CI as [18%, 32%%]. 

In your own words, explain what these two 95% CIs mean.

d) External validity. As we consider external validity, you might be tempted to focus on the total number of participants in the meta-analysis (more than 191,000 over 15 studies). Why would it be incorrect to use sample size to make decisions about external validity? 

e) Internal validity. All 15 studies in this analysis were prospective, or longitudinal. What does that mean? Use the term "temporal precedence" in your response.

Now to focus on internal validity specifically: Can you think a possible third variable that might be associated with both exercise and depression over time? (It would be potentially important for the researchers to control for this variable if they hoped to support a causal claim.)

 

 

 

 

 

 

The Pandemic and Baby IQ

Is it realistic that the pandemic reduced infant IQ by more than 20 points?
Photo credit: Cat Box/Shutterstock

Parents of young children might be alarmed if they read coverage of a recent study on baby IQ. According to one journalist who covered it, the study found that babies tested  IQ dropped over 20 points:

In the decade preceding the pandemic, the mean IQ score on standardised tests for children aged between three months and three years of age hovered around 100, but for children born during the pandemic that number tumbled to 78, according to the analysis, which is yet to be peer-reviewed.

When interviewed by the journalist, the authors of the research suggested it was the stress of the pandemic, preventing parents from having quality interactions:

The biggest reason behind the falling scores is likely the lack of stimulation and interaction at home, said Deoni. “Parents are stressed and frazzled … that interaction the child would normally get has decreased substantially.”

Any parent of small children is likely to be shocked by this finding. Twenty-two points is a huge drop in IQ.

Fortunately, Brown University economist Emily Oster provides a much-needed reality check on these results. Dr. Oster summarizes scientific studies on parenting and child development and summarizes key findings for her subscription blog. Oster summarized this empirical research for her readers as follows:

With these scores, the authors do three things. 

• First, they compare the overall scores for children between three months and three years of age evaluated between 2011 and 2021. They show huge reductions (27 to 37 IQ points) in the 2021 scores relative to earlier years.  
  
  
• Second, they analyze 39 children who were born shortly before the pandemic and were evaluated both before and during the pandemic. They actually do not find any changes in IQ for this group.
• Finally, they compare children born in the year before the pandemic with those born during the pandemic year. This sample is about 400 children in total. The reductions in IQ (Model 5, Table 5) at the lowest level of maternal education range from a 49-point reduction to an 82-point reduction. An 82-point reduction in IQ is … unfathomably large. 

Again, taken at face value, these results are worrisome. However — and I cannot stress this enough — they are completely implausible. 

Dr. Oster's position is related to statistical validity.  In general, when asking about a study's statistical validity, we inquire about effect size ("How strong?"), precision of the estimate, or 95% CI ("How wrong?") and replication ("What else is known?").  

Oster's argument that the results are "are completely implausible" is, in part, addressing effect size. It does not seem reasonable that children tested during the pandemic would have such a large decrease in IQ. One thing that can help us evaluate effect size is to compare our study's effect size to that of other known effect sizes. For example, in your textbook, Table 8.4 presents the size of several well-known relationships, such as difference in weight between males and females, or the correlation between people's height and their weight. In that table, it is suggested that certain results, such as a correlation of r = .40 in psychology might be so large as to be "too good to be true". Oster is using similar reasoning in her post when she writes that the results "are completely implausible."

Oster compares the result of the pandemic IQ study to one of the most well-known effect sizes in developmental risk:

Extremely low birth weight is among the most significant reducers of IQ, and even that is a fraction of the size of these effects. 

This statement helps put the pandemic IQ result in context, and marks it as implausible. I think her perspective is valuable and helpful, and is likely to reassure parents. 

So if the results are implausible, why did they occur?  Oster proposes an explanation for the result. Rather than blame pandemic-stressed parents (as the researchers do), she suggests that the IQ testing in the study occurred while the adult testers were wearing masks. When a new adult is administering a verbal test to young children, it may have inadvertently added a high level of difficulty to the test for these 0 to 3 year olds.

Questions

a) The Guardian summarized the IQ study even though it had not yet been peer-reviewed. Is that a wise choice for a journalist? 

b) Oster focused on the effect size of this study, but there are two other statistical validity questions we usually also ask. What question could you ask to get at the precision of this IQ estimate? 

c) What question could you ask to get at replication of this estimate? And, why is replication important for any study? 

d) Oster suggests that masked adult testers might be responsible for this unusual result. With this suggestion, which of the four big validities is she critiquing?

e) You could design a study to test whether having a masked adult tester does cause lower IQ estimates in infants they are evaluating. Propose the independent and dependent variable in such a study, and indicate what the results would be if masks do lead to a distortion of IQ scores. 

 

 

 

 

Kids who eat more fruits, vegetables have better mental health

Which alternative explanations for the relationship between diet and mental health does this study allow us to rule out? 
Photo credit: ArturVerkhovetskiy/Deposit Photos

Here's a simple headline to work with: "Kids who eat more fruits, vegetables have better mental health". The story appears on a mega-site called studyfinds.org, which provides multiple summaries of science research. 

a) What are the variables in this headline?

b) Does the headline make a frequency, association, or causal claim (and why?)

c) Before reading on, do you think that the variables in this study were measured or manipulated? 

Now here are some details about the study:

“While the links between nutrition and physical health are well understood, until now, not much has been known about whether nutrition plays a part in children’s emotional well-being,” says says lead researcher Ailsa Welch ... “So, we set out to investigate the association between dietary choices and mental wellbeing among schoolchildren.”

Researchers studied data from more than 8,500 children at 50 schools in England. 

d) Based on this description, can you tell how the "diet" and
"well being" variables were probably operationalized--Self-report? Observational? or Physiological? (If not, what would you guess?)

Here are some details of the results:

The team looked at the association between nutritional factors and mental well-being and took into account other factors that might have an impact, such as adverse childhood experiences and home situations. “We found that eating well was associated with better mental well-being in children. Among secondary school children, in particular, there was a really strong link between eating a nutritious diet, packed with fruit and vegetables, and having better mental well-being,” explains co-author Dr. Richard Hayhoe....

e) Start with this quote: "there was a really strong link between eating a nutritious diet, packed with fruit and vegetables, and having better mental well-being,” Sketch a scatterplot of this relationship, labeling your axes mindfully. 

f) Now consider this statement: The team looked at the association between nutritional factors and mental well-being and took into account other factors that might have an impact, such as adverse childhood experiences and home situations. The phrase "took into account" should alert you that regression was used. In this regression, what was the criterion (dependent) variable? What were the predictor (independent) variables? (hint: there were at least three predictor variables).

g) Suppose someone wanted to claim, based on this study, that eating fruits and vegetables causes better mental well-being. What are some internal validity problems with that claim? Come up with one or two possible C-variables that could be responsible for this relationship, and explain how your C variable goes with both A (eating fruits and veg) and B (higher mental well-being).

h) When the researcher reported that "among secondary school children, in particular, there was a really strong link between eating a nutritious diet..and having better mental well-being," he is suggesting a moderator. The term, "in particular," suggests that the relationship between diet and well-being was stronger for secondary school children than it was for younger children. Sketch what this moderator would look like, either using two scatterplots (one for each age subgroup, kind of like Figure 8.18) or making a table, similar to Table 8.5. 

i) If you like, think about mediators. Let's say that the researchers have a theory about why eating more fruit and veg is associated with better mental well-being. What might mediate this relationship? That is, why might eating fruit/veg lead to better mental well-being? Think of a mediator and draw a small diagram of it here, like Figure 9.12. 

(Please keep in mind that a mediator relationship is a causal relationship, and that can't really be supported by this correlational study. However, that does not stop us from proposing some potential mediators that we can test in a future study.)

 

 

 

 

 

 

 

Evidence-based tips for procrastination

What do the studies suggest about why people procrastinate? Photo credit: Timothy Hodgkinson/Shutterstock

The BBC magazine, Science Focus, collected a set of studies on procrastination that might help students at the beginning of a new school year.  The journalist opens with a couple of simple frequency claims. 

[If you procrastinate,] you’re not alone: an estimated 20 per cent of adults (and above 50 per cent of students) regularly procrastinate.

Then the article addresses three topics: 1. Is procrastination a sign of poor time management? 2. Is procrastination unhealthy? 3. How can we stop? Let's take each topic and apply some research methods content as we go.

1. Is procrastination a sign of poor time management?

The journalist quoted psychologist Fuschia Sirois, who explained that contrary to what many people believe, procrastinators aren't more likely to be bad at time management. Instead, she explains, it's about emotion regulation:

“At its core, procrastination is about not being able to manage your moods and emotions. Although many think impulsivity and self-control are the problems – and they do play a factor – underneath is a poor emotional response.”

As Sirois explains, every person faces stressful situations, demanding tasks that trigger brain activity that involves a brain region known amygdala. And it’s the amygdala that processes emotions and signals threats, capable of prompting a ‘fight or flight’ response linked to procrastination.

“Interestingly, people who say they are chronic procrastinators tend to have larger grey matter volume in the amygdala,” says Sirois.

“This means they will also be more sensitive to the potential negative consequences of their actions, leading to more negative emotions and procrastination.”

a) What were the variables in the amygdala study? Was this study correlational or experimental? 

b) Sketch a scatterplot (or bar graph) of the results Dr. Sirois describes here.

 

2. Is procrastination "bad for your health"?

The journalist implies causality with this headline, and with lines like these: 

...procrastination can cause a lot more problems than missed deadlines. Over decades Sirois has examined the impact of chronic procrastinating on a person’s health, her findings worrying at best – and downright terrifying at worst.

Let's see if the evidence can support these causal claims. He quotes Dr. Sirois again, who stated, 

“People who chronically procrastinate – people who make it a habit – have higher levels of stress and a greater number of acute health problems. They are more likely to have headaches or insomnia or digestive issues. And they’re more susceptible to the flu and colds.”

Even more alarming, Sirois has found that procrastination is a factor that can lead to hypertension and cardiovascular disease, with chronic procrastinators more likely to put off healthy behaviour such as exercise.

c) Is Dr. Sirois describing correlational or experimental evidence here? (Explain your answer by specifying the variables she mentions).

d) Why can't the evidence described by Dr. Sirois support the journalist's assertions about "the impact of chronic procrastinating"? Specify the temporal precedence problem and the internal validity problems here (when you discuss internal validity, specify a particular third, or "C" variable that might correlate with both A and B). 

3. How can we stop procrastinating?

In this section, the journalist describes some intervention studies targeting procrastination.

For example, one compelling Psychological Science paper described how downsizing larger metrics of time (think 48 hours instead of 2 days, or 10,950 days instead of 30 years) can make events seem more immediate, prompting people to engage in upcoming tasks.

You can follow the underlined link in the quoted text (above) to see that this paper described several experiments (for example, the abstract states, "we manipulated time metric".

e) Given the journalist's description (and the empirical study's abstract), name the independent variable in this experiment and name its levels.

f) Name the dependent variable in this experiment.

g) Sketch a bar graph of the study's result, as described by the journalist. 

h). Because this study was an experiment, it is more likely to support causation.
Think about something you have to do in the near future. How can you apply the results from the Lewis and Oyserman study in Psychological Science to that task, to prevent yourself from procrastinating on it? 

 

Selected Answers

a) One variable was "saying you are a chronic procrastinator or not" and the other was "volume of grey matter in the amygdala". Both variables must have been measured, so this was a correlational study.  

b) One axis should have "grey matter volume in amygdala" and the other should have "level of chronic procrastination"; the scatterplot should have a positive slope, but you don't have information on how strong the relationship was, so you can't really know how spread out the dots should be. 

c) This is likely correlational evidence. They are measuring (not manipulating) people's procrastination habits. They are measuring (one cannot manipulate) people's acute health problems, susceptibility to the flu, hypertension, and cardiovascular disease. 

d) We can take acute health problems as an example. Even if there is an association between chronic procrastination (A) and acute health problems (B), we still don't know if the procrastination led to the health problems (A to B), or if the health problems lead to more procrastination (B to A). In addition, there could be some outside variable such as life stressors or poverty (C) that is associated with both procrastination (A) and health problems (B). 

e) The independent variable is apparently "time metric" and the levels were hours vs. days, or perhaps days versus years. 

f) The dependent variable seems to be "how immediate events seem to be"

g) The x-axis should have hours and days, and the y-axis should indicate "how immediate events seem to be". The bar for "hours" should be higher than the bar for "days". 
This study seems to suggest that if you have homework due in two days, you might be more likely to get started on it if you say to yourself, "My homework is due in 48 hours." 

 

Frequency claims: Remembering brand logos

This project provided several frequency claims you can analyze.
Photo credit: LukaFunduk/Deposit Photos

Before you read on, take out a pencil and draw the Apple logo (no peeking at your own device!) Now try to draw the Adidas logo, too (no peeking at your feet, either). 

Which one was easier? How confident are you in the accuracy of your drawings?

After you've drawn these two logos, visit this website, where you can compare your own drawings to those of other people. You can also see the official logos for Apple and Adidas. How well did you do?  More importantly, how well did the people in their study do?

This website presents data from a study of Americans who were asked to draw a variety of brand logos. The website introduces the project:

...we asked over 150 Americans to draw 10 famous logos from memory as accurately as they could. Based on more than 1,500 drawings created over a period of 80 hours, the results reveal that, far from being stamped perfectly in our collective memory, these ubiquitous emblems largely exist as fuzzy visions in our mind's eye. 

The text on the website reports:

Overall, 16 percent of people drew near perfect logos, and 37 percent were good but not perfect. As we would expect, the more complex the logo, the less likely people are to remember it in full.

Across the 10 brands (which included not just Apple and Adidas, but also Starbucks, Foot Locker, Burger King, and Domino's), the percentage of people who sketched near-perfect logos changed. For example, the website reports that 20% drew near-perfect Apple logos, whereas 12% drew near-perfect Adidas logos.  

You can evaluate certain methodological aspects of this project. For example, here's a quote from their discussion of the Apple logo results:

Interestingly, a smaller proportion of people in our experiment put the bite on the wrong side of the apple (22 percent)

a) Wait, was this study really an experiment? why or why not?

The article made some frequency claims. For example, 

One in 5 people thinks the Foot Locker referee wears a hat (he doesn't), and nearly half of people believe the Starbucks mermaid does not wear a crown (she does).

b) Why can each of these two claims be considered frequency claims? 

c)  When people make frequency claims, external validity (also known as generalizability) is pretty crucial. Think about the external validity of this project. Do you think the study they conducted is able to support the claim that "one in 5 people thinks the Foot Locker referee wears a hat?" What information do you need to know in order to evaluate the external validity of this claim?

d) Now consider statistical validity. How precise are the estimates on this website? The study estimated that 20% of the sample drew near-perfect Apple logos. But what's the precision of that estimate? In other words, what's the 95% CI, or margin of error? You can compute it here, if you know the sample size (which, just to remind you, was around 150). 

What was the MoE you found?  What would happen to that MoE if the sample had been larger? (Caveat--MoE computations actually depend on having a random sample, which we don't have, so these estimates are not going to be perfect.)

e) Finally, consider construct validity. The study needed to decide how accurate each person's sketch was. One of their footnotes reads, "accuracy based on independent ratings by a panel of marketing professionals." How do you think they obtained these ratings--what might their procedure have been?

f) Using the large matrices of drawings, make some comments about the construct validity of their operationalization of "accuracy". 

 

Thanks to Marianne Lloyd for sending me this page!

Really? Can "hot baths protect your heart?"

The data came from Japan, where hot baths (whether at home, in local bathhouses, or in hot springs) are a well-loved practice. Photo credit: nitis.s/Shutterstock

Sigh....even the imprimatur of Harvard Medical School doesn't stop writers from tacking causal headlines onto correlational  studies. 

In this example, the journalist writes, "The study is only observational [i.e., correlational] and doesn't prove that daily tub bathing staves off heart problems".  Yet the headline, "Can hot baths protect your heart?" sounds pretty causal to me. 

Here's the journalist's summary of the method:

Researchers analyzed self-reported health and lifestyle information from more than 30,000 middle-aged people in Japan. Participants responded to a questionnaire at the start of the study and were then followed for about 20 years.

And here's the summary of the major results:

Compared with people who didn't take a tub bath more than twice a week, people who took a daily warm or hot bath had a 28% lower risk of cardiovascular disease and a 26% lower risk of stroke. 

This article provides a good opportunity to practice the four big validities. 

a) Name the three major variables in the study, as included in the journalist's summary. 

b) How do you know that the study was correlational? (remember to focus on whether the variables were measured or manipulated)

c) Ask at least one construct validity question about this study. (Hint--start your question with "How well...")

d) Now ask an external validity question about this study. (Reminder--your question should address sampling method, not sample size)

When you consider external validity, it might be relevant to consider that Japan has a well-developed bath culture. Most homes have soaking tubs, and some towns still support a neighborhood sento, or public bathhouse. In addition, many tourist areas and resorts feature beautifully landscaped hot spring baths called onsen. In this study, 2/3 of the participants reported that they bathe in a tub almost daily. What might this bathing culture suggest about the external validity of this study? Would this relationship generalize to a less-bathy culture like the U.S.?  

e) What about statistical validity? The authors report two estimates: One estimate concerns the link between bathing and cardiovascular disease (28%), and another concerns the link between bathing and risk of stroke (26%). The journalist didn't provide 95% CIs--but what does the sample size tell you about the 95% CI's of these estimates? (i.e., will they be less precise or more precise?)

f) Now let's walk through why this study can't support a causal claim. This study does have covariance (baths were associated with lower risk of cardiovascular disease and stroke).

What about temporal precedence--what is this, and why does this study achieve it?

And what about internal validity? Come up with a third variable ("C") that is reasonably associated both with taking more baths ("A"), and with having lower rates of disease ("B"). 

Bonus: The empirical article is open-access and you can read it here. Instructors--it's likely that students would understand this article well enough to read it and critique it.

 

People believe claims of sexual harassment more when women are attractive

The coded stereotypicality of the drawing was a dependent variable in one of the studies. Photo: Courtesy of Dr. Jin Goh. Citation: Goh, J. X., Bandt-Law, B., Cheek, N. N., Sinclair, S., & Kaiser, C. R. (2021). Narrow prototypes and neglected victims: Understanding perceptions of sexual harassment. Journal of Personality and Social Psychology. Advance online publication. https://doi.org/10.1037/pspi0000260

Sexual harassment (unwelcome and inappropriate sexual remarks or physical contact) harms people in schools and workplaces. One potential problem with sexual harassment is whether people who report it are taken seriously. Unfortunately, when people report sexual harassment, their claims are not always believed, which means perpetrators can go uninvestigated and victims' harassment might continue. 

The press recently covered an important series of studies that investigated the role that physical appearance plays in the reporting (and believing) process. One of the summaries appeared in the Seattle Times; another  appeared on the APA website.  The Seattle Times journalist summed up the studies like this: 

The study found people are more apt to believe sexual harassment claims by women who are young, “conventionally attractive,” and appear and act feminine. Women who don’t fit that prototype not only are less likely to be believed, but also are presumed to be unharmed by harassing behavior, the study said. 

The APA website does a better job detailing the methods:

The researchers conducted a series of 11 multimethod experiments, involving more than 4,000 total participants, designed to investigate the effect a victim’s fit to the concept of a typical woman had on participants’ view of sexual harassment and the consequences of that mental association.

In five of the experiments, participants read scenarios in which women either did or did not experience sexual harassment. Participants then assessed the extent to which these women fit with the idealized image of women, either by drawing what they thought the woman might look like or selecting from a series of photos. Across all the experiments, participants perceived the targets of sexual harassment as more stereotypical than those who did not experience harassment.

The details above describe a set of experiments (do you know why they are experiments?)

a) What was the main independent variable (IV) in the five studies above? What were its levels? 

b) What was the main dependent variable (DV) in the five studies? What do you think its levels were? 

c) The image (with the two drawings, above) that goes with this post came from the first study (Study 1), in which participants read a scenario in which " women either did or did not experience sexual harassment" and made a "drawing what they thought the woman might look like". Which drawing do you think came from each condition? 

d) The drawings from Study 1 had to be coded for stereotypical femininity. Since this variable required coding, inter-rater reliability would have been important. How do you think the researchers established inter-rater reliability for their coders? (Hint: you can find relevant information in Chapter 5.)

e) Sketch a small bar graph depicting the results of these studies, based on the journalist's description. 

Now here's some more description from the APA website:

In the next four experiments, participants [read] ambiguous sexual harassment scenarios, such as a boss inquiring about a woman’s dating life. These scenarios were paired with descriptions or photos of women who were either stereotypical or not. The participants then rated the likelihood that the incident constituted sexual harassment.

f) What was the main independent variable (IV) in these four studies? What were its levels? 

g) What was the main dependent variable (DV) in the four studies? What do its levels seem to have been? 

h) Sketch a small bar graph depicting the results based on the journalist's description. 

i) Assess the plausibility of this claim: "Being more stereotypically feminine causes people to think a woman's sexual harassment claim is true"  Apply the three causal criteria (covariance, temporal precedence, and internal validity). 

Instructors: Students are likely to enjoy reading the studies in the original journal article, which appears on my list of journal articles that are student-friendly.

Should scientists change out of pajamas?

Does the study's design support the causal claim the journalist attached to it?
Photo credit: R.O.M/Shutterstock

Some journalists simply cannot resist attaching a causal claim to a study where it doesn't belong. Here's an example from the trade magazine, Inside Higher Ed, which reprinted a piece from Times Higher Education. The headline reads, "Scientists urged to change out of pajamas." 

The first paragraph begins:

Academics who are tempted to remain in pajamas during the working day should think again, says an Australian study that has linked the practice to a deterioration in mental health.

 

a) There are actually two claim-like statements in the first paragraph. What are they? (Remember that advice is a form of causal claim). 

b) What seem to be the two variables in the association and causal claims above? 

The journalists summarizes the study this way: 

The study, which drew on 163 responses from staff at five medical institutes in Sydney, also delved into the reality of working at home for researchers. [...]

Some 28 percent of scientists said they wore pajamas at least once a week -- a cohort who were twice as likely to report worsened levels of mental health than those who dressed normally each day, according the study, by David Chapman and Cindy Thamrin [...]

Given the survey format of this study, we can assume that all the variables in this study were self-report. To learn more about this relatively simple study, you can visit the open-access original journal article here, and you can see the full, original text of the survey here. 

Let's work through the four big validities. 

c) Construct validity: Look through the original survey (here) and find the variables that measured pajama-wearing and mental health assessment. What do you think--how well do these variables seem to be measured? (Guess what? You're assessing face validity.) 

d) Construct validity: When you view the original text of the survey (still here)  you might be a bit surprised by the lighthearted tone of some the survey questions. For example, they ask about the "typical home working environment" they included the option, "hiding in the bathroom." When they asked people what they wear during remote meetings, one of the options was, "none of your business, camera turned off."  How might this casual tone affect the construct validity of the variables being measured? Do you think it will it affect the accuracy of self-reports?

e) External validity: The journalist refers to "scientists", so that seems to be the population of interest. The sample was described this way by the journalist:

The study, which drew on 163 responses from staff at five medical institutes in Sydney...

In the empirical article, there is more detail about the sample: 

An invitation to participate was emailed to all staff, students and affiliates of the Woolcock Institute of Medical Research, Sydney, and later extended to other medical research institutes in Sydney (Garvan Institute, Children’s Medical Research Institute, Centenary Institute, Brain and Mind Centre).

Can this study generalize from this sample to the population of interest? Why or why not? (Don't be tempted to focus on the size of the sample here. Remember that it's not the size that matters for external validity)

f) Statistical validity: Here is the effect size of the relationship between wearing pajamas and mental health: 

[people who wore pajamas at least once per week] were twice as likely to report worsened levels of mental health than those who dressed normally each day

In the empirical article, the rates of poorer mental health were described as 59% (for pajama-wearers) vs. 26% (for non-wearers). What do you think of the strength of this effect? What more information would you like that is relevant to statistical validity?

g)  Internal validity. Let's evaluate whether the study can support the causal claim, "wearing pajamas put the scientists at risk for poorer mental health?" In order to support a causal claim, we need to conduct an experiment. Was this an experiment or a correlational study?  Explain your answer. 

h) Internal validity: Now let's apply the three criteria for causation. 

The study does show covariance, because people who report working in pajamas did have twice the rate of mental health decline, compared to those who did not.

What about temporal precedence? Does the method establish which variable came first in time? 

What about internal validity? Can you think of a third variable (some "C" variable) that might be associated with both wearing pajamas and having worse mental health? 

Now make a conclusion--does the study support the advice to "change out of pajamas?"

i) How might you design a true experiment to study the potential causal effect of pajamas on mental health? What variable would you need to manipulate? What variable would you need to measure? 

 

 

 

Podcast: WOOP, WOOP!

 

Wish, Outcome, Obstacle, Plan: An evidence-based app. Photo courtesy of Dr. Oettingen/Woopmylife.

Here's another podcast-based post. NPR's Hidden Brain podcast frequently uses psychological research to illuminate a variety of psychological topics. Today's post is based on a Hidden Brain episode called You 2.0: WOOP, WOOP!

 

The episode asks,"is there a methodical way to achieve your dreams?"

The episode starts off with the suggestion that maybe positive thinking alone isn't the best way to achieve your dreams. Of course, in the United States, we are encouraged to "dream big" and shoot for the stars. But as host Shankar Vedantam interviews researcher Dr. Gabriele Oettingen, we learn that positive thinking--by itself--may not be very helpful. 

At minute 1:40,  you hear how Dr. Oettingen first started getting interested in positive thinking--it stems from a cultural difference she observed between Germany (her home country) and the U.S.  Then Dr. Oettingen lays out the theory. She explains that positive thinking feels good, but...

(2:45)...these positive fantasies--they seduce us to feel already accomplished, and they take our energy away.

Dr. Oettingen describes a series of correlational findings:

(3:00) We found that, for example, the more positively women enrolled in a weight reduction program fantasized about their success in the program, the fewer pounds they lost--three months later, one year later, two years later.

Or the more positively university graduates fantasized about an easy transition into work life, the fewer dollars they earned two years later, the fewer job offers they had gotten, and the fewer job applications they had sent out.

a) Consider the dieting study above. Sketch a well-labeled scatterplot of the correlation she describes (one axis should have "level of fantasizing" on it, and pick one of the weight loss variables for the other axis).

b) Does this study, by itself, show that fantasizing caused these women to lose less weight? (Apply the three causal criteria).

c) Now consider the work life study above. Sketch a well-labeled scatterplot of the correlation she describes (one axis should have "level of fantasizing" on it, and you should pick only one of the job-related outcomes for the other axis).

d) Does this study, by itself, show that fantasizing caused these graduates to do worse on the job market? (Apply the three causal criteria).

In the podcast, Dr. Oettingen shared two other examples: One study on surgery patients and another study on people who had a crush on another person. [Instructors, you could consider having four different groups of students graph each of the four studies.]

e) What are the advantages of doing this same correlational study in four samples (dieters, graduates on the job market, surgery patients, and people with a crush?) Use the term "external validity" in your answer. 

Now let's move on to the rest of the theory. Dr. Oettingen's theory proposes that fantasizing by itself makes people relax--"because mentally, they're already there"--they've achieved the goal. It's better, she argues, to pair your fantasies with thoughts about obstacles. Starting at minute 9:00, Oettingen introduces the term mental contrasting, in which people follow their fantasies with thoughts about the potential obstacles, asking the question: "What is it in you that stands in the way?" 

Listen from minutes 9:00 to 15:30, as Oettingen walks through the WOOP process of activating wishes, fantasizing about the outcome, anticipating obstacles, and making a plan. (WOOP stands for Wish, Outcome, Obstacle, Plan.) 

Oettingen and her team are well aware of the limitations of correlational evidence like that introduced above.  So, although it's not described in the podcast, her team has also conducted experimental studies. In her experimental work, Oettingen and her team have randomly assigned people to engage in WOOP and others to engage in the activities in a different order (perhaps Obstacle, Wish, Outcome). They find that people in the WOOP conditions are more likely to achieve their goals.

f) If you were designing a study for people in a weight loss program, how would you design an experimental study to test if WOOP thinking would lead to more successful weight loss? 

i) What would the IV be (What would its levels be)? What would the DV be?
ii) How would you assign people to the conditions? 
iii) How would you eliminate internal validity threats? 
iv) Sketch a bar graph of your predicted outcome. 

Are you interested in trying WOOP out on your own goals and wishes? The phone app is free, and you can also access it online: www.woopmylife.org.