Everyday Research Methods

Should scientists change out of pajamas?

Does the study's design support the causal claim the journalist attached to it?
Photo credit: R.O.M/Shutterstock

Some journalists simply cannot resist attaching a causal claim to a study where it doesn't belong. Here's an example from the trade magazine, Inside Higher Ed, which reprinted a piece from Times Higher Education. The headline reads, "Scientists urged to change out of pajamas." 

The first paragraph begins:

Academics who are tempted to remain in pajamas during the working day should think again, says an Australian study that has linked the practice to a deterioration in mental health.

 

a) There are actually two claim-like statements in the first paragraph. What are they? (Remember that advice is a form of causal claim). 

b) What seem to be the two variables in the association and causal claims above? 

The journalists summarizes the study this way: 

The study, which drew on 163 responses from staff at five medical institutes in Sydney, also delved into the reality of working at home for researchers. [...]

Some 28 percent of scientists said they wore pajamas at least once a week -- a cohort who were twice as likely to report worsened levels of mental health than those who dressed normally each day, according the study, by David Chapman and Cindy Thamrin [...]

Given the survey format of this study, we can assume that all the variables in this study were self-report. To learn more about this relatively simple study, you can visit the open-access original journal article here, and you can see the full, original text of the survey here. 

Let's work through the four big validities. 

c) Construct validity: Look through the original survey (here) and find the variables that measured pajama-wearing and mental health assessment. What do you think--how well do these variables seem to be measured? (Guess what? You're assessing face validity.) 

d) Construct validity: When you view the original text of the survey (still here)  you might be a bit surprised by the lighthearted tone of some the survey questions. For example, they ask about the "typical home working environment" they included the option, "hiding in the bathroom." When they asked people what they wear during remote meetings, one of the options was, "none of your business, camera turned off."  How might this casual tone affect the construct validity of the variables being measured? Do you think it will it affect the accuracy of self-reports?

e) External validity: The journalist refers to "scientists", so that seems to be the population of interest. The sample was described this way by the journalist:

The study, which drew on 163 responses from staff at five medical institutes in Sydney...

In the empirical article, there is more detail about the sample: 

An invitation to participate was emailed to all staff, students and affiliates of the Woolcock Institute of Medical Research, Sydney, and later extended to other medical research institutes in Sydney (Garvan Institute, Children’s Medical Research Institute, Centenary Institute, Brain and Mind Centre).

Can this study generalize from this sample to the population of interest? Why or why not? (Don't be tempted to focus on the size of the sample here. Remember that it's not the size that matters for external validity)

f) Statistical validity: Here is the effect size of the relationship between wearing pajamas and mental health: 

[people who wore pajamas at least once per week] were twice as likely to report worsened levels of mental health than those who dressed normally each day

In the empirical article, the rates of poorer mental health were described as 59% (for pajama-wearers) vs. 26% (for non-wearers). What do you think of the strength of this effect? What more information would you like that is relevant to statistical validity?

g)  Internal validity. Let's evaluate whether the study can support the causal claim, "wearing pajamas put the scientists at risk for poorer mental health?" In order to support a causal claim, we need to conduct an experiment. Was this an experiment or a correlational study?  Explain your answer. 

h) Internal validity: Now let's apply the three criteria for causation. 

The study does show covariance, because people who report working in pajamas did have twice the rate of mental health decline, compared to those who did not.

What about temporal precedence? Does the method establish which variable came first in time? 

What about internal validity? Can you think of a third variable (some "C" variable) that might be associated with both wearing pajamas and having worse mental health? 

Now make a conclusion--does the study support the advice to "change out of pajamas?"

i) How might you design a true experiment to study the potential causal effect of pajamas on mental health? What variable would you need to manipulate? What variable would you need to measure? 

 

 

 

Frequency claims: Fitter and better rested during lockdown

One of the positive changes people reported was being able to spend more time with family. Photo: Steve Skjold / Alamy Stock Photo

The first peer-reviewed studies about the COVID-19 pandemic  are starting to appear in journals, and journalists are starting to summarize them in the popular press.

Let's explore this summary of a survey study conducted in Scotland and summarized for a general audience by the scientist herself, in The Conversation. The scientist writes, 

 ... for some people lockdowns have provided an unexpected opportunity to make positive changes to their lives...

Here are some details: 

In May 2020, my colleagues and I surveyed over 3,000 people in Scotland to find out what positive changes people had made in their lives during the lockdown period. 

And some of the key results:

More than half the people we surveyed reported these changes for the better: being more appreciative of things usually taken for granted (reported by 83% of participants), having more time to do enjoyable things (by 67%), spending more time in nature or outdoors (by 65%), paying more attention to personal health (by 62%), doing more physical activity (by 54%) and spending more time with a partner or spouse (by 53%).

Each of the above results constitutes a frequency claim, because it talks about the level, or percentage, associated with one variable. Consider this one, for example: "doing more physical activity (by 54%)". The variable here is "doing more physical activity under the lockdown (or not)."

Now let's use the four big validities to systematically interrogate this frequency claim, "54% of people in this Scottish survey reported doing more physical activity during the lockdown." 

a) Construct validity: How do you think they measured the variable "doing more physical activity"? (Self-report? Observational? Physiological?) . What are your initial impressions of, or questions about, how well this variable was measured? 

b) Construct validity: Let's say you wanted to establish the criterion validity (See Ch 5) of the (presumably self-report) measure of "doing more physical activity".  What behavior should self-reports of this variable correlate with, if the self-report measure is valid? 

c) Statistical validity: The point estimate, 54%, is one element of statistical validity. What about the precision of this estimate? That is, what is the 95% CI, or margin of error of this estimate? It's not provided in the article, but you can visit this calculator to estimate the CI. Do that now. 

d) Statistical validity:  Replication is another aspect of statistical validity. The scientist-author cites at least one more, similar study that came to a similar conclusion: 

An Australian study (still in preprint, meaning its findings are yet to be reviewed by other scientists) also sought to find out similar information. In a survey of over 1,000 people, it found that 70% of participants reported having experienced at least one positive effect of the pandemic. 

e) External validity:  The sample for this survey was N =3000. Don't get distracted by the size of the sample (remember--it's not the sample size that helps establish external validity). Instead, what do you need to ask in order to decide whether the study's sample can generalize to their population of interest (Consult Ch 7)?

f) External validity: I visited the peer-reviewed article (here--open access) and found this information about their sampling technique.

The target population of the survey was adults, aged 18 years or older, currently residing in Scotland, who were interested in sharing their experience of positive change. Participants were primarily recruited through social media advertisements on Facebook and Twitter which directed participants to the online survey on Qualtrics. 

What kind of sampling did they apparently use in this study? (Name a sampling technique that is discussed in Chapter 7). 

What is their population of interest?

Does the sampling technique they used allow them to generalize from their sample to their population of interest? Why or why not? 

g) Internal validity: In a frequency claim, nobody is making a causal statement, so we don't actually need to interrogate internal validity.

h) Now that you've looked at all four validities, what do you think about the quality of this study? Make a nuanced comment about its strengths and weaknesses. 

 

P.S. Thanks to Marianne Lloyd for the reference!

Musical skill: Why does an early start seem to help?

People might start musical training early because they are living in musical families or show early evidence of musical talent. 
Photo credit: Artem Furman, Alamy stock photo

What accounts for musicality? Some people are highly skilled and accomplished musicians, while others express no aptitude. A recent correlational study tried to understand why some people are more musically accomplished. A summary of this research appears on the APS website, and the original empirical article appears in the journal Psychological Science. 

The researchers used a sample of 310 professional musicians as well as a large sample of twins (which they called "amateur musicians"). The twin sample came "from an existing research project, the Study of Twin Adults: Genes and Environment (STAGE)." The twin sample enabled the researchers to estimate genetic and environmental similarities on musicality.

The participants completed measures of musical aptitude (e.g., tests of their accuracy at pitch and rhythm) and musical achievement (e.g., having performed publicly or won prizes). Researchers also asked each person what age they had started learning music. They asked how many hours per week they'd practiced at different points in their lives. The journalist summarized one of the core results:

By comparing the results from these two independent studies, the researchers were able to show that an earlier start age is associated with musical aptitude, both in amateurs and professional musicians...

Questions for students appear in gray throughout this post:
a) Sketch two scatterplots of this result described in the green quoted text above--one scatterplot for professional musicians and one for amateurs. The x and y axes will be the same on the two scatterplots--be sure to label these carefully. Will your scatterplots show positive or negative relationships? 

 

Now, it might make sense that starting musical training at an earlier age is associated with better musical aptitude simply because of total practice time. That is, people who start training earlier have practiced more hours, and practicing more hours is usually associated with more musical skill. In other words, we might sketch this, third-variable proposal:

According to the study, was total practice time a third variable, explaining why starting age was associated with greater aptitude? Consider this full quote from the journalist's summary: 

By comparing the results from these two independent studies, the researchers were able to show that an earlier start age is associated with musical aptitude, both in amateurs and professional musicians, even after controlling for accumulated practice time.

This quote means that even though practice time is usually associated with greater aptitude, this was not the reason why starting earlier was associated with greater aptitude. The results from a regression table might have looked like this:

DV/criterion variable: Musical aptitude
IV/predictor variable:                 Beta
Age starting musical training          -.21**
Total practice time                               .03           
(The betas above are simplified versions of those reported in the original journal article, Table 3, for the musician sample)

Question
b) Write a sentence describing what the beta of -.21** means. 

Therefore, it wasn't total practice time that accounted for the benefit of starting musical training early. On to the next explanation:

[The researchers] then evaluated starting age in a manner that accounted for the genetic data from the STAGE study.  

The results indicate that genetic factors—possibly related to musical interest and talent—have a substantial influence on the age individuals start music practice and their future musical aptitude. When controlling for familial factors, namely shared genetic and environmental influences, such as a home environment that is steeped in music, there was no additional association between an earlier start age and musicality. 

So, in other words, the researchers were proposing this explanation:

And they found results that supported this explanation. The statistical calculations that are used in twin studies are actually beyond the scope of this course and book--so you'll have to trust the researchers here. Suffice it to say that when they controlled for genetic interest or talent, there was no longer an association between age at starting training and musical aptitude.

c) Consider the construct validity of two of this study's variables: Musical aptitude and age at starting training. What questions could you ask to evaluate the construct validity of these measures? 

d) Here is a challenge question: In the empirical journal article (Table 2) the researchers reported that the correlation between between total practice hours and musical aptitude in the professional sample was r = .05, while the same association in the amateurs (i.e., the twins) was r = .30. One reason the correlation was weaker in the professional musician sample might have been restriction of range. Sketch two scatterplots--one for each sample but with the same x and y axes--that illustrates how restriction of range might have limited the correlation in the professional musician sample. 

Meta-analyses: Strength training treats depression and anxiety

Strength training seems to improve both depression and anxiety, according to two meta-analyses. Photo credit: romanchazov27/Deposit Photos

Many of us start the new year resolved to improve our exercise habits. Exercise, of course, is great for our physical health, and we also know that aerobic exercise (i.e., "cardio") can improve mood. The benefits of aerobic exercise on depression and mood have been well-documented by several studies (here's an example of one meta-analysis). 

A recent article in the Washington Post describes the benefits of strength, or resistance, training for depression. The piece provides a good example of how a journalist covers meta-analysis for a popular audience.

The author himself suffers from depression, and starts his article by explaining his own workout strategy:

It’s strength training — the muscle-building, body-toning and sometimes grunt-inducing workout — that has helped transform my chronic mental illness into better mental health. In fact, once I began a regular regimen of resistance training, the positive shift in my mood, my confidence and my self-image was so pronounced that I switched careers.

Of course, while an anecdote about one person's experience can be compelling, it doesn't carry the same weight as a systematic observation. It's possible that the author was a biased observer of his own improvement. Therefore, we need experiments that randomly assign people to different exercise groups to test whether resistance training has a causal effect on depression and mood. 

It turns out that several researchers have, in fact, studied the effect of strength training, enough to collect and combine:  

[Dr. Jacob] Meyer was among the authors of a recent meta-analysis of the link between resistance training and depressive symptoms ... The study, published in the June 2018 issue of JAMA Psychiatry, analyzed 33 clinical trials for the effects of resistance exercise on depression. Results showed that resistance exercise “significantly reduced depressive symptoms” among research participants.

a) In your own words, what does a meta-analysis do?

b) In the meta-analysis described in the paragraph above, there were 33 studies, all of which were experiments. Given the context, what was the independent variable in each of these 33 studies? What was the dependent variable? 

c) Reread the description above, which states that strength training "significantly reduced depressive symptoms". This phrasing does not indicate the meta-analytic effect size (that is, the effect size they obtained by averaging the effect sizes of all 33 studies). To see that effect size, you'd have to turn to the original empirical article, here.  The article is open-access and the forest plot you will find there depicts  the mean, or overall, effect. What is the overall effect size? 

The meta-analysis above showed that strength training can help improve depression. What about a related mental health condition, anxiety? Turns out there's a separate meta-analysis on that:

Resistance exercise doesn’t affect depression alone; research shows that it can reduce symptoms of anxiety, too. For a 2017 study published in the journal Sports Medicine, Gordon and his colleagues conducted another meta-analysis of 16 studies involving 992 total participants; it concluded that resistance training “significantly improves anxiety symptoms among both healthy participants and participants with a physical or mental illness.”

d) How many studies were in this meta-analysis on strength training and anxiety? Estimate how many participants were in each study.

e) What would have been the independent variable in each of these studies? What would have been the dependent variable?

f) Once again, the journalist explains that resistance training "significantly improves" anxiety symptoms, but we don't hear about the effect size. The original empirical article  is behind a paywall. You might be able to access it from your own library. If you can't find it, I'll tell you that the mean effect size (d) of the studies was 0.31, 95% CI [0.17, 0.44]. What information do you need to evaluate this effect size? Is the effect strong or weak? 

As a bonus, the researchers are exploring why strength training might improve depression and anxiety. That is, they are exploring potential mediators of the relationship between strength training and mental health. Here is the first of two mediator explanations, as summarized by the journalist:

One notable finding was that participants showed an improvement in their mood regardless of whether they grew physically stronger from the exercises. ... “Perhaps it might be the sense of accomplishment and confidence that comes from exercising, rather than the achievement of actual strength gains, to explain why study subjects felt better,” Meyer says.

And the second:

Resistance training, like other exercise, induces the release of a protein called BDNF, or brain-derived neurotrophic factor, into the hippocampus region of the brain. Among other functions, the hippocampus is responsible for mood regulation, and in people who are depressed, it shrinks up to 25 percent of its normal volume. The release of BDNF triggers the growth of new brain cells in the hippocampus, restoring it to its full size and improving communication between cells.

“That’s the physiological theory behind why you feel so good after working out,” says Jason Sawyer, an assistant professor of wellness and exercise science at Rhode Island College in Providence.... 

g) Choose one of the mediator descriptions above and sketch a figure depicting the mediator argument. Use a box for each variable, and arrange the boxes in a triangle shape, using the figures at the end of Chapter 9 as a model. 

 

Selected answers

c) The overall (meta-analytic) effect size was d = 0.66 95% CI [0.48, 0.83]. Therefore, strength training groups scored about 2/3 of a standard deviation better than control groups, on average. To evaluate how strong this effect is, it might be helpful to compare the effect of strength training to some other treatments for depression, including cognitive-behavioral therapy or medication. (A 2013 meta-analysis found that CBT's effect on depression is at least d = 0.53, 95% CI [0.43, 0.62].)

d) There were 16 studies and a total of 992 participants, so that's about 165 participants per study. 

e) Each study would have used exercise condition as an IV and anxiety level as a DV. 

f) As you can see, the overall (meta-analytic) effect size of resistance training on anxiety was not as strong as the overall effect size of resistance training on depression (the two CIs do not overlap). This pattern suggests that strength training does more for depression than for anxiety. 

g) Here's how you might sketch one of these mediation proposals:

[strength training condition] -->  [sense of accomplishment] --> [improved mood]

Be quiet! Psychological effects of noise

In one of the studies, researchers found that students whose classrooms were on the train-track side of the school made less progress in reading than those on the quieter side.  Credit: Alon Adika/Shutterstock

The weekly Freakonomics podcast recently put together an episode about the psychological impact of loud noises in our environment. You can listen to it (or read the transcript) here. 

The host and guest review several studies with a variety of methodologies, which include some excellent examples of quasi-experiments.

The introduction to the transcript reads, 

The modern world overwhelms us with sounds we didn’t ask for, like car alarms and cell-phone “halfalogues.” What does all this noise cost us in terms of productivity, health, and basic sanity?

 

Example I.

The first example comes from animal behavior and behavioral ecologist Peter Tyack, who studied whales in the Bay of Fundy. The Bay provided lots of food for the whales, but as a site of many shipping channels, the same place always had the background noise of ships going to and fro. 

Researchers like Tyack had never really thought much about this background noise. It was just there. But then all of a sudden it wasn’t. The change came with the 9/11 terrorist attacks.

TYACK: All of a sudden, the ships that were plying the ocean in that area stopped. 

This drop in ship traffic was only temporary. But it happened to coincide with some other whale research that was happening in the Bay of Fundy:

TYACK: Researchers from the New England Aquarium had been sampling feces from whales to look at stress hormones. It wasn’t part of a noise experiment at all. 

It was just your standard whale-feces research.

TYACK: But what they found was that if you compared the stress hormones in whales before 9/11 and after 9/11, their stress hormones actually went down after 9/11. 

a) In the quasi-experiment described here, what is the quasi-independent variable? What are its levels? What is the dependent variable?

b) Let's assume that the researchers collected and tested whale feces nearly every day, several days before 9/11 and several days afterwards, as well. What type of quasi-experiment is it (nonequivalent control groups, nonequivalent control group prettest-posttest, nonequivalent control groups interrupted time series, nonequivalent control groups interrupted time series)?  

c) Sketch a graph of the results of the whale study. 

Example 2

Here's another example: This one took place in a public school in New York City. The school was built right next to an elevated subway track, so kids in one of the school were exposed to the loud noise of subway trains approximately every four minutes. 

One side of the school building faced a nearby elevated subway; the other side faced away. Bronzaft matched second-, fourth-, and sixth-grade classrooms on the quiet side and on the noisy side, where a passing train would push the sound readings from 59 decibels to 89 decibels. Then she compared the average reading scores from the two sets of classrooms.

BRONZAFT: And the children exposed to the transit noise were nearly a year behind in reading by the sixth grade, and the teacher had difficulty teaching.

d) In the quasi-experiment described here, what is the quasi-independent variable? What are its levels? What is the dependent variable?

e)  What type of quasi-experiment is it (nonequivalent control groups, nonequivalent control group prettest-posttest, nonequivalent control groups interrupted time series, nonequivalent control groups interrupted time series)?  

f) Sketch a graph of the results of the reading study. 

g) In the podcast, they make two caveats:

It’s worth noting that Bronzaft’s subway research, as with similar studies at airports and elsewhere, have some limitations. For one thing, Bronzaft couldn’t randomly assign students to the noisy versus quieter classrooms. There were also relatively few classrooms to choose from, so there might have been some natural variation.

Which of the four big validities is the first caveat about? Which of the four big validities is the second caveat about?

 

 

 

Will an "awe walk" do wonders for your well-being?

The study asked all volunteers to go on a daily walk, but randomly assigned only some of them to seek out experiences of awe.  Credit: Studio Grand Web/Shutterstock

A recent headline reads, "An 'awe walk' might do wonders for your well-being."  Here is the journalist's summary of an empirical journal article.

The journalist starts by explaining that psychological researchers already know that exercise can improve mood. And we also already know that a feeling of awe can improve mental health. The journalist introduces the concept:

A somewhat nebulous emotion, awe generally is defined as the sense that you are in the presence of something larger and more consequential than yourself and that this something is mysterious and ineffable. 

In the study,

They concentrated on people in their 60s, 70s and 80s, an age when some people can face heightened risks for declining mental health.

Read the journalist's description of the study, identifying the study's independent variable, dependent variables (there are more than one DV), and control variable(s):

The scientists asked 52 of the study volunteers if they would mind adding a weekly 15-minute walk to their normal schedules. ...Fresh, baseline studies of their mental health showed they were psychologically well-adjusted as well, with little anxiety or depression.

The scientists randomly divided these volunteers into two groups. One, as a control group, was asked to start walking, at least once a week, for 15 minutes, preferably outside, but given few other mandates.

The members of the other group likewise were asked to walk once a week, but also were instructed in how to cultivate awe as they walked.

Not surprisingly, they found that the awe walkers seemed to have become adept at discovering and amplifying awe. One volunteer reported focusing now on “the beautiful fall colors and the absence of them among the evergreen forest.” A control walker, in contrast, said she spent much of a recent walk fretting about an upcoming vacation and “all the things I had to do before we leave.”

The researchers also found small but significant differences in the groups’ sense of well-being. Over all, the awe walkers felt happier, less upset and more socially connected than the men and women in the control group. The volunteers in the control group reported some improvements in mood, but their gains were slighter.

Questions

a) What was this study's IV? What were its levels? Was the IV manipulated as independent groups or within groups? 

b) Name three distinct DVs from this study, based on the journalist's description.

c) Which type of experiment is this? Posttest only? prettest-posttest? repeated measures? concurrent measures?

d)  Pick one of the DVs you listed in (b). Sketch a bar graph of this DVs. (Hint: you are going to need to plot four data points--do you see why?).

e) Construct validity question: What's going on in the paragraph that starts, "Not surprisingly, they found that the awe walkers seemed to have become adept at discovering and amplifying awe."? Use the term "manipulation check" in your response, and explain why a manipulation check is relevant to construct validity.

f) Statistical validity question: The journalist describes "small but significant differences". What does this phrase mean? Use the term "effect size" in your answer. 

g) External validity question: Can we generalize from this sample, or is generalization unknown? defend your answer (but please don't refer to sample size!)

h) Internal validity question: What role did random assignment play in this study? What if somebody says, "some people just like going for walks"--would you consider this to be a threat to internal validity? Why or why not? 

The journalist also mentions another DV. The researchers asked people to take a couple of selfies during their walks:

Over the course of the eight weeks, the size of awe walkers’ countenances shrank in relation to the scenery around them. Their faces grew smaller, the world larger. Nothing similar occurred in the photos from the control group.

i) It sounds like the researchers did not preregister prediction about the selfie content. Is lack of preregistration a potential problem?

 

 

 

Ethics, justice, and vaccine trials

The journalist writes that Black Americans have been proportionally less willing to participate in vaccine trials, perhaps because researchers in the past have violated their trust. Photo: Andrey_Popo/Shutterstock

At time of writing (late 2020), researchers around the world are racing to test vaccines for the covid-19 virus. To test the efficacy and safety fo these vaccines, studies must recruit thousands of participants and track them over time. They need to know, Do people who get the vaccine have any negative side effects? Do people who get the real vaccine have lower rates of COVID infection compared to people who get a placebo vaccine? 

In this story, New York Times journalist Jan Hoffman explores one legacy of the Tuskegee study: It has led to pervasive distrust of research participation on behalf of African-Americans. This means that Black Americans may not be eager to participate in vaccine trials for covid-19. 

We need to include people of all ethnicities in these clinical trials. Hoffman writes:

It is essential, public health experts say, that research reflect diverse participation not only as a matter of social justice and sound practice but, when the vaccine becomes available, to help persuade Black, Latino and Native American people to actually get it. (The participation of Asian people [in vaccine trials] is close to their share of the population.)

“Historically, we test everything in white men,” said Dr. Michael, a member of the vaccine development team at Operation Warp Speed, the public-private partnership set up by the White House. “But the disease is coming after people of color, and we need to encourage them to volunteer because they have the highest burden of disease.”

Early on in vaccine trials, African-Americans were making up only 3% of the study participants (this group is 15% of the U.S. population). Therefore, recruiters like Carla Arnold were going door to door and meeting with small community groups, trying to recruit African Americans to participate in the trials. Hoffman's article includes this exchange:

...an older woman turned the question back on Carla Arnold, ... who is well-known to people in the Heights:

“Miss Carla, would you feel comfortable allowing them to inject you?”

Ms. Arnold, 62, adjusted her seat to face them down, her eyes no-nonsense above a medical mask.

“They already did,” she replied. 

[...] Ms. Arnold carefully explained to the tenants her decision to participate in the trial for the vaccine being developed by Moderna, a company that has received pledges of $1.5 billion from the Trump administration for its effort.

“I am a proud African-American woman,” she said. “As African-Americans, we always seem to get less out of things that go on. I want us at the forefront of this. I want to make sure that Black people are represented. I’m going by faith that these people won’t do to African-Americans what they did to us in Tuskegee. I’m holding them accountable.”

Questions

a) In Chapter 4 (Ethics) you read about Tuskegee Syphilis study, which violated all three ethical principles of the Belmont Report. One of these is respect for persons. How did the Tuskegee study violate this principle?

b) The second Belmont principle is beneficence. How did the Tuskegee study violate this principle?

c) The third Belmont principle is justice. How did the Tuskegee study violate this principle?

d) Which Belmont principle is most relevant to the composition of the participant pool for a vaccine trial? 

The Tuskegee study isn't the only famous historical example of research that unfairly targeted Black Americans. Hoffman reminds readers of three infamous studies here: 

[examples] include surgeries by Dr. J. Marion Sims, a 19th-century gynecologist, on enslaved Black women, the 40-year-long Tuskegee study, in which doctors deliberately allowed syphilis to progress in Black Alabama sharecroppers, and researchers’ taking of cells without permission from Henrietta Lacks, an African-American cancer patient, in 1951.

e) As Hoffman's article explains, African-Americans' reluctance to participate in vaccine trials probably reflects, in part, an understandable mistrust of researchers, based on these past examples (“'It’s not the science we distrust; it’s the scientists,' said Jamil Bey"). Reflect on this outcome of past unethical practice: Why 

In one sense, it might seem fair that in the current trial, African-Americans are not disproportionately "bearing the burden" of the vaccine trials and its potential risks (that's the justice principle). Nevertheless, by not recruiting African-Americans in the trials, this group risks being left out of its potential benefits, too. It's an issue of ethical fairness and good practice: Access to to the vaccine (even in its trial form) needs to be equal in all communities, both now and in the future. And this is especially true because Black Americans are much more likely to be affected by the virus.

Hoffman's article ends on a positive note:

By the following week, there were signs that the outreach efforts were helping. The portion of people of color in the Pittsburgh area in the vaccine registry had risen to 8 percent, up from 3 percent. Because trial leaders can choose whom they finally enroll, they have been increasing the percentage of nonwhite subjects. Moderna reported that nationwide, as of Sept. 28, 26 percent of those enrolled were Black.

Thank you to Adam Putnam of Furman University for sharing this example with me. 

State scatterplot: Mask mandates and COVID cases

Wearing masks is associated with lower rates of COVID-19 symptoms.
Photo: Syda Productions/Shutterstock

Here is an article that amplifies Chapter 8 concepts. Using U.S. states as the unit of analysis, it depicts the correlation between mask-wearing and rates of knowing a person with COVID. Please scroll to the middle of this article from the Washington Post to see the scatterplot (we don't have permissions to show it here). 

The journalist's writing makes the concepts extremely accessible to students and laypeople. Here's an example: 

For all 50 states plus D.C., this chart plots the percentage of state residents who say they wear a mask in public all or most of the time (on the horizontal axis) and the percentage who say they know someone in their community with virus symptoms (on the vertical axis).

The journalist's orienting remarks are really helpful for a person viewing a scatterplot for the first time:

Take Wyoming and South Dakota, for instance, in the upper left-hand corner of the chart. Roughly 60 to 70 percent of state residents report frequent mask use, as shown on the bottom axis, which puts them at the bottom for mask rates. They also have some of the highest levels of observed covid-19 symptoms, approaching 40 and 50 percent.

Questions for students:

a) Looking at the scatterplot, what is one of the states with the highest rates of mask-wearing?
What is a state with one of the highest rates of "knowing someone with covid-19 symptoms"?

b) Is this correlation positive or negative? Is the association strong or weak? (btw, the r is about .85)

c) Can this association, on its own, support the claim that "wearing a mask prevents covid-19?" Apply the three causal criteria.

d) Here is a quote from the article that is relevant to the causation issue. Is Dr. Reinhart's comment referring to temporal precedence or internal validity?  (and why?)

“There could be other explanations for the correlation,” [Reinhart] said. “For example, states that had worse outbreaks earlier in the pandemic both have higher mask usage now and more immunity.”

e) Let's talk about how they measured the two variables in this correlation--the construct validity. This is a special case in which construct validity overlaps with external validity. Can you explain why? 

[to measure these two variables]...the CovidCast team partnered with Facebook, which is used by 70 percent of U.S. adults and has the ability to survey tens of thousands of them every day at relatively low cost. While the resulting state-level samples aren’t perfect representations of the general population, the researchers weight the responses using Census Bureau demographic data to ensure they’re a good approximation.

e) Challenge question: How do you think this scatterplot would have looked if the data came from June (when infection rates were mostly higher in the Northeast), instead of October? 

Instructors:

The article's data comes from a Carnegie Mellon website that depicts data for several pandemic-related variables for the U.S. For example, you can see a map of the frequency of mask-wearing here. The data from both x and y axis appear to come from here. You can also search how often people in different states do Google searches for the term, "covid" and the percentages of physician appointments that are COVID-related. The site includes R code for their figures, so students might be able to download the raw data and run their own analyses, too.

 

 

 

Podcast: WOOP, WOOP!

 

Wish, Outcome, Obstacle, Plan: An evidence-based app. Photo courtesy of Dr. Oettingen/Woopmylife.

Here's another podcast-based post. NPR's Hidden Brain podcast frequently uses psychological research to illuminate a variety of psychological topics. Today's post is based on a Hidden Brain episode called You 2.0: WOOP, WOOP!

 

The episode asks,"is there a methodical way to achieve your dreams?"

The episode starts off with the suggestion that maybe positive thinking alone isn't the best way to achieve your dreams. Of course, in the United States, we are encouraged to "dream big" and shoot for the stars. But as host Shankar Vedantam interviews researcher Dr. Gabriele Oettingen, we learn that positive thinking--by itself--may not be very helpful. 

At minute 1:40,  you hear how Dr. Oettingen first started getting interested in positive thinking--it stems from a cultural difference she observed between Germany (her home country) and the U.S.  Then Dr. Oettingen lays out the theory. She explains that positive thinking feels good, but...

(2:45)...these positive fantasies--they seduce us to feel already accomplished, and they take our energy away.

Dr. Oettingen describes a series of correlational findings:

(3:00) We found that, for example, the more positively women enrolled in a weight reduction program fantasized about their success in the program, the fewer pounds they lost--three months later, one year later, two years later.

Or the more positively university graduates fantasized about an easy transition into work life, the fewer dollars they earned two years later, the fewer job offers they had gotten, and the fewer job applications they had sent out.

a) Consider the dieting study above. Sketch a well-labeled scatterplot of the correlation she describes (one axis should have "level of fantasizing" on it, and pick one of the weight loss variables for the other axis).

b) Does this study, by itself, show that fantasizing caused these women to lose less weight? (Apply the three causal criteria).

c) Now consider the work life study above. Sketch a well-labeled scatterplot of the correlation she describes (one axis should have "level of fantasizing" on it, and you should pick only one of the job-related outcomes for the other axis).

d) Does this study, by itself, show that fantasizing caused these graduates to do worse on the job market? (Apply the three causal criteria).

In the podcast, Dr. Oettingen shared two other examples: One study on surgery patients and another study on people who had a crush on another person. [Instructors, you could consider having four different groups of students graph each of the four studies.]

e) What are the advantages of doing this same correlational study in four samples (dieters, graduates on the job market, surgery patients, and people with a crush?) Use the term "external validity" in your answer. 

Now let's move on to the rest of the theory. Dr. Oettingen's theory proposes that fantasizing by itself makes people relax--"because mentally, they're already there"--they've achieved the goal. It's better, she argues, to pair your fantasies with thoughts about obstacles. Starting at minute 9:00, Oettingen introduces the term mental contrasting, in which people follow their fantasies with thoughts about the potential obstacles, asking the question: "What is it in you that stands in the way?" 

Listen from minutes 9:00 to 15:30, as Oettingen walks through the WOOP process of activating wishes, fantasizing about the outcome, anticipating obstacles, and making a plan. (WOOP stands for Wish, Outcome, Obstacle, Plan.) 

Oettingen and her team are well aware of the limitations of correlational evidence like that introduced above.  So, although it's not described in the podcast, her team has also conducted experimental studies. In her experimental work, Oettingen and her team have randomly assigned people to engage in WOOP and others to engage in the activities in a different order (perhaps Obstacle, Wish, Outcome). They find that people in the WOOP conditions are more likely to achieve their goals.

f) If you were designing a study for people in a weight loss program, how would you design an experimental study to test if WOOP thinking would lead to more successful weight loss? 

i) What would the IV be (What would its levels be)? What would the DV be?
ii) How would you assign people to the conditions? 
iii) How would you eliminate internal validity threats? 
iv) Sketch a bar graph of your predicted outcome. 

Are you interested in trying WOOP out on your own goals and wishes? The phone app is free, and you can also access it online: www.woopmylife.org.

 

 

 

Too sleepy to choose that major

These students might misattribute their early-morning fatigue to their interest in the subject. Photo credit: Gorodenkoff/Shutterstock

Podcasts have been gaining in popularity in the last few years, and several feature psychological research. I'm going to be devoting my next few posts to some podcasts. We'll start with this very short, 3-minute episode/ interview from the NPR podcast, Hidden Brain. The episode is called "How does the way you feel shape your life?" and it's about how students might be influenced by situational factors (such as how early a class is) when they select their majors. 

The podcast opens with a story:

...behavioral economist Kareem Haggag at Carnegie Mellon University... sometimes teaches early morning classes. And he's worried that when his students are tired or sleepy, they might draw the wrong conclusion about him and about the class.

The researcher himself states (around minute 1)

DR. KAREEM HAGGAG: Our hypothesis was that students who are assigned to an early morning section of the class, or to multiple back-to-back classes before a class, might mix up how tired they are in that class with how much they like the subject, thus leading them to be less likely to choose the subject as their major.

The host, Shankar Vedantam, reports:

This is a psychological phenomenon called misattribution. You know, you go to an amusement park on a sunny day and you think the park is great. You go when it's raining - you think the park is terrible. You're unconsciously confusing your feelings about the weather for the quality of the park.

The researchers used thousands of data points from students at the U.S. Military Academy at West Point to test their hypothesis. As Vedantam points out:

... students at West Point are randomly assigned classes that are scheduled at different times of the day, which means that some students are taking this introductory class first thing in the morning. Some students are taking the very same class later in the day. The researchers then measure the chance that students would later on choose that subject as their college major.

The researchers, led by Dr. Haggag, found that students were more likely to major in the topic of that introductory class if they'd taken it at a later time of the day: 

HAGGAG: We find that students who are randomly assigned to the first period, 7:30 a.m. section, are about 10% less likely to choose the corresponding major compared to a student who takes that class later in the day. 

Questions

a) What are the two main variables of interest in this study (the independent and dependent variables)? What are the levels of each variable?

b) Is this a true experiment or a quasi-experiment? If it's a true experiment, what kind is it (prettest-posttest, posttest-only, repeated measures, concurrent measures)?

If it's a quasi-experiment, which of the four types is it (nonequivalent control groups, nonequivalent control group prettest-posttest, nonequivalent control groups interrupted time series, nonequivalent control groups interrupted time series)?

c) Which is the IV and which is the DV here?

d) Sketch a small bar graph of the results they describe.

e) Can these researchers support the causal claim that "having an early morning introductory class makes you less likely to want to major in that discipline?" Apply the three rules of causation: covariance, temporal precedence, and internal validity in your response.

f) Ask a question about this study's internal, external, statistical, and construct validity.

Bonus content:

In the study above, the main IV was "time of day of the introductory class". As a bonus, here is another independent variable they studied--"number of breaks before the introductory class":

DR. KAREEM HAGGAG: We also compared two students who are sitting in the exact same classroom, but one of whom just had a free period as a break before, and the other came from one or more back-to-back classes. We find that each additional back-to-back class reduces the likelihood that that student enrolls in the major by about 12%.

g) Which is the IV and which is the DV here?

h) Sketch a small bar graph of the results they describe.

i) Can these researchers support the causal claim that "having breaks before your introductory class makes you more likely to want to major in that discipline?" Apply the three rules of causation: covariance, temporal precedence, and internal validity.