Everyday Research Methods

Kids with obesity also need acceptance

The successful intervention for obesity combined lessons about eating and exercise, family dynamics, and peer acceptance. Photo credit: Eviart/Shutterstock

Here's another example of a research summary from The Conversation, which invites researchers to produce  accessibly-written summaries of their own research. 

This one concerns a study that conducted a randomized controlled trial of interventions for kids at-risk for obesity. Here's how the researchers, Amanda Harrist and Laura Hubbs-Tait, described their research:

We conducted a randomized controlled trial in 29 Oklahoma schools. More than 500 first graders who were at-risk for obesity – meaning their BMI was above the 75th percentile – were assigned to either a control group or a group that received a combination of three interventions.

These interventions focused on family lifestyle, family dynamics and the peer group.The family lifestyle intervention focused on healthy food and physical activity. ....

The family dynamics intervention added parenting skills and healthy emotion management. 

The peer group intervention taught social acceptance in the children’s school classrooms, ... teaching children to be more accepting of one another.
(note: please read the post for full details about these three parts of the intervention)

Here are the study's results:

We measured children’s heights and weights at the beginning of first grade and then after the intervention – in first, second, third and fourth grades. Only those children with obesity who received the three interventions – family lifestyle, family dynamics and peer group – had significant decreases in BMI gains compared with the control group.

Questions

a) The authors call this a "randomized controlled trial", so that should tell you they conducted an experiment. What was this study's independent variable (IV) and what were its levels?

b) Was the IV manipulated as independent-groups or within-groups?

c) What was this study's dependent variable? 

d) Which of the four simple experimental designs does this seem to be--posttest only? Pretest-posttest? Concurrent measures? or Repeated measures? Explain your answer. 

e) Sketch a graph of the study's results, labeling your axes mindfully. 

f) In this study, the researchers randomly assigned schools to the different conditions, rather than individual children. Why do you think they might have done it this way? 

 

The researchers' summary in The Conversation focuses on two conditions (control condition and an intervention that combined three elements). But the full study actually contained more than two conditions. You can read the full study here (open access): https://www.mdpi.com/1660-4601/18/16/8744 

 

Taking Poli Sci predicts likelihood of voting

After matching students based on their pre-college voting behaviors, the researchers found that students who had taken a political science class became more likely to register to vote. Photo credit: Lakshmiprasad/Depositphotos

In the online news outlet called The Conversation, researchers write summaries of their own research studies for a general audience. 

Here's an example from political scientists Frank Fernandez and Matthew Capaldi, both of the University of Florida. They introduce their study by writing,

...improving college student voter turnout is a national issue....According to data from the National Study of Learning, Voting, and Engagement, about 1 in 4 students – including at both two- and four-year colleges – were not registered to vote in the 2016 or 2018 elections.

The researchers conducted a study of 2,000 students at community colleges. All the students had completed an extensive survey about their civic behaviors both before and during college. Students also reported the classes they had taken in college.

The researchers used a "propensity score matching procedure", which is similar in some ways to multiple regression. Specifically, they matched up students who had the same demographic characteristics (e.g., race, parental education, family income, gender). They also matched students on whether, before going to college, they had been registered to vote or voted in elections. The researchers were mainly predicting whether students had newly registered to vote (since starting college) as well as whether they had actually voted since starting college. The authors write about their results: 

After taking students’ prior civic engagement and other college experiences into account, we found that students who took at least one political science course were 9% more likely to register to vote than those who did not.

Additionally, we found that students who took at least one political science class were 8% more likely to vote.

Questions

a) Name the variables in this study--there are at least seven variables mentioned in the summary provided above.

b) Is this a correlational study or an experiment? Explain your answer.

c) The authors imply that taking a political science course seems to cause students to be more likely to register to vote and to actually vote. Let's apply the three criteria for causation to this statement. First, do the results support covariance? 

d) Second, does the method of the study clearly establish temporal precedence? Why or why not? 

e) Finally, the authors used the matching technique to control for several alternative explanations such as race, parental education, family income, gender, as well as whether, before going to college, they had been registered to vote or had voted in elections. Can you think of any additional alternative explanations that they did not control for? 

f) In their summary in The Conversation, the authors discuss some of the downsides of their methodology. For example, they write, "We relied on self-reported data, so there is no practical way to confirm that they registered to vote or turned out to vote."  This critique is focused on construct validity--specifically, criterion validity. That is, they are admitting that they don't know whether or not a self-report item about voting actually predicts the behavioral criterion of voting. How could you run a pilot study to confirm that people who say they voted in a certain election have actually voted (and vice versa--people who say they did not vote have actually not voted)?

Here's a link to their study , which was published in Educational Researcher (sorry--paywalled).

Suggested answers:

a) The variables include race, parental education, family income, gender, being registered to vote before college, voting before college, and registering to vote during college and voting in college.

b) It's a correlational study--the survey method simply measured all of the variables, and a study where all the variables are measured is correlational. 

c) Yes, the results support covariance because students who took a poli sci course were 9% more likely to have registered to vote in college (and 8% more likely to actually vote). 

d) In my opinion, temporal precedence is not established here--students reported on whether they took a poli sci course at the same time they reported on whether they had registered to vote in college. It's not clear from this whether the course came first in time, before the voting behavior.

e) The authors did control for several likely explanations, including baseline interest in voting--and that is good. Another variable they might consider could be local political climate--perhaps some community colleges are more politically active than others. Students who attend more politically active CCs could be more likely to start to vote, and also be more likely to take a poli sci course. 

f) You could recruit a sample of people and ask them if they voted in the last election. Then, you can use publicly available voting records to confirm whether or not the actually voted. You should see a strong association:  People who say they voted, should have actually voted, and people who say they did not vote, should have actually not voted. 

Does mindfulness make people selfish? It depends....

Mindfulness instructions appeared to interact with people's independence/interdependence in this set of factorial studies.  Photo: Rob Wilkinson/Alamy Stock Photo

A new set of studies on mindfulness and selfishness provides an opportunity to practice skills related to factorial designs (Chapter 12). Let's get started.

 

Here's the theoretical background of the study, as described by a BBC journalist. He explains that mindfulness can 

sharpen our concentration, extend our working memory and boost our mental flexibility. With practice, we should become less emotionally reactive – allowing us to deal with our problems more calmly. 

One ‘benefit’ that you might not expect to gain, however, is heightened egotism. Yet a recent study suggests that, in some contexts, practicing mindfulness really can exaggerate some people’s selfish tendencies. With their increased inward focus, they seem to forget about others, and are less willing to help those in need. 

The journalist then describes two studies run by psychologist Michael Poulin. The first compared the effects of a mindfulness meditation session among people from two different cultures. Notice the term "depends" in this description, which is a signal that the study was investigating an interaction. The pair of studies comes....

from Michael Poulin, an associate professor in psychology at the State University of New York at Buffalo, who wanted to investigate whether the effects of mindfulness might depend on its cultural context and the existing values of the people who are practicing it.

He was particularly interested in the ways people think about themselves – their “self-construal”. Some people take a more independent viewpoint, focused on personal characteristics. If they are asked to describe themselves, they might emphasise their intelligence or their sense of humour. People with an interdependent view, on the other hand, tend to think of themselves in terms of their relations to others. 

Read this description of the study, and identify the first independent variable (IV, or factor), the second  IV (factor), and the dependent variable:

... Poulin invited 366 college students into the lab and first gave them a questionnaire measuring their independence or interdependence. 

Half were then asked to perform a meditation focused on the sensation of breathing. The control group were given a “sham” meditation that involved sitting and letting their mind wander for 15 minutes. The exercise may have been relaxing, but it wasn’t designed to increase their mindfulness. 

Next came a test of pro-social behaviour, in which the students were told about a new project to help fund a charity for the homeless. They were then given the opportunity to stuff envelopes ...which would be sent to the university’s alumni – but they were told there was no obligation to do so, if they wished to leave early.

Now for the results:

Sure enough, Poulin found that the effects of the meditation depended on people’s existing attitudes. If they were already interdependent, then the people who took the mindfulness exercise were willing to spend much more time on the charitable task; overall, they stuffed about 17% more envelopes than the control group. If they were independent-minded, however, the exact opposite occurred – the mindfulness had made them even more self-centered, so they were less willing to help the homeless. Overall, they stuffed around 15% fewer envelopes than the control group. 

Questions about Study 1:

a) What was the study's first factor (Hint: It was an IV)? Was it independent groups or within groups?

b) What was the second factor? (hint: it was a participant variable, or PV). Was it independent groups or within groups?

c) What was the DV?

d) State the design of this study using this template: "__x__ , [independent-groups/within-groups/mixed] factorial"

e) Sketch a bar or line graph of the results, as described in the paragraph above. use Figures 12.16 and 12.17 as models.  Do you see an interaction here? 

Now here is a second study, as described by the same journalist:

To be sure the finding was robust, Poulin’s team conducted a second experiment, in which the participants were first given a short text written either in the first-person singular (I), or first-person plural (we). As they read the text, they had to click on all the pronouns – a simple task known to prime either independent or interdependent thinking. They then completed the meditation tasks and, to test their pro-sociality, were asked whether they wanted to devote time to chat online with potential donors for the homelessness charity. 

Once again, the mindfulness exercise exaggerated the effects of their self-perception, driving increased altruism among the interdependent-minded, and decreased altruism among the more independent-minded. 

Questions about Study 2:

f) What was Study 2's first factor (Hint: It was an IV)? Was this one independent groups or within groups?

g) What was the second factor? (This was also an IV). Was it independent groups or within groups?

h) What was the DV in this study?

i) State the design of this study using this template: "__x__ , [independent-groups/within-groups/mixed] factorial"

j) Sketch a bar or line graph of the results, as described in the paragraph above. use Figure 12.11 as a model. Do you see an interaction? (You should, because this study was described with the phrase, "it depends"!)

k) There's causal language in the journalist's summary (e.g., " the mindfulness exercise ...was driving increased altruism" and " the mindfulness had made them even more self-centered"  Is causal language warranted here? How come? 

Note:Poulin's study has been peer reviewed and will be appearing in the journal Psychological Science, in the future. You can read an early version of the paper here. 

 

Kids who eat more fruits, vegetables have better mental health

Which alternative explanations for the relationship between diet and mental health does this study allow us to rule out? 
Photo credit: ArturVerkhovetskiy/Deposit Photos

Here's a simple headline to work with: "Kids who eat more fruits, vegetables have better mental health". The story appears on a mega-site called studyfinds.org, which provides multiple summaries of science research. 

a) What are the variables in this headline?

b) Does the headline make a frequency, association, or causal claim (and why?)

c) Before reading on, do you think that the variables in this study were measured or manipulated? 

Now here are some details about the study:

“While the links between nutrition and physical health are well understood, until now, not much has been known about whether nutrition plays a part in children’s emotional well-being,” says says lead researcher Ailsa Welch ... “So, we set out to investigate the association between dietary choices and mental wellbeing among schoolchildren.”

Researchers studied data from more than 8,500 children at 50 schools in England. 

d) Based on this description, can you tell how the "diet" and
"well being" variables were probably operationalized--Self-report? Observational? or Physiological? (If not, what would you guess?)

Here are some details of the results:

The team looked at the association between nutritional factors and mental well-being and took into account other factors that might have an impact, such as adverse childhood experiences and home situations. “We found that eating well was associated with better mental well-being in children. Among secondary school children, in particular, there was a really strong link between eating a nutritious diet, packed with fruit and vegetables, and having better mental well-being,” explains co-author Dr. Richard Hayhoe....

e) Start with this quote: "there was a really strong link between eating a nutritious diet, packed with fruit and vegetables, and having better mental well-being,” Sketch a scatterplot of this relationship, labeling your axes mindfully. 

f) Now consider this statement: The team looked at the association between nutritional factors and mental well-being and took into account other factors that might have an impact, such as adverse childhood experiences and home situations. The phrase "took into account" should alert you that regression was used. In this regression, what was the criterion (dependent) variable? What were the predictor (independent) variables? (hint: there were at least three predictor variables).

g) Suppose someone wanted to claim, based on this study, that eating fruits and vegetables causes better mental well-being. What are some internal validity problems with that claim? Come up with one or two possible C-variables that could be responsible for this relationship, and explain how your C variable goes with both A (eating fruits and veg) and B (higher mental well-being).

h) When the researcher reported that "among secondary school children, in particular, there was a really strong link between eating a nutritious diet..and having better mental well-being," he is suggesting a moderator. The term, "in particular," suggests that the relationship between diet and well-being was stronger for secondary school children than it was for younger children. Sketch what this moderator would look like, either using two scatterplots (one for each age subgroup, kind of like Figure 8.18) or making a table, similar to Table 8.5. 

i) If you like, think about mediators. Let's say that the researchers have a theory about why eating more fruit and veg is associated with better mental well-being. What might mediate this relationship? That is, why might eating fruit/veg lead to better mental well-being? Think of a mediator and draw a small diagram of it here, like Figure 9.12. 

(Please keep in mind that a mediator relationship is a causal relationship, and that can't really be supported by this correlational study. However, that does not stop us from proposing some potential mediators that we can test in a future study.)

 

 

 

 

 

 

 

Increasing vaccination rates in evidence-based ways

One of the evidence-based techniques was to "say because," as these ads illustrate.
Images used with permission from the Government of Jersey

Jersey is a self-governing island in the English Channel off the northwest coast of France. Its population is around 108,000. After the COVID-19 vaccine became widely available, the island's government started campaigns to encourage people to get their shots.  Rather than making up communication strategies that just seemed right, they decided to design strategies based on social psychological evidence.  This article describes the studies they used and how they applied them. Let's take a look. 

Strategy: Say because

One strategy mentioned in the article involves the word "because."

The study:

Ellen Langer demonstrated the persuasive power of the word “because” in her famous studies in the 1970s. People waiting in line to use a photocopier were asked “Excuse me, I have five pages. May I use the copier?” 60 percent agreed. But when the request was accompanied with a reason — “May I use the Xerox machine because I’m in a rush?”—94 percent complied.

[...] even when the word because was followed with a completely meaningless reason — such as “May I use the Xerox machine, because I have to make copies?” just as many people said yes. The study wonderfully demonstrates the motivational influence of the word “because” and its continually reinforced association with the good rationales that typically follow it.

The application:

But just because people pay more attention to the word “because” doesn’t mean that what comes after is less important. Hence our two-step “because-then-reason” strategy. First, we prominently included the word “because” in our vaccination outreach campaigns. Then we followed the word “because” with a personalized reason meaningful to that particular messenger (and their audience).

The images in this post show examples of how the posters highlighted the word because. 

Questions about this strategy:

a)  Before basing an intervention on an empirical study, we'd better be sure that study was experimental (rather than correlational). Why?

b) What were the variables in Langer's study, and what were the levels?  Do you think that Langer's study was experimental or correlational? What are the clues? 

 

Strategy: Build ownership

The study:

A 2020 mega-study conducted by several dozen researchers led by Wharton School professor Katy Milkman demonstrated how a simple change to a text message increased uptake of flu shots. Reminder messages that included the words “your vaccine is reserved for you” boosted rates by as much as 11 percent compared to standard messages such as “protect yourself by getting a flu shot.”

The application:

Milkman’s study provided the inspiration for us to adopt a similar approach in the emails and texts we sent on behalf of the Government of Jersey to citizens whose turn it was to receive their Covid-19 vaccine. The words “your vaccine” were designed to give readers that sense of personal ownership.

c) What were the variables in Milkman's study? What were the levels of each variable?

d) Do you think Milkman's study was experimental or correlational? Explain your answer

e) Why do you think the journalist described this as a "mega-study"? Follow the link to test your hunch. 

f)  The journalist seems to mention only two levels of the independent variable in Milkman's study (the words “your vaccine is reserved for you” versus “protect yourself by getting a flu shot.”). But in the Milkman et al.'s actual study, there were many more levels. Check the link to the study to find out how many other conditions the researchers tested. 

 

Note to Instructors: Figure 1 of Milkman's study shows the effect sizes and CIs for all of the different interventions they tried. It's a really nice way to introduce these statistical concepts. 

 

Evidence-based tips for procrastination

What do the studies suggest about why people procrastinate? Photo credit: Timothy Hodgkinson/Shutterstock

The BBC magazine, Science Focus, collected a set of studies on procrastination that might help students at the beginning of a new school year.  The journalist opens with a couple of simple frequency claims. 

[If you procrastinate,] you’re not alone: an estimated 20 per cent of adults (and above 50 per cent of students) regularly procrastinate.

Then the article addresses three topics: 1. Is procrastination a sign of poor time management? 2. Is procrastination unhealthy? 3. How can we stop? Let's take each topic and apply some research methods content as we go.

1. Is procrastination a sign of poor time management?

The journalist quoted psychologist Fuschia Sirois, who explained that contrary to what many people believe, procrastinators aren't more likely to be bad at time management. Instead, she explains, it's about emotion regulation:

“At its core, procrastination is about not being able to manage your moods and emotions. Although many think impulsivity and self-control are the problems – and they do play a factor – underneath is a poor emotional response.”

As Sirois explains, every person faces stressful situations, demanding tasks that trigger brain activity that involves a brain region known amygdala. And it’s the amygdala that processes emotions and signals threats, capable of prompting a ‘fight or flight’ response linked to procrastination.

“Interestingly, people who say they are chronic procrastinators tend to have larger grey matter volume in the amygdala,” says Sirois.

“This means they will also be more sensitive to the potential negative consequences of their actions, leading to more negative emotions and procrastination.”

a) What were the variables in the amygdala study? Was this study correlational or experimental? 

b) Sketch a scatterplot (or bar graph) of the results Dr. Sirois describes here.

 

2. Is procrastination "bad for your health"?

The journalist implies causality with this headline, and with lines like these: 

...procrastination can cause a lot more problems than missed deadlines. Over decades Sirois has examined the impact of chronic procrastinating on a person’s health, her findings worrying at best – and downright terrifying at worst.

Let's see if the evidence can support these causal claims. He quotes Dr. Sirois again, who stated, 

“People who chronically procrastinate – people who make it a habit – have higher levels of stress and a greater number of acute health problems. They are more likely to have headaches or insomnia or digestive issues. And they’re more susceptible to the flu and colds.”

Even more alarming, Sirois has found that procrastination is a factor that can lead to hypertension and cardiovascular disease, with chronic procrastinators more likely to put off healthy behaviour such as exercise.

c) Is Dr. Sirois describing correlational or experimental evidence here? (Explain your answer by specifying the variables she mentions).

d) Why can't the evidence described by Dr. Sirois support the journalist's assertions about "the impact of chronic procrastinating"? Specify the temporal precedence problem and the internal validity problems here (when you discuss internal validity, specify a particular third, or "C" variable that might correlate with both A and B). 

3. How can we stop procrastinating?

In this section, the journalist describes some intervention studies targeting procrastination.

For example, one compelling Psychological Science paper described how downsizing larger metrics of time (think 48 hours instead of 2 days, or 10,950 days instead of 30 years) can make events seem more immediate, prompting people to engage in upcoming tasks.

You can follow the underlined link in the quoted text (above) to see that this paper described several experiments (for example, the abstract states, "we manipulated time metric".

e) Given the journalist's description (and the empirical study's abstract), name the independent variable in this experiment and name its levels.

f) Name the dependent variable in this experiment.

g) Sketch a bar graph of the study's result, as described by the journalist. 

h). Because this study was an experiment, it is more likely to support causation.
Think about something you have to do in the near future. How can you apply the results from the Lewis and Oyserman study in Psychological Science to that task, to prevent yourself from procrastinating on it? 

 

Selected Answers

a) One variable was "saying you are a chronic procrastinator or not" and the other was "volume of grey matter in the amygdala". Both variables must have been measured, so this was a correlational study.  

b) One axis should have "grey matter volume in amygdala" and the other should have "level of chronic procrastination"; the scatterplot should have a positive slope, but you don't have information on how strong the relationship was, so you can't really know how spread out the dots should be. 

c) This is likely correlational evidence. They are measuring (not manipulating) people's procrastination habits. They are measuring (one cannot manipulate) people's acute health problems, susceptibility to the flu, hypertension, and cardiovascular disease. 

d) We can take acute health problems as an example. Even if there is an association between chronic procrastination (A) and acute health problems (B), we still don't know if the procrastination led to the health problems (A to B), or if the health problems lead to more procrastination (B to A). In addition, there could be some outside variable such as life stressors or poverty (C) that is associated with both procrastination (A) and health problems (B). 

e) The independent variable is apparently "time metric" and the levels were hours vs. days, or perhaps days versus years. 

f) The dependent variable seems to be "how immediate events seem to be"

g) The x-axis should have hours and days, and the y-axis should indicate "how immediate events seem to be". The bar for "hours" should be higher than the bar for "days". 
This study seems to suggest that if you have homework due in two days, you might be more likely to get started on it if you say to yourself, "My homework is due in 48 hours." 

 

Frequency claims: Remembering brand logos

This project provided several frequency claims you can analyze.
Photo credit: LukaFunduk/Deposit Photos

Before you read on, take out a pencil and draw the Apple logo (no peeking at your own device!) Now try to draw the Adidas logo, too (no peeking at your feet, either). 

Which one was easier? How confident are you in the accuracy of your drawings?

After you've drawn these two logos, visit this website, where you can compare your own drawings to those of other people. You can also see the official logos for Apple and Adidas. How well did you do?  More importantly, how well did the people in their study do?

This website presents data from a study of Americans who were asked to draw a variety of brand logos. The website introduces the project:

...we asked over 150 Americans to draw 10 famous logos from memory as accurately as they could. Based on more than 1,500 drawings created over a period of 80 hours, the results reveal that, far from being stamped perfectly in our collective memory, these ubiquitous emblems largely exist as fuzzy visions in our mind's eye. 

The text on the website reports:

Overall, 16 percent of people drew near perfect logos, and 37 percent were good but not perfect. As we would expect, the more complex the logo, the less likely people are to remember it in full.

Across the 10 brands (which included not just Apple and Adidas, but also Starbucks, Foot Locker, Burger King, and Domino's), the percentage of people who sketched near-perfect logos changed. For example, the website reports that 20% drew near-perfect Apple logos, whereas 12% drew near-perfect Adidas logos.  

You can evaluate certain methodological aspects of this project. For example, here's a quote from their discussion of the Apple logo results:

Interestingly, a smaller proportion of people in our experiment put the bite on the wrong side of the apple (22 percent)

a) Wait, was this study really an experiment? why or why not?

The article made some frequency claims. For example, 

One in 5 people thinks the Foot Locker referee wears a hat (he doesn't), and nearly half of people believe the Starbucks mermaid does not wear a crown (she does).

b) Why can each of these two claims be considered frequency claims? 

c)  When people make frequency claims, external validity (also known as generalizability) is pretty crucial. Think about the external validity of this project. Do you think the study they conducted is able to support the claim that "one in 5 people thinks the Foot Locker referee wears a hat?" What information do you need to know in order to evaluate the external validity of this claim?

d) Now consider statistical validity. How precise are the estimates on this website? The study estimated that 20% of the sample drew near-perfect Apple logos. But what's the precision of that estimate? In other words, what's the 95% CI, or margin of error? You can compute it here, if you know the sample size (which, just to remind you, was around 150). 

What was the MoE you found?  What would happen to that MoE if the sample had been larger? (Caveat--MoE computations actually depend on having a random sample, which we don't have, so these estimates are not going to be perfect.)

e) Finally, consider construct validity. The study needed to decide how accurate each person's sketch was. One of their footnotes reads, "accuracy based on independent ratings by a panel of marketing professionals." How do you think they obtained these ratings--what might their procedure have been?

f) Using the large matrices of drawings, make some comments about the construct validity of their operationalization of "accuracy". 

 

Thanks to Marianne Lloyd for sending me this page!

Really? Can "hot baths protect your heart?"

The data came from Japan, where hot baths (whether at home, in local bathhouses, or in hot springs) are a well-loved practice. Photo credit: nitis.s/Shutterstock

Sigh....even the imprimatur of Harvard Medical School doesn't stop writers from tacking causal headlines onto correlational  studies. 

In this example, the journalist writes, "The study is only observational [i.e., correlational] and doesn't prove that daily tub bathing staves off heart problems".  Yet the headline, "Can hot baths protect your heart?" sounds pretty causal to me. 

Here's the journalist's summary of the method:

Researchers analyzed self-reported health and lifestyle information from more than 30,000 middle-aged people in Japan. Participants responded to a questionnaire at the start of the study and were then followed for about 20 years.

And here's the summary of the major results:

Compared with people who didn't take a tub bath more than twice a week, people who took a daily warm or hot bath had a 28% lower risk of cardiovascular disease and a 26% lower risk of stroke. 

This article provides a good opportunity to practice the four big validities. 

a) Name the three major variables in the study, as included in the journalist's summary. 

b) How do you know that the study was correlational? (remember to focus on whether the variables were measured or manipulated)

c) Ask at least one construct validity question about this study. (Hint--start your question with "How well...")

d) Now ask an external validity question about this study. (Reminder--your question should address sampling method, not sample size)

When you consider external validity, it might be relevant to consider that Japan has a well-developed bath culture. Most homes have soaking tubs, and some towns still support a neighborhood sento, or public bathhouse. In addition, many tourist areas and resorts feature beautifully landscaped hot spring baths called onsen. In this study, 2/3 of the participants reported that they bathe in a tub almost daily. What might this bathing culture suggest about the external validity of this study? Would this relationship generalize to a less-bathy culture like the U.S.?  

e) What about statistical validity? The authors report two estimates: One estimate concerns the link between bathing and cardiovascular disease (28%), and another concerns the link between bathing and risk of stroke (26%). The journalist didn't provide 95% CIs--but what does the sample size tell you about the 95% CI's of these estimates? (i.e., will they be less precise or more precise?)

f) Now let's walk through why this study can't support a causal claim. This study does have covariance (baths were associated with lower risk of cardiovascular disease and stroke).

What about temporal precedence--what is this, and why does this study achieve it?

And what about internal validity? Come up with a third variable ("C") that is reasonably associated both with taking more baths ("A"), and with having lower rates of disease ("B"). 

Bonus: The empirical article is open-access and you can read it here. Instructors--it's likely that students would understand this article well enough to read it and critique it.

 

Evidence in favor of U.S. stimulus checks

Which type of quasi-experimental design does this study seem to fit best? The dark line on top represents respondents with children. The red line represents all respondents. The lighter blue line at the bottom represents respondents without children. Credit: Courtesy Patrick Cooney and H. Luke Shaefer. From Poverty Solutions, May 2021, University of Michigan,

Since the COVID-19 pandemic began, several governments, including the U.S. federal government, have distributed cash payments to citizens in an attempt to alleviate the  financial hardship caused by the global shutdown. 

As the New York Times reports here, policymakers wonder if such payments are useful. Now we have data to help address that research question. 

In offering most Americans two more rounds of stimulus checks in the past six months, totaling $2,000 a person, the federal government effectively conducted a huge experiment in safety net policy. Supporters said a quick, broad outpouring of cash would ease the economic hardships caused by the coronavirus pandemic. Skeptics called the policy wasteful and expensive.

[...]

A new analysis of Census Bureau surveys argues that the two latest rounds of aid significantly improved Americans’ ability to buy food and pay household bills and reduced anxiety and depression, with the largest benefits going to the poorest households and those with children. The analysis offers the fullest look at hardship reduction under the stimulus aid.

The full report is available here, and provides some excellent graphed examples of quasi-experimental data. We'll focus on Figure 1 from the report, reprinted above with permission from the authors.

The y-axis on the figure represents the percentage of Americans who "sometimes or often had not enough food to eat in the last seven days." The question is asked by U.S. Census' Household Pulse survey about once per month to a random sample of Americans. The dates of the two U.S. stimulus checks are indicated in the laddered, light-blue vertical lines. Payments occurred in December, 2020 and March, 2021.

Questions

a) The journalist writes that "the federal government effectively conducted a huge experiment in safety net policy".  What is the independent variable in this "experiment?"  What is the dependent variable (i.e., the one depicted in Figure 1)?

b) The independent variable here is not a true IV--it is a quasi-independent variable. What makes this variable quasi-independent?

c) The authors claim that the results from Figure 1 are consistent with the argument that the stimulus checks helped people. The authors wrote in their report: "Based on the speed with which we see hardship fall, we suspect much of this drop was the result of EIP checks, which the federal government was able to quickly deliver to bank accounts for most U.S. households following passage of both bills." Locate the areas on the figure that the authors are referring to. 

d) This quasi-experimental study is probably best described as an interrupted time-series design. What is the "interruption" here? What is the time series?

e) A history threat is one potential internal validity problem in an interrupted time-series design. Explain what a history threat is and also why a history threat might be a problem here. What do you think--to what extent can we be sure it was the stimulus checks, and not some other pair of events, that is responsible for the increased food security depicted here? 

You can see results for other variables in the report here, including figures that tracked people's feelings of anxiety and their feelings of depression. 

 

 

 

 

 

Estimating how many U.S. adults identify as nonbinary

 

Photo credit: nito103/Deposit Photos

Press outlets have been covering a report by the Williams Institute (based at UCLA), a research organization collecting data on demographic information on sexual orientation and gender identity. The quotes in this post come from this summary from PBS.

The new study has estimated the number of LGBTQ adults in the U.S. who identify as nonbinary. (Nonbinary is a term that is often used to describe people whose gender identities are more than the man/woman category; some nonbinary people use they/them pronouns.)  It's important to estimate how many U.S. adults identify as nonbinary (as well as other sexual- and gender-minority identities) because anti-discrimination laws potentially affect them. PBS reports that:

Nonbinary people are part of a “growing subgroup” of the LGBTQ population, Wilson said, adding there’s been greater visibility for nonbinary people. More and more celebrities, like singers Demi Lovato and Sam Smith, and actors Indya Moore and Amandla Stenberg, have talked about identifying themselves as nonbinary or in related terms.

Here's a portion of the PBS summary: 

According to the study, which focused on people ages 18 to 60, nonbinary people make up about 11 percent of the adult LGBTQ population in the country. The study builds on the work of two prior surveys — one on trans adults, [TransPop] the other on cisgender LGBTQ adults [Generations]— to provide more comprehensive data on nonbinary people, a historically underresearched subgroup.

When the researchers extrapolated out form this subsample, they concluded that

There are about 1.2 million LGBTQ adults in the U.S. who are nonbinary.

The data in this report constitute a frequency claim, because they are estimating the level of one variable (that is, nonbinary identity).

First let's talk about the precision of these estimates. The report from the Williams Institute (here) includes 95% CIs for the estimates. For example, the two estimates from the PBS story were presented this way:

Number of nonbinary LGBTQ adults in the U.S., including transgender and cisgender people N [95% CI]	1,219,000 [1,000,000 - 1,481,000]
% [95% CI] of LGBTQ adult population	11.1 [9.3, 13.2]

a) In your own words, describe what the first CI means. Use this definition and template to help you: The 95% CI is meant to capture the true value in the population 95% of the time. The true value in the population might be as low as ____ or as high as ____. 

b) Sometimes the 95% CI misses the true value in the population. What percentage of the time will the CI miss?

Next lets' talk about external validity (generalizability) and sampling. As you consider these estimates keep in mind that there is a distinction between estimating the number of all U.S. adults who identify as non-binary and the number of U.S. LGBTQ adults who identify as non-binary. These values may be different. 

c) Given what you've read so far, what seems to be the population of interest of this study? 

To know if the sample is generalizable, you need to know how they obtained the sample. The PBS summary doesn't say anything about the sampling method used in this study, and neither does the Williams Institute report. To locate information about the sampling, I had to go to the methods sections of the two studies on which the nonbinary study is based: The TransPop study (which studied a sample of transgender U.S. adults) and the Generations study (which studied a sample of LGBTQ U.S. adults). 

In these methods sections, they say that these two studies (TransPop and Generations) worked with the Gallup organization to recruit nationally representative samples of transgender and LGBTQ adults. Gallup performs a Daily Tracking Survey in which they call a national probability sample of 1000 adults, almost every day of the year. These adults are sampled using random-digit dialing, including both landline and cell phone numbers. Gallup asks a variety of questions about well-being, political attitudes, and economic situations. At the end of each  telephone survey, the interviewer asks, “I have one final question we are asking only for statistical purposes. Do you, personally, identify as lesbian, gay, bisexual, or transgender?” Gallup retained the records for people who said yes to this question over one year, and the Generations and TransPop studies contacted these folks later. At this second contact, researchers checked to be sure each person was eligible for the Generations or TransPop study, and if they were, the researchers sent them a survey by mail or email. To be eligible, people had to identify as sexual minority or transgender, be within certain age ranges, and be able to complete surveys in English. 

After people were recruited into the TransPop and Generations studies in this way, they were asked multiple survey questions, including the target question about whether they identified as nonbinary. 

d) In your own words, explain why the procedure used above is likely to lead to a probability (in other words, a generalizable, or externally-valid) sample of LGBTQ adults in the United States. 

e) Reading the description above, which type of random was used here--random sampling or random assignment? 

f) To reiterate, the study concluded that about 11% of LGBTQ people in the United States identify as nonbinary. What method could we use to estimate the percentage of all people in the United States identify as nonbinary? 

g) What do you predict might happen to the estimate of nonbinary people in the U.S. if they redid this study 20 years in the future? 

 

Suggested Answers

a) The 95% CI is meant to capture the true value in the population 95% of the time. The true value in the population might be as low as 1 million or as high as 1.48 million. Importantly, our CI may have missed the true population mean. We don't know if it missed it or not, but we know the probability that our CI missed it--see question b)!

b) the 95% CI will miss the population mean in 5% of cases.  

More information: If we replicated the study, the new estimate and 95% CI will be a little bit different from the first one. But we'd still expect that the replication estimate of the number of nonbinary adults will be between 1 million and 1.48 million. We'd be surprised if our new estimate was a lot lower or a lot higher than this range. 

c) The population of interest is LGBTQ adults in the United States (who have access to a cell phone or landline)

d) The Gallup sample is generalizable because their Daily Tracking Survey goes out to a random sample U.S. adults. Any LGBTQ folks that are called in that survey are therefore also a random sample, and random samples are generalizable.  

e) They used random sampling. (As a reminder, random assignment is used in experiments, to assign participants to different groups.)  

f) One method could be to ask directly of a random sample of US adults. For example, Gallup could potentially add a nonbinary identity question to their Daily Tracking Survey for one year.

Another method was the one used in the present study. Gallup learned that 3.5% of their Daily Tracking Survey sample answered yes to the LGBTQ question, and then, of that subsample, a certain percentage (about 11%) identified as nonbinary. After adjusting weights, they got a percentage, (maybe around .05%) which they multiplied by the entire U.S. adult population, to produce the estimate that about 1,219,000 adults (or between 1 and 1.48 million) identify as nonbinary. To see some of the statistics I quoted here, visit this page)

g) Answers will vary