Everyday Research Methods

Frequency claims: Fitter and better rested during lockdown

One of the positive changes people reported was being able to spend more time with family. Photo: Steve Skjold / Alamy Stock Photo

The first peer-reviewed studies about the COVID-19 pandemic  are starting to appear in journals, and journalists are starting to summarize them in the popular press.

Let's explore this summary of a survey study conducted in Scotland and summarized for a general audience by the scientist herself, in The Conversation. The scientist writes, 

 ... for some people lockdowns have provided an unexpected opportunity to make positive changes to their lives...

Here are some details: 

In May 2020, my colleagues and I surveyed over 3,000 people in Scotland to find out what positive changes people had made in their lives during the lockdown period. 

And some of the key results:

More than half the people we surveyed reported these changes for the better: being more appreciative of things usually taken for granted (reported by 83% of participants), having more time to do enjoyable things (by 67%), spending more time in nature or outdoors (by 65%), paying more attention to personal health (by 62%), doing more physical activity (by 54%) and spending more time with a partner or spouse (by 53%).

Each of the above results constitutes a frequency claim, because it talks about the level, or percentage, associated with one variable. Consider this one, for example: "doing more physical activity (by 54%)". The variable here is "doing more physical activity under the lockdown (or not)."

Now let's use the four big validities to systematically interrogate this frequency claim, "54% of people in this Scottish survey reported doing more physical activity during the lockdown." 

a) Construct validity: How do you think they measured the variable "doing more physical activity"? (Self-report? Observational? Physiological?) . What are your initial impressions of, or questions about, how well this variable was measured? 

b) Construct validity: Let's say you wanted to establish the criterion validity (See Ch 5) of the (presumably self-report) measure of "doing more physical activity".  What behavior should self-reports of this variable correlate with, if the self-report measure is valid? 

c) Statistical validity: The point estimate, 54%, is one element of statistical validity. What about the precision of this estimate? That is, what is the 95% CI, or margin of error of this estimate? It's not provided in the article, but you can visit this calculator to estimate the CI. Do that now. 

d) Statistical validity:  Replication is another aspect of statistical validity. The scientist-author cites at least one more, similar study that came to a similar conclusion: 

An Australian study (still in preprint, meaning its findings are yet to be reviewed by other scientists) also sought to find out similar information. In a survey of over 1,000 people, it found that 70% of participants reported having experienced at least one positive effect of the pandemic. 

e) External validity:  The sample for this survey was N =3000. Don't get distracted by the size of the sample (remember--it's not the sample size that helps establish external validity). Instead, what do you need to ask in order to decide whether the study's sample can generalize to their population of interest (Consult Ch 7)?

f) External validity: I visited the peer-reviewed article (here--open access) and found this information about their sampling technique.

The target population of the survey was adults, aged 18 years or older, currently residing in Scotland, who were interested in sharing their experience of positive change. Participants were primarily recruited through social media advertisements on Facebook and Twitter which directed participants to the online survey on Qualtrics. 

What kind of sampling did they apparently use in this study? (Name a sampling technique that is discussed in Chapter 7). 

What is their population of interest?

Does the sampling technique they used allow them to generalize from their sample to their population of interest? Why or why not? 

g) Internal validity: In a frequency claim, nobody is making a causal statement, so we don't actually need to interrogate internal validity.

h) Now that you've looked at all four validities, what do you think about the quality of this study? Make a nuanced comment about its strengths and weaknesses. 

 

P.S. Thanks to Marianne Lloyd for the reference!

Tracking happiness on Twitter

A recent study used Twitter text to track people's moods throughout the day.
Here's how the study was covered by wired.com.

The journalist explained that:

Using Linguistic Inquiry and Word Count, a text analysis program that quantifies the emotional content of statements, Golder and co-author Michael Macy analyzed a total of 509 million tweets generated over two years by 2.4 million people in 84 countries.

The resulting trends — positive moods starting high in the morning and declining through the day, peaking overall on weekends — held steady around the world.

a) One thing that is novel about this research is the unique operationalization of their main variable, happiness. What is their operationalization, and why do the scientists believe it is a good measure of happiness?

b) Here is another quote from the Wired article:

Many studies of how moods — or, more technically, positive and negative affect — change from minute to minute and day to day rely on self-reported surveys, which can be inconsistent if not misleading. The subjects of these studies also tend to be undergraduate students from western colleges, a group that’s not always representative of humanity at large.

Twitter users, of course, don’t represent humanity either. But the culture- and globe-spanning size of the software platform’s community, and their constant generation of data that can be cross-referenced and correlated and otherwise computationally investigated, make them alluring to researchers.

When he writes, "Twitter users, of course, don't represent humanity either," which validity is the author addressing? What evaluation does the author make of that validity here?

c) The original research appeared the journal Science, a widely respected journal that your university library probably subscribes to. Use your library's tools to find the original article. What aspects of the original article did the journalist leave out?

d) This same study was featured by a number of websites and newspapers. Use an Internet search to find out how other journalists covered the same scientific study. Which news story do you think describes the original study most accurately, completely, and clearly?

e) The study's construct and external validities both seem strong. What does that mean to you--what conclusion can you make now?

Poll on the Tea Partiers: August 2011

Recently, two political scientists wrote an essay in the New York Times explaining some polling data they had collected on the Tea Party political group in the United States.  

Here's one of the claims in their article: 

"Tea Party supporters have slipped to 20 percent, while their opponents have more than doubled, to 40 percent."

a.  What kind of claim are the authors making here?  What questions should you ask to interrogate it?

Here's another set of statements from their article:  

"Beginning in 2006 we interviewed a representative sample of 3,000 Americans as part of our continuing research into national political attitudes, and we returned to interview many of the same people again this summer. ..

b. What does the phrase "representative sample" mean here?  Is a sample of 3,000 Americans large enough to make a sample representative?  

 

And finally, here's one more set of statements to consider: 

"We can also account for multiple influences simultaneously — isolating the impact of one factor while holding others constant....

"...Actually, the Tea Party’s supporters today were highly partisan Republicans long before the Tea Party was born, and were more likely than others to have contacted government officials. In fact, past Republican affiliation is the single strongest predictor of Tea Party support today."

c. In the first part of the quote above, what does it mean when the authors state that they can isolate one factor while holding others constant?  What kind of analysis does this suggest that the scientists did?

d. In the second part of the quote above, notice that they use the phrase "single strongest predictor."  Using what you learned in Chapter 8, can you imagine what the regression table might have looked like?   (List what the dependent variable was, and what some of the independent variables were, given your reading of the article, and suggest how the results might have supported the claim above.)

 

Suggested answers:

a.  As a frequency claim (about the level of support for the Tea Party among Americans) you should focus on external validity and construct validity in your questioning.  

b. A representative sample is one that was drawn using a random sampling technique.  Because the sample is representative, a sample of 3000 is large enough--it's how the sample is drawn, not how many are in it, that matters for external validity.

c.  This means that the authors probably used a multiple regression technique (see Chapter 8).

d.  We'd have to get more detail on the statistics to be sure.  However, according to this essay, the dependent variable is support for the Tea Party.  The independent variables seem to have included past Republican affiliation, support for religion in the government, suffering economic hardship in the past four years, being white, support for immigrants, and support for abortion, among others.  The statement in question probably means that the beta for Republican affiliation was the strongest one  (Later statements in this essay--those in the 10th paragraph--seem to indicate that support for religion in the government is the second strongest beta in the regression table.)

How much time do caregivers spend?

The Gallup organization published the results of a recent poll on caregivers.

The poll used a sample of people who are caring for another person in their family.  It reported that most of them (7 out of 10) are taking care of a parent or another elderly person.  The report indicated that caregivers spend on average 13 days per month running errands for the person and 6 days per month helping the person eat, dress, bathe, or do other personal care. 

Questions about the story:

1.  How did Gallup find and sample these caregivers?  Do their methods mean that the results are generalizable to most American caregivers?  

2.  How might you assess the construct validity of this poll?  

3.  How confident are you in concluding that about 36% of people who are caregivers in the United States are living with the person they are caring for?  

4. What's the margin of error on the 36% value?  (The one that refers to the proportion of people who are living with the person they are caring for).  How do you know?  

After reading the polling methods, can you say with confidence that most caregivers, like this New York woman, are caring for aging parents?

Photo: Nicole Bengiveno/The New York Times/Redux

 

Suggested answers

1.  The Survey Methods section indicates that Gallup re-contacted people they had polled the year before and who happened to have self-identified as "caregivers."  Since almost all of Gallup's polls are conducted on probability samples, this means that the caregivers themselves were presumably selected from a random sample of Americans originally.  They were then recontacted to participate in the present poll.  This means that the sample used is probably very close to a probability sample of American caregivers.  (Of course, some of the caregivers may not have been willing to participate the second time, or may not have been reached the second time, which could bias the sample in some way.)  All in all, this sample seems reasonably generalizable. 

2.  It is difficult to assess the construct validity of this poll because they do not include the question wording they used.  Therefore, it's hard to evaluate how well they measured their variables.

3.  In my opinion, this is probably a good estimate.  First, the sample seems fairly representative (see #1, above).  Second, it seems that the construct of living with the person they are caring for is a rather simple, behavioral question to measure.  Its construct validity is likely to be good.  (But of course, it would be best to see the actual question used.)

4.  Looking in the Survey Methods section, you can read, "For results based on the total sample of national adults, one can say with 95% confidence that the maximum margin of sampling error is ±2 percentage points. For smaller groups, such as full-time employed caregivers, the maximum margin of sampling error is ±2.3 percentage points."  

This means that the standard error of the estimate is 2%.  So if you conducted the same poll using the same methods 100 times, 95 of the times the value (are you living with the person you are caring for?) will be between 34 and 38%.  

Incidentally, you can see that the standard error of the smaller groups (such as full time employed caregivers) is larger.  That's because there are fewer people in this sample.  The smaller the sample, the larger the standard error of the estimate. (See Table 6.1 in the text.)

Online poll on disaster preparedness.

Soon after the great earthquake and tsunami in Northeast Japan, an article appeared on the CNN website about disaster preparedness in the United States:

Is “duck and cover” still the best we can do?

 This article reports an online poll conducted by CNN on people’s confidence in U.S. disaster preparedness.   It reports that 80% of the respondents did not have confidence in U.S. disaster preparedness. 

Questions:

a.  Read the first three paragraphs carefully.   What do you think of the sampling techniques used in this poll?  To what population can we generalize the results from this poll? 

b.  Your friend comments, “wow!  375,000 people responded to this survey!  We can definitely generalize this to the American population.”    What do you say? 

 

Schoolchildren practicing a "duck and cover" drill, designed to protect them from a nuclear attack.  (Photo: Bettmann/Corbis)

 

Suggested answers:

 a.  Notice that the poll was presented on the CNN website, which means that its sample was almost certainly self-selected.  (Another clue? A truly representative survey would never have called a random sample of 375,000 people—2,000 randomly selected people would have been plenty--see Table 6.1.) The upshot is that we have no idea to whom we can generalize the results, because we don’t know the characteristics of the type of people who choose to respond to this online poll.  They are cnn.com readers (who may be different from non-readers), and they also may have chosen to respond to the poll because they have a strong negative or positive opinion about the issue.

 b. Your friend is wrong; the number of participants in a sample has virtually nothing to do with the sample’s ability to generalize to a population.  What matters is not how many, but how.  If the sample is a probability sample, we can generalize to its population.  If not, we cannot.