Chapter 07; Sampling Feed

Frequency claims: Fitter and better rested during lockdown

Week 229_Alamy_HWKJ71
One of the positive changes people reported was being able to spend more time with family. Photo: Steve Skjold / Alamy Stock Photo

The first peer-reviewed studies about the COVID-19 pandemic  are starting to appear in journals, and journalists are starting to summarize them in the popular press.

Let's explore this summary of a survey study conducted in Scotland and summarized for a general audience by the scientist herself, in The Conversation. The scientist writes, 

 ... for some people lockdowns have provided an unexpected opportunity to make positive changes to their lives...

Here are some details: 

In May 2020, my colleagues and I surveyed over 3,000 people in Scotland to find out what positive changes people had made in their lives during the lockdown period. 

And some of the key results:

More than half the people we surveyed reported these changes for the better: being more appreciative of things usually taken for granted (reported by 83% of participants), having more time to do enjoyable things (by 67%), spending more time in nature or outdoors (by 65%), paying more attention to personal health (by 62%), doing more physical activity (by 54%) and spending more time with a partner or spouse (by 53%).

Each of the above results constitutes a frequency claim, because it talks about the level, or percentage, associated with one variable. Consider this one, for example: "doing more physical activity (by 54%)". The variable here is "doing more physical activity under the lockdown (or not)."

Now let's use the four big validities to systematically interrogate this frequency claim, "54% of people in this Scottish survey reported doing more physical activity during the lockdown." 

a) Construct validity: How do you think they measured the variable "doing more physical activity"? (Self-report? Observational? Physiological?) . What are your initial impressions of, or questions about, how well this variable was measured? 

b) Construct validity: Let's say you wanted to establish the criterion validity (See Ch 5) of the (presumably self-report) measure of "doing more physical activity".  What behavior should self-reports of this variable correlate with, if the self-report measure is valid? 

c) Statistical validity: The point estimate, 54%, is one element of statistical validity. What about the precision of this estimate? That is, what is the 95% CI, or margin of error of this estimate? It's not provided in the article, but you can visit this calculator to estimate the CI. Do that now. 

d) Statistical validity:  Replication is another aspect of statistical validity. The scientist-author cites at least one more, similar study that came to a similar conclusion: 

An Australian study (still in preprint, meaning its findings are yet to be reviewed by other scientists) also sought to find out similar information. In a survey of over 1,000 people, it found that 70% of participants reported having experienced at least one positive effect of the pandemic. 

e) External validity:  The sample for this survey was N =3000. Don't get distracted by the size of the sample (remember--it's not the sample size that helps establish external validity). Instead, what do you need to ask in order to decide whether the study's sample can generalize to their population of interest (Consult Ch 7)?

f) External validity: I visited the peer-reviewed article (here--open access) and found this information about their sampling technique.

The target population of the survey was adults, aged 18 years or older, currently residing in Scotland, who were interested in sharing their experience of positive change. Participants were primarily recruited through social media advertisements on Facebook and Twitter which directed participants to the online survey on Qualtrics. 

What kind of sampling did they apparently use in this study? (Name a sampling technique that is discussed in Chapter 7). 

What is their population of interest?

Does the sampling technique they used allow them to generalize from their sample to their population of interest? Why or why not? 

g) Internal validity: In a frequency claim, nobody is making a causal statement, so we don't actually need to interrogate internal validity.

h) Now that you've looked at all four validities, what do you think about the quality of this study? Make a nuanced comment about its strengths and weaknesses. 


P.S. Thanks to Marianne Lloyd for the reference!







This blog accompanies
Research Methods in Psychology:
Evaluating a World of Information
by Beth Morling

Research Methods in Psychology_9780393536294

View Sample Chapter

Instructors: Request an Exam Copy

SPSS for Research Methods:
A Basic Guide
for FREE with
Research Methods, 4e!

Research Methods in Psychology_9780393935462

Want to Contribute?

If you’re a research methods instructor or student and would like us to consider your guest post for, please contact Dr. Morling. If, as an instructor, you write your own critical thinking questions to accompany the entry, we will credit you as a guest blogger.