Everyday Research Methods

Frequency claims: Fitter and better rested during lockdown

One of the positive changes people reported was being able to spend more time with family. Photo: Steve Skjold / Alamy Stock Photo

The first peer-reviewed studies about the COVID-19 pandemic  are starting to appear in journals, and journalists are starting to summarize them in the popular press.

Let's explore this summary of a survey study conducted in Scotland and summarized for a general audience by the scientist herself, in The Conversation. The scientist writes, 

 ... for some people lockdowns have provided an unexpected opportunity to make positive changes to their lives...

Here are some details: 

In May 2020, my colleagues and I surveyed over 3,000 people in Scotland to find out what positive changes people had made in their lives during the lockdown period. 

And some of the key results:

More than half the people we surveyed reported these changes for the better: being more appreciative of things usually taken for granted (reported by 83% of participants), having more time to do enjoyable things (by 67%), spending more time in nature or outdoors (by 65%), paying more attention to personal health (by 62%), doing more physical activity (by 54%) and spending more time with a partner or spouse (by 53%).

Each of the above results constitutes a frequency claim, because it talks about the level, or percentage, associated with one variable. Consider this one, for example: "doing more physical activity (by 54%)". The variable here is "doing more physical activity under the lockdown (or not)."

Now let's use the four big validities to systematically interrogate this frequency claim, "54% of people in this Scottish survey reported doing more physical activity during the lockdown." 

a) Construct validity: How do you think they measured the variable "doing more physical activity"? (Self-report? Observational? Physiological?) . What are your initial impressions of, or questions about, how well this variable was measured? 

b) Construct validity: Let's say you wanted to establish the criterion validity (See Ch 5) of the (presumably self-report) measure of "doing more physical activity".  What behavior should self-reports of this variable correlate with, if the self-report measure is valid? 

c) Statistical validity: The point estimate, 54%, is one element of statistical validity. What about the precision of this estimate? That is, what is the 95% CI, or margin of error of this estimate? It's not provided in the article, but you can visit this calculator to estimate the CI. Do that now. 

d) Statistical validity:  Replication is another aspect of statistical validity. The scientist-author cites at least one more, similar study that came to a similar conclusion: 

An Australian study (still in preprint, meaning its findings are yet to be reviewed by other scientists) also sought to find out similar information. In a survey of over 1,000 people, it found that 70% of participants reported having experienced at least one positive effect of the pandemic. 

e) External validity:  The sample for this survey was N =3000. Don't get distracted by the size of the sample (remember--it's not the sample size that helps establish external validity). Instead, what do you need to ask in order to decide whether the study's sample can generalize to their population of interest (Consult Ch 7)?

f) External validity: I visited the peer-reviewed article (here--open access) and found this information about their sampling technique.

The target population of the survey was adults, aged 18 years or older, currently residing in Scotland, who were interested in sharing their experience of positive change. Participants were primarily recruited through social media advertisements on Facebook and Twitter which directed participants to the online survey on Qualtrics. 

What kind of sampling did they apparently use in this study? (Name a sampling technique that is discussed in Chapter 7). 

What is their population of interest?

Does the sampling technique they used allow them to generalize from their sample to their population of interest? Why or why not? 

g) Internal validity: In a frequency claim, nobody is making a causal statement, so we don't actually need to interrogate internal validity.

h) Now that you've looked at all four validities, what do you think about the quality of this study? Make a nuanced comment about its strengths and weaknesses. 

 

P.S. Thanks to Marianne Lloyd for the reference!

Producer focus: Estimating how long your survey will take

How long will your survey take? Photo: MarutStudio/Shutterstock

Here's a post for producers of research (especially students who are designing their own surveys). 

I'm a fan of the Pew Research Center's polling website, in part because they are transparent about their methods, and in part because they share educational content about how to do good surveys. For example, they have videos on random sampling and question wording that I've blogged about here before; both are worth watching.

A new example of Pew's educational content is this recent piece, called "How we keep our online surveys from running too long".

This article is referring to panel research, in which people are randomly sampled from a national address database, provided tablets and internet access if they need it, and agree to complete surveys every so often, often for several years. Pew's panel is the American Trends Panel (ATP). You can read more about panel research--and its advantages over telephone-only surveys--in Chapter 7 .

A concern for panel research is the length of a survey. You can't make your survey too long, because people will stop responding. Pew strives to keep their ATP surveys at 15 minutes or less. Here's the overview of how they do it:

In a nutshell, we classify survey questions based on their format. Each question format has a “point” value reflecting how long it usually takes people to answer. Formats that people tend to answer quickly (e.g., a single item in a larger battery) have a lower point value than questions that require more time (e.g., an open-ended question where respondents are asked to write in their own answers). A 15-minute ATP survey is budgeted at 85 points, so before the survey begins, researchers sum up all of the question point values to make sure the total is 85 or less.

There are different point values for different types of questions. You can see examples of each format in their story (here). But here's the overall approach:

Question format	Assigned points
Stand-alone question	1
Batter item (multiple, similar questions)	0.67
Open-ended question	5 or 8, depending on the requested length
Check-all that apply question	2
Vignette	1 point per 50 words
Thermometer rating	1.5

Being a data-driven company, Pew has stored information on how long each type of question takes--that's the basis for this point system. They report:

When administering an online survey, researchers can see how long it takes each respondent to answer the questions on each screen. ... more challenging open-ended questions can take people minutes to answer, while an ordinary stand-alone question takes only about 10 seconds or so.

So...if you are a student developing your own survey for a research project, you can probably use these estimates, too!  

 

Personality correlates of liking spicy foods

Photo credit: marcoscastillo/Deposit photos

Here's another article from the content farm website, StudyFinds.  This one's on the personality correlates of liking spicy foods, and it includes a combination of frequency and association claims. The same poll was also covered by the NYPost and by Yahoo! News (with a video)

Below I will post a series of quotes and questions from the StudyFinds version of the story. 

Ninety-three percent of respondents prefer some level of heat in their food. However, nearly four in 10 (36%) say they go for medium heat, 33 percent go for mild, and just 24 percent like their spice level set to hot.

a) Based on this statement, there seem to be four levels of the "Liking Spicy Foods" variable. What are they? 

The methodology of the survey is not always clear from the press stories. Here is one example:

Spicy food eaters are more likely to enjoy trying new things (76%), consider themselves attractive (62%), and are more content with their lives (66%) than those with milder heat preferences.

b) In the quote above, what is your guess about who counts as a "spicy food eater"? That is, which answer would a person have to give to the "Liking Spicy Foods" variable you listed in a) in order to be counted as a "spicy food eater"?

In the quote above, another problem is they only report the percentages for the "spicy food eaters". We'd also need to know the corresponding percentages for "those with milder heat preferences."

c) What would these corresponding percentages be able to tell us about effect size? Propose some alternative outcomes for the variable "trying new things" that would show a large effect size vs. a small one. 

Here is a quote from later in the story:

Almost one-third (32%) of spicy food fanatics follow a certain diet like vegan or vegetarian— more than double than no spice fans (13%) and more than any other spice preference.

It's great that this quote at least tells us the percentage in each group that are vegan/vegetarian. However, it still does not tell us how they operationalized being a "spicy food fanatic" vs a "no spice fan."  

d) Here's another quote from the journalist's story. Based on what you answered so far (in previous questions), what is good and bad about the coverage in this quote? 

Of those who prefer hot foods, 21 percent consider themselves extroverted, compared to 15 percent of mild food eaters. Those who pack on the heat are also more likely to describe themselves as creative (54%), confident (51%), and adventurous (44%).

Another consideration here is the cultural background of the participants. One of my colleagues grew up in India, where children often learn to eat spicy foods from an early age.

e) How might culture moderate the relationships reported here, for example, how might it moderate the relationship between liking spicy food and being extroverted? 

f) This study was conducted by a commercial polling firm, and this particular study was commissioned by the brand Frank's RedHot. Do you think this commercial funding should affect our interpretation of the data? Why or why not? 

Selected Answers:

a) The variable "Liking Spicy Foods" has four levels: No heat, mild heat preference, medium heat preference, and hot heat preference.

b) It's not clear--but it's possible that they counted anybody who picked mild, medium, or hot (that is, 93% of the sample) as a "spicy food eater"?

c) Here's an example. What if 76% of spicy food eaters like trying new things but 70% of non-spicy food eaters like trying new things? That's not a very big effect size! But if 76% of spicy food eaters like trying new things but 20% of non-spicy food eaters like trying new things, that would be a very large effect size. 

d) This quote has the same problem of not defining what a spicy food eater is. But at least they give the percentages of extroverts for two different groups.

e) If a culture serves a lot of spicy foods and even children learn to enjoy them, then it's possible that "liking spicy food" will not be strongly correlated with personality (there's a ceiling effect on the spicy food variable). The personality correlations may exist only for cultures where spicy foods are more variable.

 

 

 

 

 

 

 

Racial disparities in media coverage

The study evaluated both victims and defendants in criminal cases. Compared to white victims, the media were much less likely to depict photos of Black victims of crime with family or friends. How might this choice impact people's empathy for Black pain and suffering? Credit: Rawpixel.com/Shutterstock

The Equal Justice Initiative (EJI) is a human-rights organization whose mission is to challenge racial injustice and "end mass incarceration and excessive punishment in the United States." EJI was founded by attorney Bryan Stevenson, who wrote the book Just Mercy about his experiences advocating for people who were unjustly imprisoned. (The book is also a feature film).

The EJI recently worked with Global Strategy Group to document ways that the criminal cases of white and Black defendants have been portrayed by the media. The infographic created by their study provides a social-justice themed example of a correlational study or quasi-experiment.

In the study, the GSG:

conducted a comprehensive media analysis of national and local coverage around 10 criminal cases – 5 with a Black defendant and 5 with a white defendant. We collected and collated information on more than 20 different topics, including the use of imagery, language choices, framing of defendant and victim, and reporter background.

The main findings of the GSG/EJI study are depicted in this colorful infographic. Here are a few highlights.

First, the study found that 

Stark disparities exist between the types of images used for Black and white defendants

The study found that media outlets were four times more likely to run the mugshot of a Black defendant compared to a white defendant. How might this bias people's impressions of defendants? 
Credit: nimito/Shutterstock

Specifically, the study measured the types of photos that media outlets chose to depict white vs. Black defendants. 9% of the time, the media used mugshots for white defendants, but 45% of the time, the media used mugshots for Black defendants. In contrast, 13% of the time, the media showed white defendants in a suit and tie, but only 6% of the time for Black defendants.

Second, the media also differed in how the victims of crime were depicted. The infographic reports:

White victims [79%] were nearly 4x than Black victims [21%] to have a photo with friends or family included alongside coverage, reinforcing existing tendencies to dehumanize Black pain and suffering and, by the same token, put a face to white victims.

Third, the study also found that race was associated with whom the media quoted about the defendant. They write: 

Quotes from family and friends were nearly twice as likely to appear in articles about white defendants [50%] than articles about Black defendants [25%]. Black defendants were more likely to have judges or lawyers weigh in instead, which presents readers with a less humanizing account of Black defendants.

Questions

a) Select one of the three quoted findings above (or, find your own from the infographic). Identify the variables in the finding you chose. What are the levels of each variable?

b) All of the studies above can be described as correlational. Why? (refer to the terms "measured" and "manipulated" in your answer).

c) If you've studied Chapter 13, you might wish to describe these studies as quasi-experimental instead of correlational. (In studies like these, you could make the case either way). Why do both labels (correlational and quasi-experimental) make sense here? 

d) How might the GSG and EJI take this study further with an experimental method? What would they manipulate and what could they measure? 

Scientific reasoning questions: 

At the bottom of the infographic, the GSG included the following detail about their methodology:

Audit was conducted among ten criminal cases, 5 cases featuring a white defendant and 5 featuring a Black defendant. All cases had criminal proceedings that occurred in the past seven years. GSG analyzed 20 to 30 articles for each case, for a total of 257 articles, all of which were randomly selected and included a combination of national and local online coverage.

e) Why was it important that the media articles about each case were selected randomly? 

f) The GSG is not very clear on how they selected the 10 initial cases (the 5 with a white defendant and the 5 with a Black defendant). Why do we need to know more about how these specific 10 cases were selected? 

g) Scroll to the bottom of the infographic and consult the 10 criminal cases they selected. Do you notice any systematic differences in the cases selected for white vs. Black defendants? 

Estimating how many U.S. adults identify as nonbinary

 

Photo credit: nito103/Deposit Photos

Press outlets have been covering a report by the Williams Institute (based at UCLA), a research organization collecting data on demographic information on sexual orientation and gender identity. The quotes in this post come from this summary from PBS.

The new study has estimated the number of LGBTQ adults in the U.S. who identify as nonbinary. (Nonbinary is a term that is often used to describe people whose gender identities are more than the man/woman category; some nonbinary people use they/them pronouns.)  It's important to estimate how many U.S. adults identify as nonbinary (as well as other sexual- and gender-minority identities) because anti-discrimination laws potentially affect them. PBS reports that:

Nonbinary people are part of a “growing subgroup” of the LGBTQ population, Wilson said, adding there’s been greater visibility for nonbinary people. More and more celebrities, like singers Demi Lovato and Sam Smith, and actors Indya Moore and Amandla Stenberg, have talked about identifying themselves as nonbinary or in related terms.

Here's a portion of the PBS summary: 

According to the study, which focused on people ages 18 to 60, nonbinary people make up about 11 percent of the adult LGBTQ population in the country. The study builds on the work of two prior surveys — one on trans adults, [TransPop] the other on cisgender LGBTQ adults [Generations]— to provide more comprehensive data on nonbinary people, a historically underresearched subgroup.

When the researchers extrapolated out form this subsample, they concluded that

There are about 1.2 million LGBTQ adults in the U.S. who are nonbinary.

The data in this report constitute a frequency claim, because they are estimating the level of one variable (that is, nonbinary identity).

First let's talk about the precision of these estimates. The report from the Williams Institute (here) includes 95% CIs for the estimates. For example, the two estimates from the PBS story were presented this way:

Number of nonbinary LGBTQ adults in the U.S., including transgender and cisgender people N [95% CI]	1,219,000 [1,000,000 - 1,481,000]
% [95% CI] of LGBTQ adult population	11.1 [9.3, 13.2]

a) In your own words, describe what the first CI means. Use this definition and template to help you: The 95% CI is meant to capture the true value in the population 95% of the time. The true value in the population might be as low as ____ or as high as ____. 

b) Sometimes the 95% CI misses the true value in the population. What percentage of the time will the CI miss?

Next lets' talk about external validity (generalizability) and sampling. As you consider these estimates keep in mind that there is a distinction between estimating the number of all U.S. adults who identify as non-binary and the number of U.S. LGBTQ adults who identify as non-binary. These values may be different. 

c) Given what you've read so far, what seems to be the population of interest of this study? 

To know if the sample is generalizable, you need to know how they obtained the sample. The PBS summary doesn't say anything about the sampling method used in this study, and neither does the Williams Institute report. To locate information about the sampling, I had to go to the methods sections of the two studies on which the nonbinary study is based: The TransPop study (which studied a sample of transgender U.S. adults) and the Generations study (which studied a sample of LGBTQ U.S. adults). 

In these methods sections, they say that these two studies (TransPop and Generations) worked with the Gallup organization to recruit nationally representative samples of transgender and LGBTQ adults. Gallup performs a Daily Tracking Survey in which they call a national probability sample of 1000 adults, almost every day of the year. These adults are sampled using random-digit dialing, including both landline and cell phone numbers. Gallup asks a variety of questions about well-being, political attitudes, and economic situations. At the end of each  telephone survey, the interviewer asks, “I have one final question we are asking only for statistical purposes. Do you, personally, identify as lesbian, gay, bisexual, or transgender?” Gallup retained the records for people who said yes to this question over one year, and the Generations and TransPop studies contacted these folks later. At this second contact, researchers checked to be sure each person was eligible for the Generations or TransPop study, and if they were, the researchers sent them a survey by mail or email. To be eligible, people had to identify as sexual minority or transgender, be within certain age ranges, and be able to complete surveys in English. 

After people were recruited into the TransPop and Generations studies in this way, they were asked multiple survey questions, including the target question about whether they identified as nonbinary. 

d) In your own words, explain why the procedure used above is likely to lead to a probability (in other words, a generalizable, or externally-valid) sample of LGBTQ adults in the United States. 

e) Reading the description above, which type of random was used here--random sampling or random assignment? 

f) To reiterate, the study concluded that about 11% of LGBTQ people in the United States identify as nonbinary. What method could we use to estimate the percentage of all people in the United States identify as nonbinary? 

g) What do you predict might happen to the estimate of nonbinary people in the U.S. if they redid this study 20 years in the future? 

 

Suggested Answers

a) The 95% CI is meant to capture the true value in the population 95% of the time. The true value in the population might be as low as 1 million or as high as 1.48 million. Importantly, our CI may have missed the true population mean. We don't know if it missed it or not, but we know the probability that our CI missed it--see question b)!

b) the 95% CI will miss the population mean in 5% of cases.  

More information: If we replicated the study, the new estimate and 95% CI will be a little bit different from the first one. But we'd still expect that the replication estimate of the number of nonbinary adults will be between 1 million and 1.48 million. We'd be surprised if our new estimate was a lot lower or a lot higher than this range. 

c) The population of interest is LGBTQ adults in the United States (who have access to a cell phone or landline)

d) The Gallup sample is generalizable because their Daily Tracking Survey goes out to a random sample U.S. adults. Any LGBTQ folks that are called in that survey are therefore also a random sample, and random samples are generalizable.  

e) They used random sampling. (As a reminder, random assignment is used in experiments, to assign participants to different groups.)  

f) One method could be to ask directly of a random sample of US adults. For example, Gallup could potentially add a nonbinary identity question to their Daily Tracking Survey for one year.

Another method was the one used in the present study. Gallup learned that 3.5% of their Daily Tracking Survey sample answered yes to the LGBTQ question, and then, of that subsample, a certain percentage (about 11%) identified as nonbinary. After adjusting weights, they got a percentage, (maybe around .05%) which they multiplied by the entire U.S. adult population, to produce the estimate that about 1,219,000 adults (or between 1 and 1.48 million) identify as nonbinary. To see some of the statistics I quoted here, visit this page)

g) Answers will vary

Teaching with 100 Humans

In 2020, Netflix released an 8-episode series called "100 Humans" in which 100 (American) humans from a variety of ages and social groups participate in a series of studies. It's a gold mine of examples for critical thinking in research methods. You'll find examples of posttest-only, pretest-posttest designs, HARKing, ethics, and more. Most of the studies have flaws, but that makes them more fun to critique. 

I've cataloged ways you can use the series in your teaching: www.tinyurl.com/teachingwith100humans.

Reach out if you use it! Let me know how it works for you. 

 

Random sampling in action for public health

Why is random sampling important when trying to estimate antibody levels in the population? Photo: © Eugene Lu/Shutterstock

Here is a timely example of random sampling in action.

Watch this 2-minute video about the CDC's project in Atlanta (produced by NPR). It explains how nurses are visiting about 500 people in the Atlanta metropolitan area to see how widespread the Coronoavirus is.  In the first minute of the video, the narrator reports,

The CDC is sending out employees to conduct antibody tests on a random sample of Atlanta residents. The tests are meant to show how many people have been infected with the Coronavirus.

The CDC wants to get blood samples from 450 households. 

(Note: And here's another story of a similar effort, this time in Corvallis, Oregon, USA. It's about prevalence of live cases, rather than antibodies, but a random sample is still a key player.)

Questions:

a) Why would it be important to conduct antibody tests on a random sample of people from the Atlanta area?

b) The plan is to collect samples from 450 households. Imagine that a friend hears this figure and expresses skepticism--"how can 450 be enough to generalize to the population of Atlanta?"  Explain to your friend why a random sample--even if it has "only" 450 households--can generalize to the population.

c) What are some ways the CDC might conduct this random sample? Review the possibilities in Chapter 7 and explain:

i) how might they obtain a simple random sample of Atlanta residents?
ii) how might they obtain a cluster sample (what could they use as clusters?)
iii) how might they conduct a stratified random sample? (Imagine that they want to use neighborhoods/zipcodes as the strata.)

 d) After conducting the antibody study on a random sample of Atlanta residents, the CDC will have an estimate of the rate of infection in the sample. To whom will they be able to generalize this rate of infection--that is, to which population of interest?

e) What might happen if some of the randomly sampled people in Atlanta refuse to have their blood drawn for this CDC study? Will that affect the sample's generalizability?  Why or why  not?

f) Which big validity is this example concerning? 

Many thanks to Dr. Jess Harnett (of the notawfulandboring blog), who posted this example, and thanks to Kathy Becker-Blease for sharing the Oregon example. 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers. 

 

Question wording matters

Collecting accurate data in a poll is difficult business. Many of us focus on the sampling strategies of polling organizations, and rightly so: External validity depends on whether we include "cell phone only" numbers in our samples, how we account for different rates of responding across certain groups, and so on. However, this post reminds us that question wording also matters. Construct validity is just as important for polls.

In this report, Pew Research Center describes how people's opinions change depending on how a question is posed. The poll asked Americans whether we should increase the size of the House of Representatives. People in the poll were randomly assigned to hear either the original question, or the same question with additional context. Notice how people's support changes when additional context is added: 

“The Electoral College, Congress, and representation,” Pew Research Center, Washington, D.C. (April 26, 2018). https://pewrsr.ch/2rLd0OI, accessed on May 16, 2018

 

a) Even though this is a polling example, Pew conducted an experiment. What are its independent and dependent variables? 

b) What causal claim could this polling result support? 

c) In your opinion, which polling question--the original one or the one with added context--provides the most accurate measure of people's opinion? 

d) Find another poll question on the Pew Research web site. Speculate how some additional information might change people's responses. 

 

 

 

 

 

 

 

Below is another presentation of the same results. Notice how the two  different question wordings seem to affect people differently, depending on their political affiliation.

“The Electoral College, Congress, and representation,” Pew Research Center, Washington, D.C. (April 26, 2018). https://pewrsr.ch/2rLd0OI, accessed on May 16, 2018

 

e) Why might Democrats be more influenced by the contextual information provided? 

f) This could be viewed as a 2 x 2, IV x PV factorial design, in which the IV is "question wording" (original or added context) and the PV is "political affiliation" (Democrat or Republican). If you've studied Chapter 12, perhaps you can identify whether there are main effects and interactions in this pattern of data. 

The same Pew report contains additional examples of how question wording alters responses to a question about the Senate, and shows how different subgroups respond to questions about the Electoral College. 

Why strive for large samples?

When the general public critiques research, I often hear them say that the samples are "too small."  It's true that sample sizes (N) in psychology research should be large.  One of the outcomes of the so-called "replication crisis" is that large samples are more and more important in psychology. But why?  

A common misconception--held by both students and the general public--is that large samples are important because they ensure external validity. This misconception is incorrect. External validity (that is, the ability to generalize from a sample to a population of interest) is about how a sample has been recruited, not how many people are in it (see Chapter 7, 14). For example, say you recruited a sample of 1000 fans attending the national championship college game. You'd have a pretty large sample, but you couldn't generalize from that sample to college students in the U.S. (for example). In fact, unless the 1000 fans were selected at random from the 70,000 fans at the game, you couldn't even generalize from this sample to "people attending the national championship football game."   

If not external validity, why are large samples important? It's about accuracy of our statistical estimates. When estimating values in the population such as means or differences between means, large samples are less likely to be influenced by chance variability. For example, imagine you're estimating the mean height of kindergarteners in your local school. Now imagine that you select 5 kindergarteners at random, one of whom, by chance, turns out to be extremely tall for her age. That tall kindergartener is going to "pull" the mean estimate upwards when combined with only 4 other kids. But what if you select 25 kindergarteners instead? Now the tall kindergartener is going to be balanced out by 24 other scores, and her height will have less influence on the mean estimate. 

Below is a pair of animations that illustrate this principle. They come from the data science blog R Explorations. The animation used the program R to run a simulation study over and over and over. First, they created a very large population of scores whose mean was known to be 10.0 and whose standard deviation was known to be 1.0. Then they asked the computer to draw a random sample of size 10, compute the mean of the 10 scores, and plot them. You can watch the samples appear in real time on the animation below. Here, xbar is the sample's mean and s is the sample's standard deviation. The red line represents the mean for each sample as it is drawn: 

Questions

a) First, watch the top animation, where N = 10. What do you notice about the movement of the vertical red line representing the mean in the top animation? What is it doing, and what does that represent?

b) Now watch the bottom animation, where N = 1000. What do you notice about the movement of the vertical red line representing the mean in this second animation? What is it doing, and what does that represent?

c) What do you notice about the s values of the two animations? Which animation has a steadier estimate of s?

d) Answer this one only if you've had a statistics course: Which of the two animations will have a smaller standard error? How is the standard error represented in the two animations? 

e) Given the behavior of the two animations, explain why a large sample is important for research. 

f) Which validity does sample size best address, if not external validity?

g) Let's tie this concept back to the "replication crisis" (or, as some are now calling it, "credibility revolution"*). When a finding in psychology has not replicated in a direct replication study, one reason might be that the original study used a small sample. Another reason might be that the replication study used a small sample. Why might the sample size of a study be linked to its replicability? Explain in your own words. 

Both animations come from the R Explorations blog

*Thanks to Dr. Simine Vazire for showing us this animation at NITOP conference and for introducing me to the term "credibility revolution"

 

Online resource: Video about random sampling

Pew Research is my favorite polling resource, partly because they ask such interesting questions, and partly because they are so transparent about sharing their methodology. (For examples, see their Methods page or click on the full Report Materials for a study they did on gun ownership in America.) They make their sampling techniques and question wording easily available.

Now Pew has shared a video that explains how a sample of 1,000 can be used to draw inferences about the population. Instructors: Save this one for when you teach Chapter 7!