Everyday Research Methods

Producer focus: Estimating how long your survey will take

How long will your survey take? Photo: MarutStudio/Shutterstock

Here's a post for producers of research (especially students who are designing their own surveys). 

I'm a fan of the Pew Research Center's polling website, in part because they are transparent about their methods, and in part because they share educational content about how to do good surveys. For example, they have videos on random sampling and question wording that I've blogged about here before; both are worth watching.

A new example of Pew's educational content is this recent piece, called "How we keep our online surveys from running too long".

This article is referring to panel research, in which people are randomly sampled from a national address database, provided tablets and internet access if they need it, and agree to complete surveys every so often, often for several years. Pew's panel is the American Trends Panel (ATP). You can read more about panel research--and its advantages over telephone-only surveys--in Chapter 7 .

A concern for panel research is the length of a survey. You can't make your survey too long, because people will stop responding. Pew strives to keep their ATP surveys at 15 minutes or less. Here's the overview of how they do it:

In a nutshell, we classify survey questions based on their format. Each question format has a “point” value reflecting how long it usually takes people to answer. Formats that people tend to answer quickly (e.g., a single item in a larger battery) have a lower point value than questions that require more time (e.g., an open-ended question where respondents are asked to write in their own answers). A 15-minute ATP survey is budgeted at 85 points, so before the survey begins, researchers sum up all of the question point values to make sure the total is 85 or less.

There are different point values for different types of questions. You can see examples of each format in their story (here). But here's the overall approach:

Question format	Assigned points
Stand-alone question	1
Batter item (multiple, similar questions)	0.67
Open-ended question	5 or 8, depending on the requested length
Check-all that apply question	2
Vignette	1 point per 50 words
Thermometer rating	1.5

Being a data-driven company, Pew has stored information on how long each type of question takes--that's the basis for this point system. They report:

When administering an online survey, researchers can see how long it takes each respondent to answer the questions on each screen. ... more challenging open-ended questions can take people minutes to answer, while an ordinary stand-alone question takes only about 10 seconds or so.

So...if you are a student developing your own survey for a research project, you can probably use these estimates, too!  

 

Frequency claims: Remembering brand logos

This project provided several frequency claims you can analyze.
Photo credit: LukaFunduk/Deposit Photos

Before you read on, take out a pencil and draw the Apple logo (no peeking at your own device!) Now try to draw the Adidas logo, too (no peeking at your feet, either). 

Which one was easier? How confident are you in the accuracy of your drawings?

After you've drawn these two logos, visit this website, where you can compare your own drawings to those of other people. You can also see the official logos for Apple and Adidas. How well did you do?  More importantly, how well did the people in their study do?

This website presents data from a study of Americans who were asked to draw a variety of brand logos. The website introduces the project:

...we asked over 150 Americans to draw 10 famous logos from memory as accurately as they could. Based on more than 1,500 drawings created over a period of 80 hours, the results reveal that, far from being stamped perfectly in our collective memory, these ubiquitous emblems largely exist as fuzzy visions in our mind's eye. 

The text on the website reports:

Overall, 16 percent of people drew near perfect logos, and 37 percent were good but not perfect. As we would expect, the more complex the logo, the less likely people are to remember it in full.

Across the 10 brands (which included not just Apple and Adidas, but also Starbucks, Foot Locker, Burger King, and Domino's), the percentage of people who sketched near-perfect logos changed. For example, the website reports that 20% drew near-perfect Apple logos, whereas 12% drew near-perfect Adidas logos.  

You can evaluate certain methodological aspects of this project. For example, here's a quote from their discussion of the Apple logo results:

Interestingly, a smaller proportion of people in our experiment put the bite on the wrong side of the apple (22 percent)

a) Wait, was this study really an experiment? why or why not?

The article made some frequency claims. For example, 

One in 5 people thinks the Foot Locker referee wears a hat (he doesn't), and nearly half of people believe the Starbucks mermaid does not wear a crown (she does).

b) Why can each of these two claims be considered frequency claims? 

c)  When people make frequency claims, external validity (also known as generalizability) is pretty crucial. Think about the external validity of this project. Do you think the study they conducted is able to support the claim that "one in 5 people thinks the Foot Locker referee wears a hat?" What information do you need to know in order to evaluate the external validity of this claim?

d) Now consider statistical validity. How precise are the estimates on this website? The study estimated that 20% of the sample drew near-perfect Apple logos. But what's the precision of that estimate? In other words, what's the 95% CI, or margin of error? You can compute it here, if you know the sample size (which, just to remind you, was around 150). 

What was the MoE you found?  What would happen to that MoE if the sample had been larger? (Caveat--MoE computations actually depend on having a random sample, which we don't have, so these estimates are not going to be perfect.)

e) Finally, consider construct validity. The study needed to decide how accurate each person's sketch was. One of their footnotes reads, "accuracy based on independent ratings by a panel of marketing professionals." How do you think they obtained these ratings--what might their procedure have been?

f) Using the large matrices of drawings, make some comments about the construct validity of their operationalization of "accuracy". 

 

Thanks to Marianne Lloyd for sending me this page!

Teaching with 100 Humans

In 2020, Netflix released an 8-episode series called "100 Humans" in which 100 (American) humans from a variety of ages and social groups participate in a series of studies. It's a gold mine of examples for critical thinking in research methods. You'll find examples of posttest-only, pretest-posttest designs, HARKing, ethics, and more. Most of the studies have flaws, but that makes them more fun to critique. 

I've cataloged ways you can use the series in your teaching: www.tinyurl.com/teachingwith100humans.

Reach out if you use it! Let me know how it works for you. 

 

Training tools for information literacy

Photo: fotogestoeber/Shutterstock

More than ever, our citizenry needs skills for information literacy. A recent study of almost 4000 U.S. teens, conducted by Stanford's History Education group, concluded that teenagers' ability to think critically about online information is "troubling." Their report included the findings that: 

• Fifty-two percent of students believed a grainy video claiming to show ballot stuffing in the 2016 Democratic primaries (the video was actually shot in Russia) constituted “strong evidence” of voter fraud in the U.S. [...]
• Two-thirds of students couldn’t tell the difference between news stories and ads (set off by the words “Sponsored Content”) on Slate’s homepage.
• Ninety-six percent of students did not consider why ties between a climate change website and the fossil fuel industry might lessen that website’s credibility. [...]

If you're inspired to build skills in your own students, consider assigning an activity designed to develop students' critical reading skills as a supplement to Chapter 2. The following tools have been developed by experts in library science, communication, and social cognition. All of these raise our awareness of the information we may encounter online: 

Games that Teach

1. The free online game Bad News  trains people to spread false ideas with the goal of teaching them how propaganda works. It also includes an (optional) embedded test of people's ability to detect real and fake Twitter accounts. Users earn badges for impersonating experts, exploiting emotion, and polarization. It's a legitimately fun game! What's more, two recent studies have indicated that this game improves people's reasoning about the news (Roozenbeek et al., 2019;  Basol et al., 2020).

2. The Fake News Game (still in development) plans to teach about fake news by challenging people to create their own fake news, building in elements intended to make it go viral. 

Online Curricula

3. Stanford's Civic Online Reasoning (COR) course presents several planned sets of lessons:

• • An three-lesson introduction to skeptical reading, looking for evidence, and fact-checking
  • An six-lesson overview of lateral reading, the strategy used by professional fact-checkers (and a skill associated with better information literacy)
  • A combination package of six lessons offering a little bit of everything (lateral reading, introduction to evidence, and "click restraint"). This might be the package to assign if you're only assigning one set of lessons.

4. Carl Bergstrom and Jevin West's site, Calling Bullshit, offers a syllabus for a full semester college course on misinformation. You can also assign some of their "tools and tricks" to give people practice in the following skills:

• • Spotting misleading axes and formatting in graphs
  • Deciding if a scientific paper is legitimate
  • Deciding if an online face is real or computer-generated
  • They also have case studies in misinformation, sorted by difficulty. 

5. Morton Ann Gernsbacher has developed and shared her free, online methods course focused primarily on scientific information literacy. Check it out for a full set of resources!

My own students have benefitted from working through various modules from these resources, in both methods and content courses. 

I'm grateful to David Dunning (University of Michigan), who shared some of these resources at his NITOP general session. I'm also grateful to UD librarians Meg Grotti and Alison Wessel for sharing resources with me. 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers. 

 

Articles that are student-friendly

A list of recent articles that are student-accessible (updated Feb 2021)

Compiled by Beth Morling and Jeong-Min Lee, Georgia State

Our students have used these articles in their final projects for Research Methods in the past. In this final project, students summarize and analyze the validities for two empirical studies. (I can provide the assignment upon request: [email protected])

Most of these articles report more than one study, and at least one of the studies is simple enough for a student with basic skills in research methods to understand. The most complex experimental design would be a 3-way factorial design. The most complex correlational design uses multiple regression and (sometimes) mediation.

Recent additions (teal highlights were added Jan '22):

Broda, M., Yun, J., Schneider, B., Yeager, D. S., Walton, G. M., & Diemer, M. (2018). Reducing inequality in academic success for incoming college students: A randomized trial of growth mindset and belonging interventions. Journal of Research on Educational Effectiveness, 11(3), 317–338. https://doi.org/10.1080/19345747.2018.1429037

Buckels, E. E., Jones, D. N., & Paulhus, D. L. (2013). Behavioral confirmation of everyday sadism. Psychological Science, 24(11), 2201–2209. https://doi.org/10.1177/0956797613490749  [good example of criterion validity]

Chang, F., Luo, M., Walton, G. M., Aguilar, L., & Bailenson, J. (2019). Stereotype threat in virtual learning environments: Effects of avatar gender and sexist behavior on women’s math learning outcomes. Cyberpsychology, Behavior, and Social Networking, 22, 1-7.
[This is a 2x2]

Dietze, P., & Knowles, E. D. (2021). Social class predicts emotion perception and perspective-taking performance in adults. Personality and Social Psychology Bulletin, 47(1), 42–56. doi: doi.org/10.1177/0146167220914116

Goh, J. X., Bandt-Law, B., Cheek, N. N., Sinclair, S., & Kaiser, C. R. (2021). Narrow prototypes and neglected victims: Understanding perceptions of sexual harassment. Journal of Personality and Social Psychology. https://doi.org/10.1037/pspi0000260 
[Focus on Study A1]

Guhn, M., Emerson, S. D., & Gouzouasis, P. (2020). A population-level analysis of associations between school music participation and academic achievement. Journal of Educational Psychology, 112(2), 308-320. https://doi.org/10.1037/edu0000376 [Focus on Figure 2]

Halberstadt, A. G., Cooke, A. N., Garner, P. W., Hughes, S. A., Oertwig, D., & Neupert, S. D. (2020). Racialized emotion recognition accuracy and anger bias of children’s faces. Emotion, http://dx.doi.org/10.1037/emo0000756  [Focus on Figure 3]

Hehman, E., Flake, J. K., & Calanchini, J. (2018). Disproportionate use of lethal force in policing is associated with regional racial biases of residents. Social Psychological and Personality Science, 9(4), 393–401. doi: 10.1177/1948550617711229 
[good example of regression]

Hittner, E. F., Stephens, J. E., Turiano, N. A., Gerstorf, D., Lachman, M. E., & Haase, C. M. (2020). Positive affect is associated with less memory decline: Evidence from a 9-year longitudinal study. Psychological Science, 31(11), 1386–1395. https://doi.org/10.1177/0956797620953883
[Focus on Table 2; good example of regression]

Leonard, J. A., Martinez, D. N., Dashineau, S. C., Park, A. T., & Mackey, A. P. (2021). Children persist less when adults take over. Child Development, cdev.13492. https://doi.org/10.1111/cdev.13492
[correlational and experimental designs]

Scullin, M. K., Gao, C., & Fillmore, P. (2021). Bedtime music, involuntary musical imagery, and sleep. Psychological Science, 32(7), 985–997. https://doi.org/10.1177/0956797621989724

 

Schünemann, B., Keller, J., Rakoczy, H., Behne, T., & Bräuer, J. (2021). Dogs distinguish human intentional and unintentional action. Scientific Reports, 11(1). https://doi.org/10.1038/s41598-021-94374-3

 

Silvia, P., Christensen, A. P., & Cotter, K. N. (2020). Right-wing authoritarians aren't very funny: RWA, personality, and creative humor production. Personality and Individual Differences, 170, https://doi.org/10.1016/j.paid.2020.110421. [Nice bivariate correlation matrix]

 

Siniscalchi, M., Lusito, R., Vallortigara, G., & Quaranta, A. (2013). Seeing left- or right-asymmetric tail wagging produces different emotional responses in dogs. Current Biology, 23(22), 2279-2282. https://doi.org/10.1016/j.cub.2013.09.027 

Thomas, C. C., Otis, N. G., Abraham, J. R., Markus, H. R., & Walton, G. M. (2020). Toward a science of delivering aid with dignity: Experimental evidence and local forecasts from Kenya. Proceedings of the National Academy of Sciences, 117(27), 15546–15553. https://doi.org/10.1073/pnas.1917046117

Webb, N. L., & Drummond, P. D. (2001). The effect of swimming with dolphins on human well-being and anxiety. Anthrozoös, 14(2), 81–85. https://doi.org/10.2752/089279301786999526   [Quasi-experiment; nice selection effect example]

Wolf, W., & Tomasello, M. (2020). Watching a video together creates social closeness between children and adults. Journal of Experimental Child Psychology, 189, 104712. https://doi.org/10.1016/j.jecp.2019.104712
[Study 1]

Older examples

Brummelman, E., Thomaes, S., de Castro, B. O., Overbeek, G., & Bushman, B. J. (2014). “That’s not just beautiful—that’s incredibly beautiful!”: The adverse impact of inflated praise on children with low self-esteem. Psychological Science, 25, 728-735. doi:http://dx.doi.org/10.1177/0956797613514251

Bryan, C.J., Yeager, D.S., Hinojosa, C.P., Chabot, A., Bergen, H., Kawamura, M., & Steubing, F. (2016). Harnessing adolescent values to motivate healthier eating. PNAS, 113, 10830-10835. doi: 10.1073/pnas.1604586113

Cantú, S. M., Simpson, J. A., Griskevicius, V., Weisberg, Y. J., Durante, K. M., & Beal, D. J. (2014). Fertile and selectively flirty: Women’s behavior toward men changes across the ovulatory cycle. Psychological Science, 25, 431-438. Retrieved from http://search.proquest.com/docview/1501839110?accountid=10457

Chan, K. Q., Tong, E. M. W., & Tan, Y. L. (2014). Taking a leap of faith: Reminders of God lead to greater risk taking. Social Psychological and Personality Science, 5, 901-909. doi: 10.1177/1948550614537309

Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251–19257. https://doi.org/10.1073/pnas.1821936116

Fraley, R.C., Griffin, B.N., Belsky, J., & Roisman, G.I. (2012). Developmental antecedents of political ideology: A longitudinal investigation from birth to age 18 years. Psychological Science, 23, 1425-1431. doi: 10.1177/0956797612440102

Francis, H. M., Stevenson, R. J., Chambers, J. R., Gupta, D., Newey, B., & Lim, C. K. (2019). A brief diet intervention can reduce symptoms of depression in young adults – A randomised controlled trial. PLOS ONE, 14(10), e0222768. https://doi.org/10.1371/journal.pone.0222768

 Frenda, S., Patihis, L., Loftus, E., Lewis, H., Fenn, K.  (2014) Sleep deprivation and false memories. Psychological Science, 25, 1674-1681. doi: 10.1177/0956797614534694

 Gino, F., & Wiltermuth, S. S. (2014). Evil genius? How dishonesty can lead to greater creativity. Psychological Science, 25, 973-981.doi: 10.1177/0956797614520714

Gino, F., Norton, M. I., & Ariely, D. (2010). The counterfeit self: The deceptive costs of faking it. Psychological Science, 21, 712-720. doi:http://dx.doi.org/10.1177/0956797610366545

Gunraj, D. N., Drumm-Hewitt, A. M., Dashow, E. M., Upadhyay, S. S. N., & Klin, C. M. (2016). Texting insincerely: The role of the period in text messaging. Computers in Human Behavior, 55, 1067–1075. https://doi.org/10.1016/j.chb.2015.11.003

House, J., DeVoe, S. E., & Zhong, C. (2014). Too impatient to smell the roses: Exposure to fast food impedes happiness. Social Psychological and Personality Science, 5, 534-541. doi:http://dx.doi.org/10.1177/1948550613511498 [use study 3]

Jirout, J. J., & Newcombe, N. S. (2015). Building blocks for developing spatial skills: Evidence from a large, representative U.S. sample. Psychological Science, 26, 302-310. doi:http://dx.doi.org/10.1177/0956797614563338

 Kruse, E., Chancellor, J., Ruberton, P. M., & Lyubomirsky, S. (2014). An upward spiral between gratitude and humility. Social Psychological and Personality Science, 5, 805-814. doi:http://dx.doi.org/10.1177/1948550614534700 [Study 2]

Lee, K., Talwar, V., McCarthy, A., Ross, I., Evans, A., & Arruda, C. (2014). Can classic moral stories promote honesty in children? Psychological Science, 25, 1630-1636. doi:10.1177/0956797614536401

Levy, K. N., & Kelly, K. M. (2010). Sex differences in jealousy: A contribution from attachment theory. Psychological Science, 21, 168-173. doi:http://dx.doi.org/10.1177/0956797609357708

Markey, P. M., Ivory, J. D., Slotter, E. B., Oliver, M. B., & Maglalang, O. (2019). He does not look like video games made him do it: Racial stereotypes and school shootings. Psychology of Popular Media Culture. https://doi.org/10.1037/ppm0000255

 Mrazek, M. D., Franklin, M. S., Phillips, D. T., Baird, B., & Schooler, J. W. (2009). Mindfulness training improves working memory capacity and GRE performance while reducing mind wandering. Psychological Science, 24, 776-781. DOI: 10.1177/0956797612459659.

Nelson, S. K., Kushlev, K., English, T., Dunn, E. W., & Lyubomirsky, S. (2013). In defense of parenthood: Children are associated with more joy than misery. Psychological Science, 24, 3-10. doi:http://dx.doi.org/10.1177/0956797612447798

 Nusbaum, E. C., & Silvia, P. J. (2011). Shivers and timbres: Personality and the experience of chills from music. Social Psychological and Personality Science, 2, 199-204. doi:http://dx.doi.org/10.1177/1948550610386810 [Focus on “personality and chills during music” and “do music preferences mediate effects of openness” results]

 Ronay, R., & von Hippel, W. (2010). The presence of an attractive woman elevates testosterone and physical risk taking in young men. Social Psychological and Personality Science, 1, 57-64. doi:http://dx.doi.org/10.1177/1948550609352807 [focus on “skateboarding” and “testosterone mediation” results]

 Saslow, L. R., Muise, A., Impett, E. A., & Dubin, M. (2013). Can you see how happy we are? facebook images and relationship satisfaction. Social Psychological and Personality Science, 4, 411-418. doi:http://dx.doi.org/10.1177/1948550612460059 [Study 2]

 Seder, J. P., & Oishi, S. (2012). Intensity of smiling in facebook photos predicts future life satisfaction. Social Psychological and Personality Science, 3, 407-413. doi:http://dx.doi.org/10.1177/1948550611424968 [focus on Study 1]

Tice, D. M., & Baumeister, R. F. (1997). Longitudinal study of procrastination, performance, stress, and health: The costs and benefits of dawdling. Psychological Science, 8, 454-458. Retrieved from http://search.proquest.com/docview/619221802?accountid=10457

 Valor-Segura, I., Expósito, F., Moya, M., & Kluwer, E. (2014). Don't leave me: The effect of dependency and emotions in relationship conflict. Journal of Applied Social Psychology, 44, 579-587. doi:http://dx.doi.org/10.1111/jasp.12250 [use Study 1]

 Vezzali, L., Stathi, S., Giovannini, D., Capozza, D., & Trifiletti, E. (2014). The greatest magic of harry potter: Reducing prejudice. Journal of Applied Social Psychology, 45, 105-121. doi:http://dx.doi.org/10.1111/jasp.12279  [Study 1]

Witkower, Z., & Tracy, J. L. (2019). A facial-action imposter: How head tilt influences perceptions of dominance from a neutral face. Psychological Science, 30(6), 893–906. https://doi.org/10.1177/0956797619838762

 Wong, E., Ormiston, M., Haselhuhn, M. (2011). A face only an investor could love: CEOs’ facial structures predict their firms’ financial performance. Psychological Science, 22, 1478-1483. doi: 10.1177/0956797611418838.

Online resource: Writing good questions

Here's a second video in a series by Pew Research. This 5 minute clip describes some of the issues in writing good questions for an opinion poll. It basically summarizes the first part of Chapter 6, and provides some new, concrete examples. 

You can use this in combination with Pew's first video about random sampling, which I described here.

Why strive for large samples?

When the general public critiques research, I often hear them say that the samples are "too small."  It's true that sample sizes (N) in psychology research should be large.  One of the outcomes of the so-called "replication crisis" is that large samples are more and more important in psychology. But why?  

A common misconception--held by both students and the general public--is that large samples are important because they ensure external validity. This misconception is incorrect. External validity (that is, the ability to generalize from a sample to a population of interest) is about how a sample has been recruited, not how many people are in it (see Chapter 7, 14). For example, say you recruited a sample of 1000 fans attending the national championship college game. You'd have a pretty large sample, but you couldn't generalize from that sample to college students in the U.S. (for example). In fact, unless the 1000 fans were selected at random from the 70,000 fans at the game, you couldn't even generalize from this sample to "people attending the national championship football game."   

If not external validity, why are large samples important? It's about accuracy of our statistical estimates. When estimating values in the population such as means or differences between means, large samples are less likely to be influenced by chance variability. For example, imagine you're estimating the mean height of kindergarteners in your local school. Now imagine that you select 5 kindergarteners at random, one of whom, by chance, turns out to be extremely tall for her age. That tall kindergartener is going to "pull" the mean estimate upwards when combined with only 4 other kids. But what if you select 25 kindergarteners instead? Now the tall kindergartener is going to be balanced out by 24 other scores, and her height will have less influence on the mean estimate. 

Below is a pair of animations that illustrate this principle. They come from the data science blog R Explorations. The animation used the program R to run a simulation study over and over and over. First, they created a very large population of scores whose mean was known to be 10.0 and whose standard deviation was known to be 1.0. Then they asked the computer to draw a random sample of size 10, compute the mean of the 10 scores, and plot them. You can watch the samples appear in real time on the animation below. Here, xbar is the sample's mean and s is the sample's standard deviation. The red line represents the mean for each sample as it is drawn: 

Questions

a) First, watch the top animation, where N = 10. What do you notice about the movement of the vertical red line representing the mean in the top animation? What is it doing, and what does that represent?

b) Now watch the bottom animation, where N = 1000. What do you notice about the movement of the vertical red line representing the mean in this second animation? What is it doing, and what does that represent?

c) What do you notice about the s values of the two animations? Which animation has a steadier estimate of s?

d) Answer this one only if you've had a statistics course: Which of the two animations will have a smaller standard error? How is the standard error represented in the two animations? 

e) Given the behavior of the two animations, explain why a large sample is important for research. 

f) Which validity does sample size best address, if not external validity?

g) Let's tie this concept back to the "replication crisis" (or, as some are now calling it, "credibility revolution"*). When a finding in psychology has not replicated in a direct replication study, one reason might be that the original study used a small sample. Another reason might be that the replication study used a small sample. Why might the sample size of a study be linked to its replicability? Explain in your own words. 

Both animations come from the R Explorations blog

*Thanks to Dr. Simine Vazire for showing us this animation at NITOP conference and for introducing me to the term "credibility revolution"

 

Start next semester with a fake-news lesson

Credit: IFLA.org (Open source image)

Most research methods instructors hope their course will teach students to be better consumers of information. They want to not only help students read empirical journals; they also want to help students become critical thinkers about anything they encounter in the "real world" of the Internet. 

Maybe you'd like to start next semester with a few outside readings on spotting fake news.  If so, here are some resources you . might use.  I got these from Morton Ann Gernsbacher's wonderful online, open-access methods course, which focuses on identifying and critically reading online news. Check it out for other wonderful resources.

1. The International Federation of Library Associations (IFLA) has a tip list for spotting fake news-- featured in the image above. 
2. Facebook prepared a list of ideas for spotting fake news. 
3. WNYC's On the Media customizes their "Breaking News Consumers Handbook" for most major news stories. Here's one for fake news in general. 
4. And here's a column in Psychology Today about a study that tested how we might inoculate people against fake news about climate science, by exposing them to a small amount of fake news in controlled conditions. The original empirical article is open-access, too. (With six experimental conditions, it's probably not a good study for students to start the semester with)
5. Here's another approach, called SIFT (better than the old CRAAP test).  It asks you to Stop, Investigate the source, Find better coverage, and Trace claims. 

Students can respond to these readings by answering questions such as:

a) Are any of the tips for spotting fake news on all three of the lists? 

b) What are three new ideas that you learned from these sources? What are three ideas they suggested here, which you already do?

c) If a friend shared a story that you suspect is fake news, what might you say to the friend to discourage him or her?

d) How might our political values make us more or less susceptible to thinking critically about fake news stories? 

 

Online resource: Video about random sampling

Pew Research is my favorite polling resource, partly because they ask such interesting questions, and partly because they are so transparent about sharing their methodology. (For examples, see their Methods page or click on the full Report Materials for a study they did on gun ownership in America.) They make their sampling techniques and question wording easily available.

Now Pew has shared a video that explains how a sample of 1,000 can be used to draw inferences about the population. Instructors: Save this one for when you teach Chapter 7!