Everyday Research Methods

Frequency claims: Fitter and better rested during lockdown

One of the positive changes people reported was being able to spend more time with family. Photo: Steve Skjold / Alamy Stock Photo

The first peer-reviewed studies about the COVID-19 pandemic  are starting to appear in journals, and journalists are starting to summarize them in the popular press.

Let's explore this summary of a survey study conducted in Scotland and summarized for a general audience by the scientist herself, in The Conversation. The scientist writes, 

 ... for some people lockdowns have provided an unexpected opportunity to make positive changes to their lives...

Here are some details: 

In May 2020, my colleagues and I surveyed over 3,000 people in Scotland to find out what positive changes people had made in their lives during the lockdown period. 

And some of the key results:

More than half the people we surveyed reported these changes for the better: being more appreciative of things usually taken for granted (reported by 83% of participants), having more time to do enjoyable things (by 67%), spending more time in nature or outdoors (by 65%), paying more attention to personal health (by 62%), doing more physical activity (by 54%) and spending more time with a partner or spouse (by 53%).

Each of the above results constitutes a frequency claim, because it talks about the level, or percentage, associated with one variable. Consider this one, for example: "doing more physical activity (by 54%)". The variable here is "doing more physical activity under the lockdown (or not)."

Now let's use the four big validities to systematically interrogate this frequency claim, "54% of people in this Scottish survey reported doing more physical activity during the lockdown." 

a) Construct validity: How do you think they measured the variable "doing more physical activity"? (Self-report? Observational? Physiological?) . What are your initial impressions of, or questions about, how well this variable was measured? 

b) Construct validity: Let's say you wanted to establish the criterion validity (See Ch 5) of the (presumably self-report) measure of "doing more physical activity".  What behavior should self-reports of this variable correlate with, if the self-report measure is valid? 

c) Statistical validity: The point estimate, 54%, is one element of statistical validity. What about the precision of this estimate? That is, what is the 95% CI, or margin of error of this estimate? It's not provided in the article, but you can visit this calculator to estimate the CI. Do that now. 

d) Statistical validity:  Replication is another aspect of statistical validity. The scientist-author cites at least one more, similar study that came to a similar conclusion: 

An Australian study (still in preprint, meaning its findings are yet to be reviewed by other scientists) also sought to find out similar information. In a survey of over 1,000 people, it found that 70% of participants reported having experienced at least one positive effect of the pandemic. 

e) External validity:  The sample for this survey was N =3000. Don't get distracted by the size of the sample (remember--it's not the sample size that helps establish external validity). Instead, what do you need to ask in order to decide whether the study's sample can generalize to their population of interest (Consult Ch 7)?

f) External validity: I visited the peer-reviewed article (here--open access) and found this information about their sampling technique.

The target population of the survey was adults, aged 18 years or older, currently residing in Scotland, who were interested in sharing their experience of positive change. Participants were primarily recruited through social media advertisements on Facebook and Twitter which directed participants to the online survey on Qualtrics. 

What kind of sampling did they apparently use in this study? (Name a sampling technique that is discussed in Chapter 7). 

What is their population of interest?

Does the sampling technique they used allow them to generalize from their sample to their population of interest? Why or why not? 

g) Internal validity: In a frequency claim, nobody is making a causal statement, so we don't actually need to interrogate internal validity.

h) Now that you've looked at all four validities, what do you think about the quality of this study? Make a nuanced comment about its strengths and weaknesses. 

 

P.S. Thanks to Marianne Lloyd for the reference!

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers. 

 

Question wording matters

Collecting accurate data in a poll is difficult business. Many of us focus on the sampling strategies of polling organizations, and rightly so: External validity depends on whether we include "cell phone only" numbers in our samples, how we account for different rates of responding across certain groups, and so on. However, this post reminds us that question wording also matters. Construct validity is just as important for polls.

In this report, Pew Research Center describes how people's opinions change depending on how a question is posed. The poll asked Americans whether we should increase the size of the House of Representatives. People in the poll were randomly assigned to hear either the original question, or the same question with additional context. Notice how people's support changes when additional context is added: 

“The Electoral College, Congress, and representation,” Pew Research Center, Washington, D.C. (April 26, 2018). https://pewrsr.ch/2rLd0OI, accessed on May 16, 2018

 

a) Even though this is a polling example, Pew conducted an experiment. What are its independent and dependent variables? 

b) What causal claim could this polling result support? 

c) In your opinion, which polling question--the original one or the one with added context--provides the most accurate measure of people's opinion? 

d) Find another poll question on the Pew Research web site. Speculate how some additional information might change people's responses. 

 

 

 

 

 

 

 

Below is another presentation of the same results. Notice how the two  different question wordings seem to affect people differently, depending on their political affiliation.

“The Electoral College, Congress, and representation,” Pew Research Center, Washington, D.C. (April 26, 2018). https://pewrsr.ch/2rLd0OI, accessed on May 16, 2018

 

e) Why might Democrats be more influenced by the contextual information provided? 

f) This could be viewed as a 2 x 2, IV x PV factorial design, in which the IV is "question wording" (original or added context) and the PV is "political affiliation" (Democrat or Republican). If you've studied Chapter 12, perhaps you can identify whether there are main effects and interactions in this pattern of data. 

The same Pew report contains additional examples of how question wording alters responses to a question about the Senate, and shows how different subgroups respond to questions about the Electoral College. 

Why strive for large samples?

When the general public critiques research, I often hear them say that the samples are "too small."  It's true that sample sizes (N) in psychology research should be large.  One of the outcomes of the so-called "replication crisis" is that large samples are more and more important in psychology. But why?  

A common misconception--held by both students and the general public--is that large samples are important because they ensure external validity. This misconception is incorrect. External validity (that is, the ability to generalize from a sample to a population of interest) is about how a sample has been recruited, not how many people are in it (see Chapter 7, 14). For example, say you recruited a sample of 1000 fans attending the national championship college game. You'd have a pretty large sample, but you couldn't generalize from that sample to college students in the U.S. (for example). In fact, unless the 1000 fans were selected at random from the 70,000 fans at the game, you couldn't even generalize from this sample to "people attending the national championship football game."   

If not external validity, why are large samples important? It's about accuracy of our statistical estimates. When estimating values in the population such as means or differences between means, large samples are less likely to be influenced by chance variability. For example, imagine you're estimating the mean height of kindergarteners in your local school. Now imagine that you select 5 kindergarteners at random, one of whom, by chance, turns out to be extremely tall for her age. That tall kindergartener is going to "pull" the mean estimate upwards when combined with only 4 other kids. But what if you select 25 kindergarteners instead? Now the tall kindergartener is going to be balanced out by 24 other scores, and her height will have less influence on the mean estimate. 

Below is a pair of animations that illustrate this principle. They come from the data science blog R Explorations. The animation used the program R to run a simulation study over and over and over. First, they created a very large population of scores whose mean was known to be 10.0 and whose standard deviation was known to be 1.0. Then they asked the computer to draw a random sample of size 10, compute the mean of the 10 scores, and plot them. You can watch the samples appear in real time on the animation below. Here, xbar is the sample's mean and s is the sample's standard deviation. The red line represents the mean for each sample as it is drawn: 

Questions

a) First, watch the top animation, where N = 10. What do you notice about the movement of the vertical red line representing the mean in the top animation? What is it doing, and what does that represent?

b) Now watch the bottom animation, where N = 1000. What do you notice about the movement of the vertical red line representing the mean in this second animation? What is it doing, and what does that represent?

c) What do you notice about the s values of the two animations? Which animation has a steadier estimate of s?

d) Answer this one only if you've had a statistics course: Which of the two animations will have a smaller standard error? How is the standard error represented in the two animations? 

e) Given the behavior of the two animations, explain why a large sample is important for research. 

f) Which validity does sample size best address, if not external validity?

g) Let's tie this concept back to the "replication crisis" (or, as some are now calling it, "credibility revolution"*). When a finding in psychology has not replicated in a direct replication study, one reason might be that the original study used a small sample. Another reason might be that the replication study used a small sample. Why might the sample size of a study be linked to its replicability? Explain in your own words. 

Both animations come from the R Explorations blog

*Thanks to Dr. Simine Vazire for showing us this animation at NITOP conference and for introducing me to the term "credibility revolution"

 

Online resource: Video about random sampling

Pew Research is my favorite polling resource, partly because they ask such interesting questions, and partly because they are so transparent about sharing their methodology. (For examples, see their Methods page or click on the full Report Materials for a study they did on gun ownership in America.) They make their sampling techniques and question wording easily available.

Now Pew has shared a video that explains how a sample of 1,000 can be used to draw inferences about the population. Instructors: Save this one for when you teach Chapter 7!

 

Many students don't know when news is fake

The researchers asked students if this element probably linked to real news or fake news.What's the clue that tells you this story is probably not "real news"?

 

Fake news is in the (real) news lately. Whether you're looking at Facebook, Buzzfeed, or your online newspaper, companies may try to clickbait you into reading a story that's false. Companies may want you to read the story so that you'll be exposed to their advertising. Or a political group may want to persuade you of an extreme opinion. In some recent cases, people have read fake news stories, believed them, and then acted according to what they thought was true (here's an example).

How often do people mistake fake news for real news?  

A team at Stanford University recently attempted to measure the problem in a large sample of high school students. The results of their study were summarized by the Wall Street Journal. The journalist from the WSJ reported the following: 

...82% of middle-schoolers couldn’t distinguish between an ad labeled “sponsored content” and a real news story on a website, according to a Stanford University study of 7,804 students from middle school through college. The study, set for release Tuesday, is the biggest so far on how teens evaluate information they find online. 

The study apparently showed students several examples, asking them for each one if it was a real story or fake news. You'll see an example of one of their study's stimuli in the photo to the left. You can see the other samples in the full report from Stanford's website (scroll to p. 9). 

Here are some more results, reported by the WSJ: 

More than two out of three middle-schoolers couldn’t see any valid reason to mistrust a post written by a bank executive arguing that young adults need more financial-planning help. And nearly four in 10 high-school students believed, based on the headline, that a photo of deformed daisies on a photo-sharing site provided strong evidence of toxic conditions near the Fukushima Daiichi nuclear plant in Japan, even though no source or location was given for the photo.

a) What kind of claim is it to say that "82% of middle-schoolers couldn’t distinguish between an ad labeled “sponsored content” and a real news story" (Frequency, association, or cause?)  What is (are) the variable(s) in the claim?

b) In order to claim that "82% of middle schoolers" do something, you'd probably need to be sure that the study included a generalizable sample of middle schoolers. What are some ways the researchers could have obtained an externally valid sample?

c)  For a frequency claim like this one, construct validity is also important. The construct validity of the Stanford study seems excellent, because the researchers asked students questions about realistic-looking mockups of online content. Reading back through the green quotes above, you'll see three  different ways they measured the variable, "knowing when news is fake." What are the three ways?

 

I can't help but point out that in your research methods class, you will learn several media literacy skills. You're learning that journalists might not always get the details of a scientific study right--they might not even read the original article! Journalists might slap a causal claim on a correlational study. Or they might write a sensational study about a single study without reviewing the entire literature on a topic. Being a good consumer of information means you'll be able to critically evaluate media stories about science (and other topics, too). 

 

What ends up on the Yelp Top 100 list?

What construct and external validity considerations affect Yelp ratings, like this 4-star review of a restaurant near my campus? Screenshot from Yelp website (by author)

In this interesting piece, Slate writer Will Oremus asks why the top-rated restaurants on Yelp are places that "nobody has ever heard of."  He explains that the top 10 rated restaruants on Yelp change from year to year, and furthermore, they tend to be places like Copper Top BBQ of CA and Art of Flavors of Las Vegas.  The top 100 are not big, famous restaurants--in fact, they tend to be simple, local, even touristy places with "styrofoam and paper plates."

Oremus points out that Yelp ratings are subject to "biases are quite different from the ones we’re used to" in other ratings websites.  Are these biases of the external validity variety? Or the construct validity variety?  I'll quote some passages from Oremus' Slate article for your consideration.

Here's Oremus's first observation:

I have not eaten at Copper Top, but I have little doubt that these restaurants also share a consistently high quality of food for the money. On Yelp, that’s usually a recipe for a four-star rating. Compared to professional critics, Yelp reviewers skew young and budget-conscious, which is part of the site’s appeal. By and large, they’re happier paying $8 for a very good burrito than $23 for a fancy one, and the ratings reflect that.

a.  What kind of validity is this conclusion directed at--external or construct? Can you say anything specific about this type of validity, as used by this journalist? 

Here's another observation from Oremus' article.

Part of the explanation lies in the distribution of ratings on the site’s five-star scale. Only a handful or restaurants in the world rate three Michelin stars. But more than 40 percent of all Yelp reviews are perfect scores, suggesting that five stars on Yelp entails satisfaction rather than perfection. Average hundreds of reviews of the same establishment, and you’ll find that its overall rating is influenced far more by the number of dissatisfied customers than by how much the five-star reviewers loved it. The best-rated restaurants on Yelp, then, are not so much the most loved as the least hated.

b.  What kind of validity is this conclusion directed at--external or construct? Can you say anything specific about this type of validity, as used by this journalist?

Oremus also points out the power of incidental influences, such as neighborhood and weather, on Yelp reviews. To wit:

...researchers at Georgia Tech and Yahoo Labs found that online restaurant reviews are significantly influenced by at least three factors that have nothing to do with the operation of the business:

• Neighborhood demographics: Restaurants in neighborhoods with high education levels don’t get better reviews, but they do get more reviews. That matters, because Yelp’s top-100 rankings are based not only on average ratings, ...so a place with 100 five-star reviews will rank higher than one with 50.
• Time of year: Restaurants get more reviews in July and August than they do in the winter, but the average ratings in the summer months are lower.
• Weather: One of the strongest exogenous effects on restaurant ratings, according to the study, is the weather at the time of the review. As you might guess, warm temperatures and sunshine mean higher reviews. Cold temperatures or extreme heat mean lower reviews, as does precipitation of any kind. The researchers attribute this to weather’s well-documented effects on mood and memory.

c. What kind of validity is the above research concerned with?

d.  How might this information affect your own use of Yelp in the future?  Think of two possible ways you can use this information.

 

Suggested Answers

a. When Oremus writes that Yelp's users tend to be younger and budget-conscious, he's describing how the sample of people who choose to post on Yelp is biased. This is an external validity point. One could say that the sample of Yelp reviews is self-selected, and is biased toward restaurants that are cheaper.  Cheaper restaurants may get more reviews on Yelp than more expensive ones; in addition, cheaper restaurants may be rated more positively, all because of this sampling bias.  Therefore, ratings on Yelp might not generalize to how older people would evaluate the same restaurants. The situation seems similar to conducting an opinion poll and including (or not) cell-phone only households.

b. This is a construct validity point. People tend to use the 5-star end of the Yelp scale the most, he says. This means that people have a "yea saying" bias on Yelp reviews (to use a Chapter 6 term). As a result, it might be hard to decide if a positive review on Yelp is truly good, or if people just tend to like everything! 

Another point Oremus made is that a high Yelp rating means the restaurant is more consistent--not necessarily more delicious. That again is a construct validity point.

c. The weather bias suggests problems with construct validity. We might not know if a positive rating reflects the quality of the restaurant (the construct in question) or the type of weather outside!  However, the "neighborhood bias" seems to be an external validity issue--restaurants in highly educated neighborhoods get more reviews, so this is a sampling bias. 

 d.  Answers will vary. All in all, it seems that these construct and external validity issues mean that Yelp reviews are unlikely to parallel what a professional restaurant critic would say. Does that affect your restaurant behavior, or not?  

 

Thanks again to Carrie Smith of Ole Miss, who, as usual, is a fount of bloggable Slate pieces!

 

 

 

 

Cell phones only--whom should polls call?

My favorite polling organization, Pew Research, has reported that people have become more and more likely to have "cell phones only," with no landline phones.  Here is a graph showing the percentage of American adults who have cell-phone only households:

Scott Keeter and Kyley McGeeney. “Pew Research will call more cellphones in 2015.” Pew Research Center, Washington, D.C. (January 7, 2015). http://www.pewresearch.org/fact-tank/2015/01/07/pew-research-will-call-more-cellphones-in-2015/ , accessed on January 8, 2015.

Questions

a) This figure provides a good opportunity to interpret a graph. Study the figure carefully. The figure seems to be illustrating two main points--one about time, and one about age. What are these two points?

Here's another figure from the story:

Scott Keeter and Kyley McGeeney. “Pew Research will call more cellphones in 2015.” Pew Research Center, Washington, D.C. (January 7, 2015). http://www.pewresearch.org/fact-tank/2015/01/07/pew-research-will-call-more-cellphones-in-2015/ , accessed on January 8, 2015.

b) This figure, as well, seems to be illustrating two main points. What are the two main takeaway messages from this figure? (Hint: One takeaway message is the same here as in Question a.)

Given the data presented in this figure, Pew Research reports that it will call a higher proportion of cell-phone only households this year. Here is a quote from their website:

To keep pace with this rapid trend, the Pew Research Center will increase the percentage of respondents interviewed on cellphones in its typical national telephone surveys to 65%; 35% of interviews will be conducted by landline. Last year, we increased the ratio to 60% cellphone, with 40% conducted on landline. Back in 2008, when we first started routinely including cellphones in our phone surveys, just one-fourth (25%) of all interviews were done by cellphone.

c) Remind yourself that external validity (through generalizable sampling techniques) is especially important for frequency claims.  Give two or three examples of research questions that fit this kind of claim.

d) Explain why it is important for polling organizations to know which households are cell-phone only and which are not.  What specific kinds of political and social questions might be especially affected by cell-phone only proportions, given the data presented above? 

e) Here's an interesting question: How might an organization like Pew Research obtain an accurate estimate of the number of cell-phone only households in the first place?  What kind of sample would you need to get this estimate? How would you contact this sample?

Finally, here's an interesting description from Pew Research, discussing whether it would be smart to conduct polls only on cell phones, not land lines:

The question naturally arises: Why not interview everyone on a cellphone? In fact, at least one major national survey is going to do just that. The Surveys of Consumers, conducted by the University of Michigan, will begin calling only cellphones this month.

But we are not ready to make that change just yet, for at least two reasons. One is that there remains a small share of the public that is not reachable by cellphone. In the newly released data from the National Center for Health Statistics, 7% of adults live in households with a landline phone but no wireless phone. In addition, some people with landlines and cellphones may turn their cellphones on only to make calls or when they are expecting to be called. If these kinds of people are demographically different from those who are more easily reached on a cellphone, then the resulting sample will be less representative of the full population.

Thanks, Pew Research, for your transparency! 

 

Recent Poll on Americans and Fast Food

Gallup released the results of some polling questions on Americans' use of fast food. According to the poll, most Americans eat fast food once a month or more. Click over to the poll results to see how many Americans eat fast food at least once per week.

Despite eating fast food fairly often, Americans do not think it is good for you.

76% in the U.S. think the food served in fast-food restaurants is "not too good" or "not good at all for you," the same percentage who said so in 2003.

Gallup's analysis of their polling data shows the percentage of different ethnic and gender groups that report eating fast food regularly.For example, they broke the results down by income, reporting that:

But fast food is hardly the province solely of those with lower incomes; in fact, wealthier Americans -- those earning $75,000 a year or more -- are more likely to eat it at least weekly (51%) than are lower-income groups. Those earning the least actually are the least likely to eat fast food weekly -- 39% of Americans earning less than $20,000 a year do so.

When you analyze the methodological strengths of a set of data like these, you should focus mainly on the construct and external validity of the study.

a) What questions could you ask about the construct validity of this poll?  What do you think about the construct validity of the different questions?

b) Do you think the people they called all think about "fast food" in the same way? Think about what you'd call fast food (Does the local pizza place count? Does Subway? Does Burger King?) Then ask a couple of friends what they think fast food is. Do you all agree?  What might that mean about the results of Gallup's survey?

c) What questions could you ask about the external validity of the poll? Gallup includes a section on Survey Methods on the bottom of the story. Read this section--you  may not understand all of it, but see what you can. What do you think about the sampling? What kind of sampling did they use? Can they use their sample to generalize to the population of Americans? Why or why not?

d) There were 2,027 people in the Gallup sample. Is this enough?

 

 

 

How Do Young Parents Feel about New Media?

This recent piece in USNews describes the results of a poll about how young parents feel about their kids' use of smartphones and tablets.

Let's use this story to practice what you are learning about surveys and polls.

Here's one excerpt from the journalist's story.

Surveying more than 2,300 parents of children up to age 8, researchers from Northwestern University found that the vast majority -- 78 percent -- report that their children's media use is not a source of family conflict, and 59 percent said they aren't concerned their kids will become addicted to new media.

a. Will this sample of 2300 parents be able to generalize to American parents overall?  What do you need to know?  (and, what kind of validity are we asking about here?)

b. Based on the report above, what kind of question do you think the researchers asked--a forced choice question? An open ended one?

Here's another excerpt:

"We asked parents what their challenges were as the parents of young children . . . and sometimes media was never mentioned," said study author Ellen Wartella, director of Northwestern's Center on Media and Human Development. "Parents of children this age are concerned about their health, safety, nutrition and exercise, and media concerns are much lower down the list. That was a surprise."

c. Based on this information, do you think the "challenges" question was closed ended? open ended? Do you think they used a construct valid way to ask this kind of question?

 

Finally, here's one more part of the story:

The notion that parents are apt to shush their kids by handing them a smartphone or tablet also appears to be false, according to results. To keep their children quietly occupied, moms and dads said they were more apt to turn to toys or activities (88 percent), books (79 percent) or TV (78 percent). Of parents with smartphones or iPads, only 37 percent reported being somewhat or very likely to turn to those devices.

A reader of this news article submitted the following comment:

Parents never tell the truth in situations like this. Either they hate to admit that they shut little junior up by putting him in front of a phone, or they don't even realize how much they do it.
Look around you in restaurants and just watch how many parents with small children let them sit there playing with mom's phone or a Nintendo DS.

d. Which validity is the reader criticizing?  What do you think of the reader's comment?

Suggested answers:

a. 2300 is probably plenty--as long as it is a sample that was obtained via some probability sampling method (such as random sampling or cluster sampling), 2300 is an adequate sample size. Remember--when it comes to external validity, it's not how many are in your sample, it's how you got it.

b.We can't be sure without checking, but I suspect this was a closed ended question, something like, "is your child's media use a source of family conflict?" or "to what extent is your child's media use a source of family conflict?" 

c. In this case, it seems clear that the question was open-ended, such as, "what are your challenges as a parent?"or, "Tell us about some challenges you face as a parent. If you are interested in measuring what parents think about first when it comes to parenting challengies, an open-ended question might be the best. If you asked them, "is media use your biggest challenge?" you might get more parents to say "yes" than if you simply ask them, "what are your biggest challenges?"

d. This reader is criticing the study's construct validity. While it is probably very easy and efficient to ask parents to self-report their ways of calming their kids, the reader has a good point, in my opinion. Parents probably do not want to admit that they quiet their children with a smartphone or tablet. A better way to find out what parents do to keep kids quiet is by visiting a few restaurants and observing what children are doing. Can you design a study that would do this? How can you be sure your study has good external and construct validity?

 

 

Malnutrition statistics in North Korea

This Wall Street Journal article describes how a team of U.S. and European professors traveled to North Korea to teach students there about quantitative methods for studying their population.

North Korea, the story says, has "some of the least reliable statistics in the world."

Please take a look at the pop-out map of North Korea in this story.  The map shows the estimated rates of child malnutrition in each of North Korea's provinces. For example, it shows that the estimated rate of child malnutrition in the province of Jagong was 9.8%. That is a frequency claim: a claim about a single variable--the rate of malnutrition--in each province.

The caption on the map is worth attending to from a research methods perspective. The caption reads,

Researchers in North Korea often face challenges. In this 2012 study of child malnutrition, conducted with the support of three U.N. agencies, local village leaders chose the children from which researchers drew their subjects, so they could have excluded the most malnourished kids.

This first part of the caption describes a potential problem with the external validity of this claim. If officials did in fact select only the most healthy kids for the sampling frame, then the estimate of the true rate of child malnutrition would be too low.

Here's the second part of the map's caption:

And the data were collected at the end of the harvest, possibly producing a temporary uptick in nutrition levels.

You could say that this part of the caption is describing a potential problem with the construct validity of this claim. By measuring child malnutrition during a time of plenty, officials did not get a valid measure of the true health of the kids in the sample.

What lessons do you think the instructors are teaching to the students in North Korea?