Everyday Research Methods

Frequency claims: Fitter and better rested during lockdown

One of the positive changes people reported was being able to spend more time with family. Photo: Steve Skjold / Alamy Stock Photo

The first peer-reviewed studies about the COVID-19 pandemic  are starting to appear in journals, and journalists are starting to summarize them in the popular press.

Let's explore this summary of a survey study conducted in Scotland and summarized for a general audience by the scientist herself, in The Conversation. The scientist writes, 

 ... for some people lockdowns have provided an unexpected opportunity to make positive changes to their lives...

Here are some details: 

In May 2020, my colleagues and I surveyed over 3,000 people in Scotland to find out what positive changes people had made in their lives during the lockdown period. 

And some of the key results:

More than half the people we surveyed reported these changes for the better: being more appreciative of things usually taken for granted (reported by 83% of participants), having more time to do enjoyable things (by 67%), spending more time in nature or outdoors (by 65%), paying more attention to personal health (by 62%), doing more physical activity (by 54%) and spending more time with a partner or spouse (by 53%).

Each of the above results constitutes a frequency claim, because it talks about the level, or percentage, associated with one variable. Consider this one, for example: "doing more physical activity (by 54%)". The variable here is "doing more physical activity under the lockdown (or not)."

Now let's use the four big validities to systematically interrogate this frequency claim, "54% of people in this Scottish survey reported doing more physical activity during the lockdown." 

a) Construct validity: How do you think they measured the variable "doing more physical activity"? (Self-report? Observational? Physiological?) . What are your initial impressions of, or questions about, how well this variable was measured? 

b) Construct validity: Let's say you wanted to establish the criterion validity (See Ch 5) of the (presumably self-report) measure of "doing more physical activity".  What behavior should self-reports of this variable correlate with, if the self-report measure is valid? 

c) Statistical validity: The point estimate, 54%, is one element of statistical validity. What about the precision of this estimate? That is, what is the 95% CI, or margin of error of this estimate? It's not provided in the article, but you can visit this calculator to estimate the CI. Do that now. 

d) Statistical validity:  Replication is another aspect of statistical validity. The scientist-author cites at least one more, similar study that came to a similar conclusion: 

An Australian study (still in preprint, meaning its findings are yet to be reviewed by other scientists) also sought to find out similar information. In a survey of over 1,000 people, it found that 70% of participants reported having experienced at least one positive effect of the pandemic. 

e) External validity:  The sample for this survey was N =3000. Don't get distracted by the size of the sample (remember--it's not the sample size that helps establish external validity). Instead, what do you need to ask in order to decide whether the study's sample can generalize to their population of interest (Consult Ch 7)?

f) External validity: I visited the peer-reviewed article (here--open access) and found this information about their sampling technique.

The target population of the survey was adults, aged 18 years or older, currently residing in Scotland, who were interested in sharing their experience of positive change. Participants were primarily recruited through social media advertisements on Facebook and Twitter which directed participants to the online survey on Qualtrics. 

What kind of sampling did they apparently use in this study? (Name a sampling technique that is discussed in Chapter 7). 

What is their population of interest?

Does the sampling technique they used allow them to generalize from their sample to their population of interest? Why or why not? 

g) Internal validity: In a frequency claim, nobody is making a causal statement, so we don't actually need to interrogate internal validity.

h) Now that you've looked at all four validities, what do you think about the quality of this study? Make a nuanced comment about its strengths and weaknesses. 

 

P.S. Thanks to Marianne Lloyd for the reference!

Many students don't know when news is fake

The researchers asked students if this element probably linked to real news or fake news.What's the clue that tells you this story is probably not "real news"?

 

Fake news is in the (real) news lately. Whether you're looking at Facebook, Buzzfeed, or your online newspaper, companies may try to clickbait you into reading a story that's false. Companies may want you to read the story so that you'll be exposed to their advertising. Or a political group may want to persuade you of an extreme opinion. In some recent cases, people have read fake news stories, believed them, and then acted according to what they thought was true (here's an example).

How often do people mistake fake news for real news?  

A team at Stanford University recently attempted to measure the problem in a large sample of high school students. The results of their study were summarized by the Wall Street Journal. The journalist from the WSJ reported the following: 

...82% of middle-schoolers couldn’t distinguish between an ad labeled “sponsored content” and a real news story on a website, according to a Stanford University study of 7,804 students from middle school through college. The study, set for release Tuesday, is the biggest so far on how teens evaluate information they find online. 

The study apparently showed students several examples, asking them for each one if it was a real story or fake news. You'll see an example of one of their study's stimuli in the photo to the left. You can see the other samples in the full report from Stanford's website (scroll to p. 9). 

Here are some more results, reported by the WSJ: 

More than two out of three middle-schoolers couldn’t see any valid reason to mistrust a post written by a bank executive arguing that young adults need more financial-planning help. And nearly four in 10 high-school students believed, based on the headline, that a photo of deformed daisies on a photo-sharing site provided strong evidence of toxic conditions near the Fukushima Daiichi nuclear plant in Japan, even though no source or location was given for the photo.

a) What kind of claim is it to say that "82% of middle-schoolers couldn’t distinguish between an ad labeled “sponsored content” and a real news story" (Frequency, association, or cause?)  What is (are) the variable(s) in the claim?

b) In order to claim that "82% of middle schoolers" do something, you'd probably need to be sure that the study included a generalizable sample of middle schoolers. What are some ways the researchers could have obtained an externally valid sample?

c)  For a frequency claim like this one, construct validity is also important. The construct validity of the Stanford study seems excellent, because the researchers asked students questions about realistic-looking mockups of online content. Reading back through the green quotes above, you'll see three  different ways they measured the variable, "knowing when news is fake." What are the three ways?

 

I can't help but point out that in your research methods class, you will learn several media literacy skills. You're learning that journalists might not always get the details of a scientific study right--they might not even read the original article! Journalists might slap a causal claim on a correlational study. Or they might write a sensational study about a single study without reviewing the entire literature on a topic. Being a good consumer of information means you'll be able to critically evaluate media stories about science (and other topics, too). 

 

What ends up on the Yelp Top 100 list?

What construct and external validity considerations affect Yelp ratings, like this 4-star review of a restaurant near my campus? Screenshot from Yelp website (by author)

In this interesting piece, Slate writer Will Oremus asks why the top-rated restaurants on Yelp are places that "nobody has ever heard of."  He explains that the top 10 rated restaruants on Yelp change from year to year, and furthermore, they tend to be places like Copper Top BBQ of CA and Art of Flavors of Las Vegas.  The top 100 are not big, famous restaurants--in fact, they tend to be simple, local, even touristy places with "styrofoam and paper plates."

Oremus points out that Yelp ratings are subject to "biases are quite different from the ones we’re used to" in other ratings websites.  Are these biases of the external validity variety? Or the construct validity variety?  I'll quote some passages from Oremus' Slate article for your consideration.

Here's Oremus's first observation:

I have not eaten at Copper Top, but I have little doubt that these restaurants also share a consistently high quality of food for the money. On Yelp, that’s usually a recipe for a four-star rating. Compared to professional critics, Yelp reviewers skew young and budget-conscious, which is part of the site’s appeal. By and large, they’re happier paying $8 for a very good burrito than $23 for a fancy one, and the ratings reflect that.

a.  What kind of validity is this conclusion directed at--external or construct? Can you say anything specific about this type of validity, as used by this journalist? 

Here's another observation from Oremus' article.

Part of the explanation lies in the distribution of ratings on the site’s five-star scale. Only a handful or restaurants in the world rate three Michelin stars. But more than 40 percent of all Yelp reviews are perfect scores, suggesting that five stars on Yelp entails satisfaction rather than perfection. Average hundreds of reviews of the same establishment, and you’ll find that its overall rating is influenced far more by the number of dissatisfied customers than by how much the five-star reviewers loved it. The best-rated restaurants on Yelp, then, are not so much the most loved as the least hated.

b.  What kind of validity is this conclusion directed at--external or construct? Can you say anything specific about this type of validity, as used by this journalist?

Oremus also points out the power of incidental influences, such as neighborhood and weather, on Yelp reviews. To wit:

...researchers at Georgia Tech and Yahoo Labs found that online restaurant reviews are significantly influenced by at least three factors that have nothing to do with the operation of the business:

• Neighborhood demographics: Restaurants in neighborhoods with high education levels don’t get better reviews, but they do get more reviews. That matters, because Yelp’s top-100 rankings are based not only on average ratings, ...so a place with 100 five-star reviews will rank higher than one with 50.
• Time of year: Restaurants get more reviews in July and August than they do in the winter, but the average ratings in the summer months are lower.
• Weather: One of the strongest exogenous effects on restaurant ratings, according to the study, is the weather at the time of the review. As you might guess, warm temperatures and sunshine mean higher reviews. Cold temperatures or extreme heat mean lower reviews, as does precipitation of any kind. The researchers attribute this to weather’s well-documented effects on mood and memory.

c. What kind of validity is the above research concerned with?

d.  How might this information affect your own use of Yelp in the future?  Think of two possible ways you can use this information.

 

Suggested Answers

a. When Oremus writes that Yelp's users tend to be younger and budget-conscious, he's describing how the sample of people who choose to post on Yelp is biased. This is an external validity point. One could say that the sample of Yelp reviews is self-selected, and is biased toward restaurants that are cheaper.  Cheaper restaurants may get more reviews on Yelp than more expensive ones; in addition, cheaper restaurants may be rated more positively, all because of this sampling bias.  Therefore, ratings on Yelp might not generalize to how older people would evaluate the same restaurants. The situation seems similar to conducting an opinion poll and including (or not) cell-phone only households.

b. This is a construct validity point. People tend to use the 5-star end of the Yelp scale the most, he says. This means that people have a "yea saying" bias on Yelp reviews (to use a Chapter 6 term). As a result, it might be hard to decide if a positive review on Yelp is truly good, or if people just tend to like everything! 

Another point Oremus made is that a high Yelp rating means the restaurant is more consistent--not necessarily more delicious. That again is a construct validity point.

c. The weather bias suggests problems with construct validity. We might not know if a positive rating reflects the quality of the restaurant (the construct in question) or the type of weather outside!  However, the "neighborhood bias" seems to be an external validity issue--restaurants in highly educated neighborhoods get more reviews, so this is a sampling bias. 

 d.  Answers will vary. All in all, it seems that these construct and external validity issues mean that Yelp reviews are unlikely to parallel what a professional restaurant critic would say. Does that affect your restaurant behavior, or not?  

 

Thanks again to Carrie Smith of Ole Miss, who, as usual, is a fount of bloggable Slate pieces!

 

 

 

 

Cell phones only--whom should polls call?

My favorite polling organization, Pew Research, has reported that people have become more and more likely to have "cell phones only," with no landline phones.  Here is a graph showing the percentage of American adults who have cell-phone only households:

Scott Keeter and Kyley McGeeney. “Pew Research will call more cellphones in 2015.” Pew Research Center, Washington, D.C. (January 7, 2015). http://www.pewresearch.org/fact-tank/2015/01/07/pew-research-will-call-more-cellphones-in-2015/ , accessed on January 8, 2015.

Questions

a) This figure provides a good opportunity to interpret a graph. Study the figure carefully. The figure seems to be illustrating two main points--one about time, and one about age. What are these two points?

Here's another figure from the story:

Scott Keeter and Kyley McGeeney. “Pew Research will call more cellphones in 2015.” Pew Research Center, Washington, D.C. (January 7, 2015). http://www.pewresearch.org/fact-tank/2015/01/07/pew-research-will-call-more-cellphones-in-2015/ , accessed on January 8, 2015.

b) This figure, as well, seems to be illustrating two main points. What are the two main takeaway messages from this figure? (Hint: One takeaway message is the same here as in Question a.)

Given the data presented in this figure, Pew Research reports that it will call a higher proportion of cell-phone only households this year. Here is a quote from their website:

To keep pace with this rapid trend, the Pew Research Center will increase the percentage of respondents interviewed on cellphones in its typical national telephone surveys to 65%; 35% of interviews will be conducted by landline. Last year, we increased the ratio to 60% cellphone, with 40% conducted on landline. Back in 2008, when we first started routinely including cellphones in our phone surveys, just one-fourth (25%) of all interviews were done by cellphone.

c) Remind yourself that external validity (through generalizable sampling techniques) is especially important for frequency claims.  Give two or three examples of research questions that fit this kind of claim.

d) Explain why it is important for polling organizations to know which households are cell-phone only and which are not.  What specific kinds of political and social questions might be especially affected by cell-phone only proportions, given the data presented above? 

e) Here's an interesting question: How might an organization like Pew Research obtain an accurate estimate of the number of cell-phone only households in the first place?  What kind of sample would you need to get this estimate? How would you contact this sample?

Finally, here's an interesting description from Pew Research, discussing whether it would be smart to conduct polls only on cell phones, not land lines:

The question naturally arises: Why not interview everyone on a cellphone? In fact, at least one major national survey is going to do just that. The Surveys of Consumers, conducted by the University of Michigan, will begin calling only cellphones this month.

But we are not ready to make that change just yet, for at least two reasons. One is that there remains a small share of the public that is not reachable by cellphone. In the newly released data from the National Center for Health Statistics, 7% of adults live in households with a landline phone but no wireless phone. In addition, some people with landlines and cellphones may turn their cellphones on only to make calls or when they are expecting to be called. If these kinds of people are demographically different from those who are more easily reached on a cellphone, then the resulting sample will be less representative of the full population.

Thanks, Pew Research, for your transparency! 

 

Recent Poll on Americans and Fast Food

Gallup released the results of some polling questions on Americans' use of fast food. According to the poll, most Americans eat fast food once a month or more. Click over to the poll results to see how many Americans eat fast food at least once per week.

Despite eating fast food fairly often, Americans do not think it is good for you.

76% in the U.S. think the food served in fast-food restaurants is "not too good" or "not good at all for you," the same percentage who said so in 2003.

Gallup's analysis of their polling data shows the percentage of different ethnic and gender groups that report eating fast food regularly.For example, they broke the results down by income, reporting that:

But fast food is hardly the province solely of those with lower incomes; in fact, wealthier Americans -- those earning $75,000 a year or more -- are more likely to eat it at least weekly (51%) than are lower-income groups. Those earning the least actually are the least likely to eat fast food weekly -- 39% of Americans earning less than $20,000 a year do so.

When you analyze the methodological strengths of a set of data like these, you should focus mainly on the construct and external validity of the study.

a) What questions could you ask about the construct validity of this poll?  What do you think about the construct validity of the different questions?

b) Do you think the people they called all think about "fast food" in the same way? Think about what you'd call fast food (Does the local pizza place count? Does Subway? Does Burger King?) Then ask a couple of friends what they think fast food is. Do you all agree?  What might that mean about the results of Gallup's survey?

c) What questions could you ask about the external validity of the poll? Gallup includes a section on Survey Methods on the bottom of the story. Read this section--you  may not understand all of it, but see what you can. What do you think about the sampling? What kind of sampling did they use? Can they use their sample to generalize to the population of Americans? Why or why not?

d) There were 2,027 people in the Gallup sample. Is this enough?

 

 

 

How Do Young Parents Feel about New Media?

This recent piece in USNews describes the results of a poll about how young parents feel about their kids' use of smartphones and tablets.

Let's use this story to practice what you are learning about surveys and polls.

Here's one excerpt from the journalist's story.

Surveying more than 2,300 parents of children up to age 8, researchers from Northwestern University found that the vast majority -- 78 percent -- report that their children's media use is not a source of family conflict, and 59 percent said they aren't concerned their kids will become addicted to new media.

a. Will this sample of 2300 parents be able to generalize to American parents overall?  What do you need to know?  (and, what kind of validity are we asking about here?)

b. Based on the report above, what kind of question do you think the researchers asked--a forced choice question? An open ended one?

Here's another excerpt:

"We asked parents what their challenges were as the parents of young children . . . and sometimes media was never mentioned," said study author Ellen Wartella, director of Northwestern's Center on Media and Human Development. "Parents of children this age are concerned about their health, safety, nutrition and exercise, and media concerns are much lower down the list. That was a surprise."

c. Based on this information, do you think the "challenges" question was closed ended? open ended? Do you think they used a construct valid way to ask this kind of question?

 

Finally, here's one more part of the story:

The notion that parents are apt to shush their kids by handing them a smartphone or tablet also appears to be false, according to results. To keep their children quietly occupied, moms and dads said they were more apt to turn to toys or activities (88 percent), books (79 percent) or TV (78 percent). Of parents with smartphones or iPads, only 37 percent reported being somewhat or very likely to turn to those devices.

A reader of this news article submitted the following comment:

Parents never tell the truth in situations like this. Either they hate to admit that they shut little junior up by putting him in front of a phone, or they don't even realize how much they do it.
Look around you in restaurants and just watch how many parents with small children let them sit there playing with mom's phone or a Nintendo DS.

d. Which validity is the reader criticizing?  What do you think of the reader's comment?

Suggested answers:

a. 2300 is probably plenty--as long as it is a sample that was obtained via some probability sampling method (such as random sampling or cluster sampling), 2300 is an adequate sample size. Remember--when it comes to external validity, it's not how many are in your sample, it's how you got it.

b.We can't be sure without checking, but I suspect this was a closed ended question, something like, "is your child's media use a source of family conflict?" or "to what extent is your child's media use a source of family conflict?" 

c. In this case, it seems clear that the question was open-ended, such as, "what are your challenges as a parent?"or, "Tell us about some challenges you face as a parent. If you are interested in measuring what parents think about first when it comes to parenting challengies, an open-ended question might be the best. If you asked them, "is media use your biggest challenge?" you might get more parents to say "yes" than if you simply ask them, "what are your biggest challenges?"

d. This reader is criticing the study's construct validity. While it is probably very easy and efficient to ask parents to self-report their ways of calming their kids, the reader has a good point, in my opinion. Parents probably do not want to admit that they quiet their children with a smartphone or tablet. A better way to find out what parents do to keep kids quiet is by visiting a few restaurants and observing what children are doing. Can you design a study that would do this? How can you be sure your study has good external and construct validity?

 

 

Malnutrition statistics in North Korea

This Wall Street Journal article describes how a team of U.S. and European professors traveled to North Korea to teach students there about quantitative methods for studying their population.

North Korea, the story says, has "some of the least reliable statistics in the world."

Please take a look at the pop-out map of North Korea in this story.  The map shows the estimated rates of child malnutrition in each of North Korea's provinces. For example, it shows that the estimated rate of child malnutrition in the province of Jagong was 9.8%. That is a frequency claim: a claim about a single variable--the rate of malnutrition--in each province.

The caption on the map is worth attending to from a research methods perspective. The caption reads,

Researchers in North Korea often face challenges. In this 2012 study of child malnutrition, conducted with the support of three U.N. agencies, local village leaders chose the children from which researchers drew their subjects, so they could have excluded the most malnourished kids.

This first part of the caption describes a potential problem with the external validity of this claim. If officials did in fact select only the most healthy kids for the sampling frame, then the estimate of the true rate of child malnutrition would be too low.

Here's the second part of the map's caption:

And the data were collected at the end of the harvest, possibly producing a temporary uptick in nutrition levels.

You could say that this part of the caption is describing a potential problem with the construct validity of this claim. By measuring child malnutrition during a time of plenty, officials did not get a valid measure of the true health of the kids in the sample.

What lessons do you think the instructors are teaching to the students in North Korea?

 

More than half of US teens text while driving...

This recent story from msnbc.com reports that "nearly 60 percent of teens text while driving." From the story:

An anonymous national survey conducted last year found that 58 percent of high school seniors said they had texted or emailed while driving during the previous month. About 43 percent of high school juniors acknowledged they did the same thing.

Let's interrogate this claim.

a) What kind of claim is the headline making, and which big validities do we care about for a claim like this?

b) The msnbc story gives some detail about the original study's methodology. Does it give you enough detail to assess external validity? If not, what else do you need to know?

c) Does it give you enough detail to assess construct validity? What else might you need to know?

One part of the msnbc article mentions a teen who got into a minor accident while texting behind the wheel:

"I felt like an idiot," said her 18-year-old son, Dylan Young.

"It caused me to be a lot more cautious," said the high school senior, although he conceded that he still texts behind the wheel.

d) Is this story (about Dylan Young) an example of empirical evidence? Why do you think the journalist includes this story? 

 

Report: Internet hyperconnectivity might change young adults

The Pew Research Center recently released a report on the internet's effect on millenials. The report notes that

Teens and young adults brought up from childhood with a continuous connection to each other and to information will be nimble, quick-acting multitaskers who count on the Internet as their external brain and who approach problems in a different way from their elders, according to a new survey of technology experts.

a) Note that this report isn't about what actually will happen to young adults becuase of their hyperconnectivity; it's about what certain people think will happen. According to the quote above, who was polled for this report?

Pew Research Center is helpfully up-front about its sampling techniques. Read this excerpt about their sampling for this particular survey, taken from this page:

The survey results are based on a non-random, opt-in, online sample of 1,021 internet experts and other internet users, recruited via email invitation.

And in case you weren't sure what the implications of that sampling method are, they also add:

Since the data are based on a non-random sample, a margin of error cannot be computed, and the results are not projectable to any population other than the experts in this sample.

b) Among the sampling strategies described in Chapter 6 and summarized in Figure 6.12, what kind of sampling did Pew use?

c) What would Pew have to do differently in order to get a truly representative sample of internet experts?

d) How might you study the true effect of internet hyperconnectivity on young adults? How could you test the hypothesis that they will "solve problems differently," for example?

Most Americans report 2 friends

A report suggests that among Americans, the average number of close friends people report having is just two. You can read an msnbc report of the study here.

The study reports that the researchers

...surveyed more than 2,000 adults ages 18 and older from the nationally representative Time-sharing Experiments for the Social Sciences (TESS) program. The Internet surveys were conducted between April 23 and May 5, 2010.

The story also reports that people were

...asked to list the names of people they had discussed "important matters" with over the previous six months. If respondents said "none," they asked whether this was because they didn't have any important matters to discuss or no one with whom to discuss them in the past six months.About 48 percent of participants listed one name, 18 percent listed two, and roughly 29 percent listed more than two names for these close friends.

The average number of confidants listed was 2.03, the story reports.

 

a.) The headline of the study was, "You gotta have friends? Most have just 2 true pals." What kind of claim is made in this headline?

b.) Which two validities do we prioritize when evaluating this kind of claim?

c.) Based on the information from the article quoted above, how well does the study fulfill these two validities?

Fewer babies on Halloween?

This story reports that pregnant women are less likely to go into spontaneous labor on a negatively toned holiday (Halloween).  Similarly, they are more likely to do so on a positively toned holiday (Valentine's Day).

Here's a quote from the New York Times story:

In a study published this month, however, researchers at the Yale School of Public Health reviewed millions of birth certificates and found a 5.3 percent dip in spontaneous births on Halloween, suggesting that a mother’s mental state may play a role in when she goes into labor.

a.) Is this a frequency claim or an association claim?

b.) This research result might be seen as sort of a cute, public-interest story.  But it also seems to be testing a theory.  What is the theory, and how do the data support it?  

c.) The abstract for the empirical journal article that originally published this result provides a lot of information on the study.  According to the abstract, what was the comparison condition for the study? What was their sampling technique? 

Suggested answers

a.)  This is a tough call!  On the one hand, simply stating that there are fewer spontaneous births on Halloween is a frequency claim.  On the other hand, it would be an association claim if you stated that the positivity of a holiday (Halloween versus Valentine's Day) is associated with the number of spontaneous births. 

b.) According to the abstract, the theory is that health outcomes can be affected by cultural representations and stereotypes.  The data support the theory by using results from two holidays with distinct cultural representations.  Spontaneous births appear to fluctuate with the positivity of these two holidays, supporting the theory.

c.) In this study, the dependent variable is spontaneous birth rate.  The comparison conditions (the independent variable levels) were achieved by comparing birth rates on the target day (Halloween or Valentine's Day) with birth rates during the week before and after the target day. 

According to the abstract, the study appears to have conducted a census--they used all possible U.S. birth certificates for the selected date ranges.