Everyday Research Methods

Frequency claims: Fitter and better rested during lockdown

One of the positive changes people reported was being able to spend more time with family. Photo: Steve Skjold / Alamy Stock Photo

The first peer-reviewed studies about the COVID-19 pandemic  are starting to appear in journals, and journalists are starting to summarize them in the popular press.

Let's explore this summary of a survey study conducted in Scotland and summarized for a general audience by the scientist herself, in The Conversation. The scientist writes, 

 ... for some people lockdowns have provided an unexpected opportunity to make positive changes to their lives...

Here are some details: 

In May 2020, my colleagues and I surveyed over 3,000 people in Scotland to find out what positive changes people had made in their lives during the lockdown period. 

And some of the key results:

More than half the people we surveyed reported these changes for the better: being more appreciative of things usually taken for granted (reported by 83% of participants), having more time to do enjoyable things (by 67%), spending more time in nature or outdoors (by 65%), paying more attention to personal health (by 62%), doing more physical activity (by 54%) and spending more time with a partner or spouse (by 53%).

Each of the above results constitutes a frequency claim, because it talks about the level, or percentage, associated with one variable. Consider this one, for example: "doing more physical activity (by 54%)". The variable here is "doing more physical activity under the lockdown (or not)."

Now let's use the four big validities to systematically interrogate this frequency claim, "54% of people in this Scottish survey reported doing more physical activity during the lockdown." 

a) Construct validity: How do you think they measured the variable "doing more physical activity"? (Self-report? Observational? Physiological?) . What are your initial impressions of, or questions about, how well this variable was measured? 

b) Construct validity: Let's say you wanted to establish the criterion validity (See Ch 5) of the (presumably self-report) measure of "doing more physical activity".  What behavior should self-reports of this variable correlate with, if the self-report measure is valid? 

c) Statistical validity: The point estimate, 54%, is one element of statistical validity. What about the precision of this estimate? That is, what is the 95% CI, or margin of error of this estimate? It's not provided in the article, but you can visit this calculator to estimate the CI. Do that now. 

d) Statistical validity:  Replication is another aspect of statistical validity. The scientist-author cites at least one more, similar study that came to a similar conclusion: 

An Australian study (still in preprint, meaning its findings are yet to be reviewed by other scientists) also sought to find out similar information. In a survey of over 1,000 people, it found that 70% of participants reported having experienced at least one positive effect of the pandemic. 

e) External validity:  The sample for this survey was N =3000. Don't get distracted by the size of the sample (remember--it's not the sample size that helps establish external validity). Instead, what do you need to ask in order to decide whether the study's sample can generalize to their population of interest (Consult Ch 7)?

f) External validity: I visited the peer-reviewed article (here--open access) and found this information about their sampling technique.

The target population of the survey was adults, aged 18 years or older, currently residing in Scotland, who were interested in sharing their experience of positive change. Participants were primarily recruited through social media advertisements on Facebook and Twitter which directed participants to the online survey on Qualtrics. 

What kind of sampling did they apparently use in this study? (Name a sampling technique that is discussed in Chapter 7). 

What is their population of interest?

Does the sampling technique they used allow them to generalize from their sample to their population of interest? Why or why not? 

g) Internal validity: In a frequency claim, nobody is making a causal statement, so we don't actually need to interrogate internal validity.

h) Now that you've looked at all four validities, what do you think about the quality of this study? Make a nuanced comment about its strengths and weaknesses. 

 

P.S. Thanks to Marianne Lloyd for the reference!

Fidget spinners seem to cause worse memory and attention

The fidget spinner IV was manipulated as independent groups in the first study and as within groups in the second. Photo: Marcel Jancovic/Shutterstock

Fidget spinners were extremely popular a couple of years ago. When people promoted them, they often suggested that the toys could help people with ADHD focus and learn more. At the time, such claims were not scrutinized empirically (here's a nice summary of this lack of evidence). It takes time for empirical research to be conducted, analyzed, peer-reviewed, and published, and even though fidget spinners were hot in 2017, we now have a study that tested the effect of using a fidget spinner during a learning task. At least one press outlet covered this study. Here's their summary:

In the first experiment, 98 undergraduate students watched a 15-minute educational video lecture on baking bread while either using a fidget spinner or not using a fidget spinner [randomly assigned]. In addition, half of the participants not using a fidget spinner were randomly assigned to watch the lecture while sitting near someone using a fidget spinner.

A fill-in-the blank memory test showed that participants who used fidget spinners answered significantly fewer questions correctly about the material covered in the video. Participants who used fidget spinners were also more likely to report “zoning out” and having difficulty staying on task while watching the video.

Participants who were sitting near someone using a fidget spinner, however, did not appear to be affected.

Questions

a) This was an experiment. How do you know?

b) What were the IV and DV? How many levels were in the IV and what were they? Was the IV manipulated as independent groups or within groups? 

c) Which of the four simple experiments is this: postttest-only? Pretest-posttest? Repeated-measures? Concurrent measures? 

d) The journalist attached a causal claim to these results, saying "study finds using a fidget spinner can impair memory and attention." Apply the three criteria for causation to this claim to explain why the causal claim is probably valid. How does random assignment assist this study's internal validity?

e) Ask one question each about the study's statistical, construct, and external validities. 

The researchers replicated the effect in a second study. The wondered if fidget spinners might only work among people who believed in their powers: 

The researchers then conducted the experiment again. However, this time around they specifically recruited 48 undergraduates who believed that fidget spinners could help them focus in class.

But the results were the same.

According to the empirical journal article, the second study on 48 students was conducted as a repeated measures design, using two videos. Students used a fidget spinner during one video and no spinner during the other one. Students were tested on their recall for the videos after each one.

f) What are the IV and DV in the second study? How many levels were there in the IV?

g) How do you think the researchers used counterbalancing in this second study?  Why would that have been important? 

You can access the original empirical article here, though you may need to log in via your university library. 

 

 

 

 

 

How many have Internet access? Question wording matters

CSP_rasstock/agefotostock

What percentage of Americans would you guess have access to the Internet?  How would you know?

When you consider whether people have computers at home, computers at work, smart phones, ortablets, you might predict that a large majority of Americans are able to go online. In fact, it's probably around 90%.  But, as with all frequency claims, the answer to the question hinges on two key things: Construct validity (How well do you measure "access the Internet?") and external validity, ("how well does your sample represent the population?").

In this article, Pew Research explains how they get different estimates depending on how they ask. They write, 

While our approach has evolved over time, the current measurement consists of two questions: “Do you use the internet or email, at least occasionally?” and “Do you access the internet on a cellphone, tablet or other mobile handheld device, at least occasionally?” In a January 2018 phone survey, 89% of U.S. adults said “yes” to at least one of these questions.

They compare their own questions to that of the U.S. Census Bureau's American Community Survey (ACS):

The ACS asks, “At this house, apartment, or mobile home – do you or any member of this household have access to the internet?”

Yes, by paying a cell phone company or Internet service provider
Yes, without paying a cell phone company or Internet service provider
No access to the Internet at this house, apartment, or mobile home. 

This question led to a similar estimate:

88% of American adults had either free or paid Internet access at home.

But what about this question, from the Bureau of Labor Statistics CPS data: 

That survey asks, “Does anyone in this household, including you, use the internet at home?” Based on this question alone, 81% of American adults live in households where someone uses the internet at home. The difference between the two figures can be at least partly attributed to the questions’ wording and scope....  And households can have internet access without anyone actually using it – for example, if it is provided for free by their community or included in a bundle with other services.

The CPS separately asks, “Does anyone in this household access the internet using a data plan for a cellphone, smartphone, tablet, mobile hotspot, or other device? This type of internet service is provided by a wireless carrier, and may be part of a package that also includes voice calls from a cellphone or smartphone.” If we include Americans who say yes to this question, the previous 81% figure rises to 83%.

And what about using the Internet somewhere other than home? 

The CPS also asks a battery of questions about whether anyone in the household uses the internet at work; at school; at a coffee shop or other business that offers internet access; while traveling between places; at a library, community center, park or other public place; at someone else’s home; or at some other location. (Each location is asked about separately.) If we once again expand the previous CPS estimate to include people who may not use the internet at home but do so somewhere else, the estimate ticks up to 85%.

Questions

a) What companies or organizations might be most interested in having an accurate answer to the question of how many Americans are online? 

b) Make a table where you have the wording of a question, followed by the estimate of the number of Americans on the Internet. There seem to be at least four different ways of asking. (You can see trends over time by scrolling down to the bottom of the Pew article.)

c) How important, in your view, are the different ways of asking the question? That is, some might say that all the estimates are virtually the same. But others might say that differences of 1 or 2% percentages matters. What do you think--are the differences in these estimates important or not? 

d) What subgroups of Americans would you estimate are least likely to access the Internet? 

 

 

Observing how people talk to their dogs

What's she saying to her dog? What category would her statement fall into?
Photo Credit: Shift Drive/Shutterstock

This entertaining article quotes several cute examples of how people talk to their dogs while they are out in public. The author, Alexandra Horowitz, is a psychologist who studies dog cognition. She has been recording the snippets of owner-dog conversation she hears as she goes throughout her day. (Most of those doing the talking are the owners.) She shares examples such as "You’re so cute and so smart. And worth money! I could marry you" and "You’ve got a lot to learn! A lot to learn!” and "Don’t even think about it.”

The main body of the article shares the author's categorization system. After jotting down dozens of dog-directed human comments, the author was able to classify them into categories. One category, for example, she titled Cheering Squad, for comments in which the owner is congratulating the dog's behavior. She dubbed another category Mom Commentary, described as, "Eyes on the dog, she sees everything. And she’s gotta talk about it." Another is Perfectly Implausible Instructions, in which owners direct the dog to do impossible things. And there are two other categories in the article: "We've Discussed This" and "Rhetorical Realm".

Read the article to find examples of each of these categories. The author's system provides a clear example of the observational research practices from Chapter 6. The author observed a large set of behaviors. Then she developed a categorization system in order to simplify what she observed.

1. This piece is meant to be entertaining. But if the author were to formally study how people talk to their dogs, the next step would involve coming up with clear operational definitions of each of the three categories. Take a look at the article and try your hand at developing operational definitions of Cheering Squad, Mom Commentary, and Perfectly Implausible Instructions. Write clear instructions that would enable a coder to decide which comments go in only one category (and not in the others).
   
   
2. Test out your operational definitions in the real world. Go out and find some people with dogs, and record some of your own snippets of conversation. Classify each snippet into the categories using your operational definitions, keeping in mind you'll probably need an "other" category. 
   
   
3. A good categorization system requires inter-rater reliability. How would you establish inter-rater reliability for your system? (Hint: You'll need to involve another rater who uses the same categories to categorize the same pieces of conversation.
   
   
4. Any researcher will tell you how difficult it is to develop a reliable coding system. What issues did you have when you tried this out yourself? 

Which Ice Cream Flavor Are You?

Many flavors, many personalities: How could we test whether an ice cream personality test is reliable and valid? Photo credit: Okanmetin/Getty Images

At the height of summer, we all start considering the same, timeless question: Can your personality can be accurately summarized by an ice cream flavor? This Buzzfeed poll asks you to answer a series of summer-activity-related questions and then delivers a verdict. Go take it yourself! I'll wait. 

The first time I took the poll, I answered the questions seriously and was told that I am characterized as "Coffee," along with this description: 

You’re a go-getter, a hard worker who doesn’t pause to reflect on what you have accomplished in the past because you are so focused on what you want to accomplish in the future. You seek to inspire the people around you and help to propel them to greater achievements. At the same time, you are approachable and kind rather than snotty and entitled. It’s basically the best of all worlds.

I sure felt good reading that personality description! The second time I took the quiz, I clicked on questions randomly haphazardly, just to see what other feedback there might be. This time I got "Chocolate", with this description:  

Mmm, you’ve got a truly rich and decadent side that you aren’t afraid to share. You make people feel good — really good. You’ve got a knack for excelling in social situations and know just how to be a crowd-pleaser. Keep doing your thing and living that luscious life.

Of course, nobody is going to mistake a Buzzfeed poll for a legitimate personality measure. But it does provide a fun way to consider issues raised in Chapters 5 and 6. Take the quiz yourself and then prepare these questions:

 

1.     What flavor ice cream did you get? What did the results say about you?

 

2.     Do you think this quiz is reliable? What kind of reliability is relevant for this measure? How would you empirically establish the quiz's reliability? Here's a hint: You'd start with a large sample of people and have them take this ice cream quiz, and then..... (what?)

3.  Now think about the measurement validity of this quiz. I'll narrow down the question for you here: Are people who score in the Coffee category actually more likely to be "hardworking," "inspiring," and "approachable," as the description suggests? How would you establish the criterion validity of this quiz? Here's a hint: You'd start with a large sample of people and have them take the ice cream quiz. You'd separate them into people who got the Coffee category and those who did not get Coffee. How should the Coffees be different from the non-coffees?

 

4.     What might we change about the quiz if we wanted to study ice cream preferences in a scientific way? 

 

5.  Read this description on the Barnum effect. Reflect on how the Barnum effect might be at play for people who take this Buzzfeed quiz. 

 

Many thanks for this idea to Josh Maxwell of the University of New Mexico!

 

Suggested answers to selected questions.

2. Test-retest reliability is the most relevant. To establish it, you'd have a large sample of people take the Buzzfeed quiz at one time point and record everyone's category (e.g., Coffee, Chocolate, etc.) Then you'd track down the same group of people again some time later (maybe 4 weeks later) and have them take the same quiz again. People who were scored Coffee at Time 1 should also score Coffee at Time 2, and people who were scored as Chocolate at Time 1 should also score as Chocolate at Time 2. (You'd compute the correlation between these two category assignments using Cohen's Kappa.)

3. To establish this criterion validity, you'd recruit a large sample of people and have them take the quiz, recording those who got Coffee and those who did not. Then you'd need to measure variables such as "hardworking" "inspiring" and "approachable." I'm not sure how to measure "inspiring" or "hardworking," but you could probably measure "approachable" using the Big Five trait of Agreeableness (here's one measure of that). If the Buzzfeed poll is valid, then a simple bar graph (and an independent groups t-test) should show that people who are assigned Coffee should score higher on agreeableness than people who were assigned some other flavor. 

 

 

Which brand of car is the most reliable?

How might advertisers operationalize the concept of "reliability" in ways favorable to their own brand?  Credit: BNFWork/Shutterstock

You may have seen these ads on TV: Real people stand in a warehouse and are asked which cars are "most reliable." A Honda driver might predict, "Hondas are the most reliable." But then the cover is whisked off in a flourish: The most reliable car is actually the Chevy! In one recent ad (viewable here), the narrator reports: 

"Based on a recent nationwide survey, Chevy is more reliable than Toyota, Honda, and Ford."

Should statements like these convince you? The website Jalopnik publishes journalism about cars. In this column, a journalist walks readers through how they might think critically about  reliability claims in the Chevy commercials. For example, the journalist writes that the reliability claim was based on a survey.

...the survey was sent in 2018 to owners of 2015 model year cars, and the reporting of the “repairs in the past 12 months” relates to those 2015 cars in their third year of service. Chevy reiterated that “Independent statisticians reviewed the materials and concluded with 95% confidence that Chevy’s percentage of no parts repaired or replaced is better than Toyota, Honda, Ford or 23 other brands.”

The journalist adds:

Anyone who has studied research and statistics will tell you that it’s not difficult to form the conclusions you want to based on your use of what is called an “operational definition.” For this particular study, Ipsos is operationally defining reliability as the repairs a car has had within a 12 month period once the car has reached three years old.

a) How did Chevy operationalize "reliability" in their study? 

b) What are some alternative ways to operationalize a car brand's "reliability?"

c) Can you think of downsides to the operationalization that Chevy used? 

The journalist pointed out that, other than the operationalization of reliability, the survey's number of respondents might matter. To wit:

Setting aside the fact that self-reported surveys can be flawed, fewer than 49,000 respondents completed them out of almost 840,000. What Chevrolet did not provide is a breakdown of those respondents by brand, because if we were looking at the percentage of repairs made by various brand owners this could make a difference.

For example, if only 1,000 of those 49,000 respondents were Honda owners and 100 of them reported repairs within that 12 month period, that means that Honda would have a 10 percent repair rate. But if 15,000 of the respondents were Chevrolet owners and 1,200 of them reported repairs, that is an 8 percent repair rate. Yet this distribution was not provided so it’s difficult to ascertain exactly how those percentages compare within each sample size of brand owners.

d) What do you think of the journalist's reasoning, above? 

In another argument, the journalist compares the self-report survey used by Chevy with two other sources of reliability data, Consumer Reports and J.D. Power: 

...in a 2018 ranking of brand reliability, Consumer Reports ranked both Honda and Toyota well above Chevrolet, [which] occupied the lower part of the list along with several other domestic brands.

But like Chevrolet’s survey, this data is also self-reported based on the ownership experiences of Consumer Reports subscribers.

I also reached out to J.D. Power, as they are a respected reference when it comes to these kinds of “quality” rankings and asked them to comment on the methodology behind Chevrolet’s study. They declined. (Recall that Chevy has touted J.D. Power awards in this very same series of ads.)

e) Why might the journalist extol Consumer Reports or J.D. Power over Chevy's Ipsos survey? What might these organizations doing differently--if anything--in their measurement of reliability? 

f) Reflect on how the availability heuristic (the pop-up principle), cherry picking, bias blind spots, or the present-present bias (all from Chapter 2) might lead individuals to make faulty conclusions about which car brands are most reliable.

 

Thanks to Marianne Lloyd for sharing this idea! 

 

How much do chimpanzees cooperate?

Which behaviors were used to operationalize cooperation in chimps?
Photo credit: apple2499/Shutterstock

Human beings are a fundamentally prosocial and cooperative species, but what about one of our close relatives, the chimpanzee?  A science writer summarized a series of studies on chimpanzees who live in the Tchimpounga Chimpanzee Sanctuary, in the Republic of the Congo. The summary is here and you can read the empirical journal article here.

The studies illustrate how researchers can operationalize a variable like "cooperativeness" in other species. It provides a clear example of observational research, too. 

Researcher Alexandra Rosati conducted three observational tasks on the sanctuary, all meant to operationalize the construct cooperativeness. The first task took place with chimp volunteers who were alone in a dormitory area. One experimenter grabbed another one's stick and threw it out of reach into the dormitory. The second researcher made a show of trying to get the stick back through the bars of the dormitory. Would the chimp help her out by bringing her the stick? The researcher conducted 10 trials with each chimp. You can see the task in action at this link--watch Video 2. The journalist writes:

On average, the chimpanzees, who received no reward for cooperating, chose to hand the out-of-reach object to the researcher just over 50% of the time – but the rate of cooperation varied significantly among the individuals, with some helping in all 10 trials and others not at all. Chimps who chose to help more often also chose to help more quickly, supporting the conclusion that more cooperative individuals tend to make decisions faster than more selfish individuals do, the authors wrote.

Notice that there are two findings here. The first is the simple result of how many chimps helped (that's a frequency claim result).The second result is a correlational pattern. 

a.) What is the frequency claim you can make here? 

b.) Reread the description of the "stick" study (above) and decide what two variables are correlated.

c.) Placing these two variables on the x and y axis, sketch a scatterplot of the result that is described. (Compare your sketch to Figure 2a in the empirical journal article.) 

d.) Comment on the operationalization of the construct "Cooperation". Why might the researchers have chosen this operationalization? 

Here's a second way they studied chimp cooperation. This operationalization asks whether the chimps would punish individuals who don't follow rules:

When they had an opportunity to punish a thief who stole food from them, the chimps often chose to collapse the experimenter’s table [by pulling out the table's false leg] to prevent the thief from consuming the ill-gotten treats. 

e.) You can watch the "stealing" task here, under Video 3. What behavior (by the chimps) was used to operationalize "cooperativeness" here?  Why might this behavior be considered as an operationalization of Cooperation?

Finally, another task used in the study presented chimpanzees with two options for food. This task is viewable in Video 1 here: 

When presented with two sets of plates, one that contained a snack for both the chimpanzee and a familiar caretaker and another that had a treat only for the chimp, the apes were basically indifferent to whether the experimenter got something to eat. 

f.) The three operationalizations of cooperation in chimps found variable results. Two operationalizations showed some "cooperation," and one apparently did not. What does this pattern of results suggest to you about the importance of how well we operationalize variables? 

 

 

Question wording matters

Collecting accurate data in a poll is difficult business. Many of us focus on the sampling strategies of polling organizations, and rightly so: External validity depends on whether we include "cell phone only" numbers in our samples, how we account for different rates of responding across certain groups, and so on. However, this post reminds us that question wording also matters. Construct validity is just as important for polls.

In this report, Pew Research Center describes how people's opinions change depending on how a question is posed. The poll asked Americans whether we should increase the size of the House of Representatives. People in the poll were randomly assigned to hear either the original question, or the same question with additional context. Notice how people's support changes when additional context is added: 

“The Electoral College, Congress, and representation,” Pew Research Center, Washington, D.C. (April 26, 2018). https://pewrsr.ch/2rLd0OI, accessed on May 16, 2018

 

a) Even though this is a polling example, Pew conducted an experiment. What are its independent and dependent variables? 

b) What causal claim could this polling result support? 

c) In your opinion, which polling question--the original one or the one with added context--provides the most accurate measure of people's opinion? 

d) Find another poll question on the Pew Research web site. Speculate how some additional information might change people's responses. 

 

 

 

 

 

 

 

Below is another presentation of the same results. Notice how the two  different question wordings seem to affect people differently, depending on their political affiliation.

“The Electoral College, Congress, and representation,” Pew Research Center, Washington, D.C. (April 26, 2018). https://pewrsr.ch/2rLd0OI, accessed on May 16, 2018

 

e) Why might Democrats be more influenced by the contextual information provided? 

f) This could be viewed as a 2 x 2, IV x PV factorial design, in which the IV is "question wording" (original or added context) and the PV is "political affiliation" (Democrat or Republican). If you've studied Chapter 12, perhaps you can identify whether there are main effects and interactions in this pattern of data. 

The same Pew report contains additional examples of how question wording alters responses to a question about the Senate, and shows how different subgroups respond to questions about the Electoral College. 

Online resource: Writing good questions

Here's a second video in a series by Pew Research. This 5 minute clip describes some of the issues in writing good questions for an opinion poll. It basically summarizes the first part of Chapter 6, and provides some new, concrete examples. 

You can use this in combination with Pew's first video about random sampling, which I described here.

How fast do people talk? Depends on where you live....

How fast do people talk? How wordy are they? Do they swear a lot? Photo credit: Eugenio Marongiu/Shutterstock

There's a new piece in the Atlantic summarizing some research on phone calls. The journalist's piece is tantalizing, but it doesn't provide some of the information you need to evaluate the research behind it.

According to the article, a data analytics firm analyzed a bunch of recorded phone calls--about 4 million of them--from customer service interactions. According to the journalist, the firm counted how fast people talked, how many words they used, and even how long they would wait on hold before hanging up! The journalist reported:

In some sense, [the] findings hew to cultural stereotypes. The fast-talkers are concentrated in the North; the slow-talkers are concentrated in the South.

Specifically, the fastest talkers were found in Oregon, Minnesota, Massachusetts, Kansas, and Iowa. The slowest were in North Carolina, Alabama, South Carolina, Louisiana, and Mississippi.

The Atlantic piece includes some colorful maps of the USA, indicating the fast, slow, and medium-talking states. Check them out!

a) While you're there, see if you can figure out how they defined "speed of talking" in this research. Also see if you can find out how much faster the Oregonians talk, compared to the Mississippians (how big is the effect? Is it statistically significant?). Before playing up these regional differences in a magazine article, this seems like important stuff to know, right?

The Atlantic piece goes on to discuss how "wordy" the phone calls were. Here the results were different. The journalist explains:

...speedy speech doesn’t necessarily equate to dense speech. [The analytic form] also used its dataset to analyze the wordiest speakers, state by state—the callers who, regardless of their tempo, used the most words during their interactions with customer service agents.

and also:

Some of the slower-talking states (Texas, New Mexico, Virginia, etc.) are also some of the wordier, suggesting a premium on connection over efficiency. Some of the fastest-talking states (Idaho, Wyoming, New Hampshire) are also some of the least talkative, suggesting the get-down-to-business mentality commonly associated with those states.

b) Can you find out, from the article, how much wordier the "wordy" states are compared to the less wordy states? How big are the effects? 

c) And can you figure out from the Atlantic article exactly how the researchers operationalized "being wordy" and "talking fast?"  How are these two variables different, exactly? (I couldn't find it from the article--let me know if you can!)

I'm being critical of the journalist's coverage here, but I'm also keeping in mind that it might not be her fault. The source of the data is not a peer-reviewed journal article (which would have been required to present this information). Instead, the source was a report by the for-profit analytics firm that had collected the data. Perhaps this firm did not report enough methodological and statistical details to allow careful reporting by the journalist.

However, I checked the website of the analytic firm, There, you can find the information you need. Their report mentions exactly how the two variables were operationalized and gave effect sizes of the results.

Specifically, you learn that talking speed was operationalized as "words per minute". And the differences between states were described this way:

For every 5 words a slow talking state utters, a fast talking state will utter 6.

You also learn that wordiness was operationalized as "total words in a phone conversation." and: 

How big is the difference? A New Yorker will use 62% more words than someone from Iowa to have the same conversation with a business,

d) What do you think of these operationalizations of talking speed and wordiness? And what do you think of the magnitudes of the effects? Are these large differences--worthy of a news story on differences between states? If you were the journalist, would you have incorporated this information, or not?

You might also be interested in a second Atlantic story about which states are the "sweariest." Compared to the story on talking fast (and being wordy), this story is more informative. For one, it describes how swearing was operationalized. They

...examin[ed] more than 600,000 phone calls from the past 12 months—calls placed by consumers to businesses across 30 different industries. It then used call mining technology to isolate the curses therein, cross-referencing them against the state the calls were placed from.

The journalist also opined, (probably correctly):

...cursing is also conveniently specific as a data set; you've got your f-bombs and your double hockey sticks and your bodily functions, and, factoring in their permutations, you're good to go. Plus, you don't need much sophisticated sentiment analysis to ensure that your data are accurate: An f-bomb is pretty much an f-bomb, regardless of the contextual subtleties.

In the swearing story, the journalist also described the data better, explaining how the rates of swearing in the highest-swearing state and the lowest state compare:

People in Ohio cursed the most as compared to every other state in the Union: They swore in one out of about every 150 phone conversations. Ohio was followed, respectively, by Maryland, New Jersey, Louisiana, and Illinois.

And who swore the least? Washingtonians. They cursed, on average, during one out of every 300 conversations. (Yes, this means that Ohioans swear at more than twice the rate of Washingtonians....)

d) Now you have a better picture of the data. What do you think of the sizes of these effects? Big or small? Important, or not?

e) The headline of this story announces that Ohio "is the sweariest state in the Union." What do you think of that headline? Specifically, does that headline overstate the results? Does it overgeneralize from the measures they used?

Meow! "Viewing cat videos boosts energy and mood" Really?

This respondent said that watching cat videos causes her mood to improve. What do you think? Photo: Rasulovs/Thinkstock.

Here's a provocative headline from Medical News Today:  Viewing cat videos boosts energy and positive emotions. That sounds like exciting news to me: The causal claim implies that we should watch some of those cute kitty videos to feel better!

Let's take a look at the study behind the headline. The journalist's story covers a recent publication in the journal, Computers in Human Behavior. As the  study's author told the journalist,

"We all have watched a cat video online, but there is really little empirical work done on why so many of us do this, or what effects it might have on us," added Myrick, who owns a pug but no cats. "As a media researcher and online cat video viewer, I felt compelled to gather some data about this pop culture phenomenon."

So far, so good. Empiricism is a great way to answer questions about pop culture phenomena. So what did they do?

The study, by assistant professor Jessica Gall Myrick, surveyed almost 7,000 people about their viewing of cat videos and how it affects their moods. It was published in the latest issue of Computers in Human Behavior. Lil Bub's owner, Mike Bridavsky, who lives in Bloomington, helped distribute the survey via social media.

A survey, then. Here are some of the results:

Participants in Myrick's study reported that:

• They were more energetic and felt more positive after watching cat-related online media than before
• The pleasure they got from watching cat videos outweighed any guilt they felt about procrastinating
• About 25 percent of the cat videos they watched were sought out; the rest were ones they happened upon

Time to analyze this causal claim.

a) What are the variables in the journalist's causal headline, "viewing cat videos boosts energy and mood?"

b) What kind of a study does it take to establish this causality? How might you have designed such a study? 

c) What kind of study seems to have been conducted here? Do the results support the causal claim? If not, what would a more appropriate headline be for a story on this study?

Finally, the journalist mentions that LilBub's owner advertised the study on his website, and that for each person who took the survey, he donated money to the APSCA, which supports animal welfare. Over 7000 people completed it. 

d) One result from the study is that "about 36 percent described themselves as a cat person, while about 60 percent said they like both cats and dogs." Can we assume from the results and the methodology that 96% of people are animal (either cat or dog) lovers? 

If you would like to read the published empirical study, you can download it here.