Everyday Research Methods

Observing how people talk to their dogs

What's she saying to her dog? What category would her statement fall into?
Photo Credit: Shift Drive/Shutterstock

This entertaining article quotes several cute examples of how people talk to their dogs while they are out in public. The author, Alexandra Horowitz, is a psychologist who studies dog cognition. She has been recording the snippets of owner-dog conversation she hears as she goes throughout her day. (Most of those doing the talking are the owners.) She shares examples such as "You’re so cute and so smart. And worth money! I could marry you" and "You’ve got a lot to learn! A lot to learn!” and "Don’t even think about it.”

The main body of the article shares the author's categorization system. After jotting down dozens of dog-directed human comments, the author was able to classify them into categories. One category, for example, she titled Cheering Squad, for comments in which the owner is congratulating the dog's behavior. She dubbed another category Mom Commentary, described as, "Eyes on the dog, she sees everything. And she’s gotta talk about it." Another is Perfectly Implausible Instructions, in which owners direct the dog to do impossible things. And there are two other categories in the article: "We've Discussed This" and "Rhetorical Realm".

Read the article to find examples of each of these categories. The author's system provides a clear example of the observational research practices from Chapter 6. The author observed a large set of behaviors. Then she developed a categorization system in order to simplify what she observed.

1. This piece is meant to be entertaining. But if the author were to formally study how people talk to their dogs, the next step would involve coming up with clear operational definitions of each of the three categories. Take a look at the article and try your hand at developing operational definitions of Cheering Squad, Mom Commentary, and Perfectly Implausible Instructions. Write clear instructions that would enable a coder to decide which comments go in only one category (and not in the others).
   
   
2. Test out your operational definitions in the real world. Go out and find some people with dogs, and record some of your own snippets of conversation. Classify each snippet into the categories using your operational definitions, keeping in mind you'll probably need an "other" category. 
   
   
3. A good categorization system requires inter-rater reliability. How would you establish inter-rater reliability for your system? (Hint: You'll need to involve another rater who uses the same categories to categorize the same pieces of conversation.
   
   
4. Any researcher will tell you how difficult it is to develop a reliable coding system. What issues did you have when you tried this out yourself? 

Which Ice Cream Flavor Are You?

Many flavors, many personalities: How could we test whether an ice cream personality test is reliable and valid? Photo credit: Okanmetin/Getty Images

At the height of summer, we all start considering the same, timeless question: Can your personality can be accurately summarized by an ice cream flavor? This Buzzfeed poll asks you to answer a series of summer-activity-related questions and then delivers a verdict. Go take it yourself! I'll wait. 

The first time I took the poll, I answered the questions seriously and was told that I am characterized as "Coffee," along with this description: 

You’re a go-getter, a hard worker who doesn’t pause to reflect on what you have accomplished in the past because you are so focused on what you want to accomplish in the future. You seek to inspire the people around you and help to propel them to greater achievements. At the same time, you are approachable and kind rather than snotty and entitled. It’s basically the best of all worlds.

I sure felt good reading that personality description! The second time I took the quiz, I clicked on questions randomly haphazardly, just to see what other feedback there might be. This time I got "Chocolate", with this description:  

Mmm, you’ve got a truly rich and decadent side that you aren’t afraid to share. You make people feel good — really good. You’ve got a knack for excelling in social situations and know just how to be a crowd-pleaser. Keep doing your thing and living that luscious life.

Of course, nobody is going to mistake a Buzzfeed poll for a legitimate personality measure. But it does provide a fun way to consider issues raised in Chapters 5 and 6. Take the quiz yourself and then prepare these questions:

 

1.     What flavor ice cream did you get? What did the results say about you?

 

2.     Do you think this quiz is reliable? What kind of reliability is relevant for this measure? How would you empirically establish the quiz's reliability? Here's a hint: You'd start with a large sample of people and have them take this ice cream quiz, and then..... (what?)

3.  Now think about the measurement validity of this quiz. I'll narrow down the question for you here: Are people who score in the Coffee category actually more likely to be "hardworking," "inspiring," and "approachable," as the description suggests? How would you establish the criterion validity of this quiz? Here's a hint: You'd start with a large sample of people and have them take the ice cream quiz. You'd separate them into people who got the Coffee category and those who did not get Coffee. How should the Coffees be different from the non-coffees?

 

4.     What might we change about the quiz if we wanted to study ice cream preferences in a scientific way? 

 

5.  Read this description on the Barnum effect. https://openpsychometrics.org/tests/IPIP-BFFM/Reflect on how the Barnum effect might be at play for people who take this Buzzfeed quiz. 

 

Many thanks for this idea to Josh Maxwell of the University of New Mexico!

 

Suggested answers to selected questions.

2. Test-retest reliability is the most relevant. To establish it, you'd have a large sample of people take the Buzzfeed quiz at one time point and record everyone's category (e.g., Coffee, Chocolate, etc.) Then you'd track down the same group of people again some time later (maybe 4 weeks later) and have them take the same quiz again. People who were scored Coffee at Time 1 should also score Coffee at Time 2, and people who were scored as Chocolate at Time 1 should also score as Chocolate at Time 2. (You'd compute the correlation between these two category assignments using Cohen's Kappa.)

3. To establish this criterion validity, you'd recruit a large sample of people and have them take the quiz, recording those who got Coffee and those who did not. Then you'd need to measure variables such as "hardworking" "inspiring" and "approachable." I'm not sure how to measure "inspiring" or "hardworking," but you could probably measure "approachable" using the Big Five trait of Agreeableness (here's one measure of that). If the Buzzfeed poll is valid, then a simple bar graph (and an independent groups t-test) should show that people who are assigned Coffee should score higher on agreeableness than people who were assigned some other flavor. 

 

 

Tilt your head down to appear more intimidating

According to the study, Captain Marvel's eyebrows have something to do with her dominant pose. Credit: Collection Christophel/Alamy Stock Photo

Captain Marvel's looks tough. Perhaps one reason is the tilt of her head, which enhances the appearance of dominance, according to a series of studies that appears in Psychological Science. The studies were summarized by journalist Alice Walton for Forbes. 

Here are some excerpts from the journalist's description of the study:

...there’s a way to communicate dominance that doesn’t involve moving the facial muscles at all: just tilt your head downward slightly and maintain eye contact.

In one experiment, participants in the University of British Columbia study rated [several] avatars who were tilted upward 10 degrees, tilted downward 10 degrees, or level (neutral). Then they rated statements designed to gauge how dominant the person appeared, including “This person would enjoy having control over others,” and “This person would often try to get his way regardless of what people may want.”

Participants rated the downward-tilting heads significantly more dominant than either of the other two positions. When the experiment was repeated using photos of actual people rather than avatars, the same pattern was found.

a) The journalist labels this an experiment, which it really is. What is the independent variable? What are its levels? Is the IV manipulated as independent groups or within-groups? (By the way, the Forbes article shows photos of the three levels.)

b) What is the dependent variable in this study?

c) What's the experimental design: Posttest only? Pretest-posttest? Repeated measures? Concurrent measures? 

d) Sketch a bar graph of the results described here: "Participants rated the downward-tilting heads significantly more dominant than either of the other two positions."

e) What do you think of the construct validity of the dependent variable? Does this operationalization really get at "dominance?" Why or why not? What questions from Chapter 5 might you ask here? 

f) The journalist mentions that "the experiment was repeated using photos of actual people rather than avatars". It's an example of replication. Which type of replication study would you call it: direct replication? conceptual replication, or replication plus extension? (Chapter 14 defines these terms.)

g) By replicating the avatar study on actual people, which validity (or validities)  are these researchers enhancing? 

More Advanced Questions

After establishing the basic effect, the researchers conducted some additional studies to test the mechanism, or mediator. Read this paragraph (written by the journalist) to decide what the mechanism was, and think about how they tested it. 

In another experiment, the team found that the key to the effect was the eyes and eyebrows: when they showed participants faces with the eyes/eyebrows occluded, the tilted head/dominance effect disappeared. But then they showed participants pictures of just the eyes and brow portion of the face, the effect was present, which suggests that this part of the face is both necessary and sufficient—meaning it's not only required but is all that's necessary to elicit the effect.

The above argument suggests that the researchers' theory goes like this:

    Tilting head downward ---> changes in eyebrow position --> dominant appearance

h) To find evidence for their theory, the authors tested four conditions in a 2x2 factorial design. Downward tilt (vs. neutral), and face only vs. eyes/eyebrows only. You can see examples of the stimuli they used in Figure 4 of the original article in Psychological Science. Sketch the predictions of these two studies. Like this:

If changes in eyebrow position are important for tilted faces to appear dominant, then the results should show [sketch a result].

If changes in eyebrow position are NOT important for tilted faces to appear dominant, then the results should show [sketch a result].

 

Of head horns and cell phones

Did the study collect the right data to conclude that "screen time found to cause 'head horns'?"  Photo Credit: Nature Scientific Reports

Maybe you heard the story last week about how cell phones are making horns grow on people's heads. Take a minute to watch this  short video of the morning crew from ABC news discussing it. The link is captioned, "Screen time found to cause head horns" In this video, you'll notice that the journalist announces, "New research shows that constantly looking down at your screen could give you so-called 'head horns'" .

And here's another video that recycles the same material along with extra commentary under the title, "Tech neck causing skull horns"

There's so much to unpack about this story. If you read beyond the first journalists' coverage, you'll find some great analyses of this study and the journalism around it. Turns out, the study itself had problems, and the journalistic coverage didn't help. 

Take a moment to read one of these two critiques of the "head horns" moment: 

Smartphones aren't making millenials grow horns. Here's how to spot a bad study. or Hold the phone: Are you really going to grow a horn from slouching over your phone? 

Use the information in the two critical stories to research the following questions.

First, there are issues related to the "journal to journalism" cycle (Chapter 1), 

a) The original journalist changed the name of the protuberances to "horns." Every journalist thereafter copied this nickname in their own reporting.  What were they originally called? Why do you think the journalist changed their name?

b) In Chapter 1, you learn that journalists don't always pick the most important studies to summarize. Why do you think journalists picked this particular story? Do you think the study is important? 

Second, there are problems with the study itself, which many journalists did not detect--at least at first.

c) One criticism with the study concerns its sample. What are the criticisms raised about this study's sample of 1200 x-rays?

d) Another criticism has to do with the researcher's potential conflicts of interest (CoIs). What were these, and why are CoIs a potential problem in scientific research and publication? 

e) Another criticism has to do with the proposed link between cell phone use and the "horns."  In order to establish a link between having these protuberances and screen time, the scientists would have had to measure both variables and compute the correlation. Did the researchers do this? 

f) Finally, many of the news headlines used causal verbs ("Screen time found to cause head horns"). Even the scientists overclaimed here when they were interviewed by the press. What kind of study would a researcher have to do in order to establish such a causal link? 

g) Are there any additional problems you discovered?

The news cycle around the head horns study illustrates the benefit of cross-reading and triangulating. If you come across a journalist's story that's just too weird to be true, read about the same topic on other sources, or on snopes.com. 

 

Did the show "Thirteen Reasons Why" cause an increase in suicides?

The study used a quasi-experimental design.  Credit: Beth Dubber/Netflix/Kobal/Shutterstock

Update 7/19/2019: Netflix has decided to remove the scene depicting the main character's actual suicide, perhaps in response to data from this and other studies. Here's one story on the change. 

The Netflix series "Thirteen reasons why" tells the story of a teenage girl who commits suicide, leaving behind audio recordings about people who contributed to her decision. When the series was released in April, 2017, mental health and school professionals became concerned about the possibility that young viewers of the show would experience increased risk of suicide themselves. The phenomenon of "suicide contagion" occurs when a publicized suicide leads to other suicide attempts. Young people may be particularly susceptible to its effects. Suicide is the third leading cause of death among American teenagers (ages 12 to 19, according to the CDC). 

This Mashable journalist covers an empirical journal article published in JAMA Psychiatry in 2019.  The Mashable story reports: 

A new study, conducted by different authors, came to a similar conclusion: In the three months after 13 Reasons Why launched, the youth suicide rate for 10- to 19-year-olds rose unexpectedly by 13 percent. Based on historical trends dating back to 1999, 94 more American adolescents and teens died by suicide during the three-month period than otherwise expected. Similarly, the study published last month looked at suicide deaths in the nine months following the show's launch and found an additional 195 fatalities. 

Students and instructors will find it useful to consult the publicly-available JAMA Psychiatry article that contains graphs of this study. Specifically, the solid orange lines in Figure 2 show the suicide rates for 24 months of time: 15 months before the show debuted, and about 9 months after. 

a) This study used a quasi-experimental design. Which of the designs from Chapter 13 was it? (Non-equivalent control group posttest only? Non-equivalent control group pretest-posttest? Interrupted time series design? Non-equivalent control group posttest-only design?)

b) In the quote above, the journalist reported that an 13 percent increase in suicide equals an additional 94 adolescent teen deaths. This figure captures the effect size of the study's result. What do you think of this effect size? Would you call it small or large? 

The empirical journal article also documents the publicity the show received via social media. Here's how the journalist describes the evidence: 

Niederkrotenthaler and his co-authors looked at potential exposure to the show between April 1 to June 30, 2017, when the first season of 13 Reasons Why had just debuted and was most popular on Twitter and Instagram. Using social media data as a proxy for understanding peak interest in the show, the researchers suggest the show's portrayal of suicide created a 'contagion effect' The phenomenon can happen when media coverage of suicide leads to suicide clusters. 

Students and instructors can consult the empirical article's Figure 1 to see the results of the social media analysis. 

b) Why did the authors of the JAMA article include the social media piece? How does this evidence help them make their argument? 

The researchers cannot establish for certain that viewing the show caused the increase in suicide. Here's the Mashable journalist: 

While the researchers couldn't determine whether the children who took their own lives watched the series, an accompanying editorial written by researchers at Harvard and Stanford argues the study provides "strong evidence" that the Netflix show may have led to that increase. 

"One cannot draw definitive causal conclusions from such data, but that no similar increase was seen in suicide rates other than for the age group to which the media portrayal pertained provides some further compelling evidence that the excess suicides may indeed have been owing to the series," the editorial's authors wrote. 

c) The quote above adds an additional detail about the study. The researchers not only tracked the suicide rates of people aged 10-19; they also tracked suicide rates of people in the 20-29 and 30+ age group. The researchers found an increase in suicide after April 2017 only for the teenage group. How does this pattern of results support the argument that the show led to increased teen suicide? 

If you want to talk to someone or are experiencing suicidal thoughts, text the Crisis Text Line at 741-741 or call the National Suicide Prevention Lifeline at 1-800-273-8255. Here is a list of international resources. 

Best toys for 2-year olds

In one of the conditions of this experiment, parents interacted with their young children while playing a shapes game on an ipad. What was the other condition? Photo credit: Gettyimages

For young children, play is school: Children can learn math, physics, language, social skills, and more. And when caregivers interact with them, children at play can learn even more about their world. Caregivers can direct children's attention and teach them new words and concepts. Research from the laboratory of Temple University professor Dr. Nora Newcombe studies children's cognitive development. In particular, this group have studied what children learn about shapes and spatial information from playing with toys with their caregivers. 

Here's a one-minute video clip that portrays an experiment conducted in their laboratory. The study tested how caregivers and children interact when playing with different kinds of spatial toys. Watch the video with the following questions in mind. 

a) The video describes an experiment. What is the independent variable (IV)? How many levels of this IV were there, and what were they? 

b) What are the dependent variables (DVs)? (There are at least two DVs). 

c) Can you tell, based just on this description, if the IV was manipulated as independent groups or within groups?

d) Sketch a graph of the study's result. Would you use a bar graph or line graph?  

e) Sometimes a journalist pairs pre-existing stock video (i.e., video filmed for different purposes) with a story about a research study. Using other videos may be necessary because scientists do not release videos of their research participants for ethical reasons. How well do you think the videos that accompanied the story matched the study's design? 

f) What questions should you ask about this study's construct validity? internal validity? statistical validity? external validity? 

Who's more likely to claim expertise?

The journalist's article was illustrated with a photo of a businessman. Is that the best choice, given the study? Photo credit: Yuri Arcurs/Getty Images

Are you familiar with proper numbers? How about declarative fractions? 

If you claimed that you knew these concepts, you're probably fooling yourself....because declarative fractions and proper numbers are not real math things. 

A Washington Post journalist summarized a study on who is more likely to falsely claim expertise, or as some folks might put it: to "BS." The article's headline reads, "Rich guys are most likely to have no idea what they are talking about, study suggests." 

What did the study do?

Study participants were asked to assess their knowledge of 16 math topics on a five-point scale ranging from “never heard of it” to “know it well, understand the concept.” Crucially, three of those topics were complete fabrications: “proper numbers,” “subjunctive scaling” and “declarative fractions.” Those who said they were knowledgeable about the fictitious topics were categorized as BSers.

The sample for the study was international:

The study drew from the Program for International Student Assessment, which is administered to tens of thousands of 15-year-olds worldwide. The test included a background questionnaire that captures demographic information, along with students’ attitudes toward the subjects they study in school. That section of the test included the questions about math knowledge.

Here is the pattern of results for gender (You can see tables of the data in the journalist's article).  

The data revealed that boys across all nine countries were significantly more likely than girls to pretend expertise, with the difference between the two working out to nearly half a standard deviation in some countries — a big gap, statistically speaking.

And here's the description of the results for income:

There’s also a significant class-based difference, with respondents from the wealthiest households showing a greater proclivity toward overstatement than those from the poorest.  

a) Here's a statistical validity question. When the authors say that the boys are about a half a standard deviation higher than girls in some countries and report it's "a big gap, statistically speaking," they are referring to effect size. In fact, if two groups are half a standard deviation apart, the effect size is probably about d = 0.5. Is that considered a small, medium, or large effect according to Cohen's conventions? 

b) Here's an external validity question. What were the age and income levels of the students in the study? Reflect on the headline, which refers to "rich guys," and the photo accompanying the story, which shows an adult man. What do you think--what external validity issues are raised here? 

c) Here's a construct validity question. The construct the journalist claims is being measured here is the "tendency to BS." How was that construct operationalized in this study? What do you think of that operationalization? Can you think of other ways to operationalize it? 

d) You might think we're ready for an internal validity question. However, internal validity doesn't seem relevant here, because nobody seems to be making a causal claim....or are they? Could you make a causal claim here? 

e) Advanced question: You might wish to investigate the table of data in the article that depicts the "BS gaps between boys and girls." Do you think that this table really needed separate columns for boys, girls, and "gap"? Why was it formatted this way? How else might you present these data? 

f) At the end of the article, the scientists are quoted about the study's implications and limitations. One of the study's original authors says, 

The authors also point out that their study was narrowly defined and restricted to the realm of mathematics. “Ideally,” they wrote, “future research should try to include a greater number of fake constructs in order to maximize precision of the bulls--- scale.”

Which of the four big validities seems to be addressed in this quote? 

How we spot leadership potential in men and women

Which one of these people will benefit most from sharing good ideas with the group? Credit: Dean Drobot/Shutterstock

Imagine you're working at a company and you'd like people to view you as a potential leader. How might you go about it? Perhaps you'd share your good ideas at meetings, hoping they'll notice your talent. Perhaps you'd pursue higher level training to improve your credentials. Maybe you'd point out ways your group's processes could improve. Would these strategies work? The answer may depend on your gender. 

The trouble is that people's prototype of a "leader" is a man. As this journalist describes, when people are asked to draw a picture of a leader, they almost always draw a male figure (by the way, that's a frequency claim). How might people's default assumption--that leaders are male--affect their ability to spot leadership potential in their colleagues? 

According to the journalist,

...getting noticed as a leader in the workplace is more difficult for women than for men. Even when a man and a woman were reading the same words off a script, only the man’s leadership potential was recognized.

Here is a study described in the piece. It's a factorial design (that is, it has two independent variables). As you read, decide what the two IVs are and what their levels are: 

In one experiment, participants were asked to call into a monthly sales team meeting of a fictional insurance company, during which they would hear from either an Eric or an Erica. Later they were asked to rate the speaker on the degree to which he or she had “exhibited leadership,” “influenced the team” or “assumed a leadership role.”

The Erics who spoke up with change-oriented ideas were far more likely to be identified as leaders [rated higher on the leadership questions] than Erics who simply critiqued their team’s performance. But Ericas did not receive a boost in status from sharing ideas even though they were exactly the same as the Erics’.

a) What is the first IV? How many levels does it have and what are they? Is it manipulated or measured? 

b) What is the second IV? How many levels does it have and what are they? Is it manipulated or measured? 

c) What are the three DVs? 

d) Sketch a well-labeled graph of the results of this study, using the DV of "exhibited leadership". Remember to put one of the IVs on the x-axis and the other IV as a line color or bar color. 

Here's another Factorial design described in the journalist's article. 

...in a second experiment, the researchers attempted to investigate the benefits of speaking up in the “wild.” The researchers asked participants in a 36-team skills competition at the U.S. Military Academy at West Point to fill out a series of questionnaires before, during and after their competition.

The authors acknowledged that there were limitations in studying this atypically male-dominated place, but the nature of the competition allowed them to control for athletic ability, cognitive ability and other factors.

Still, after the competition, when participants ranked who they wanted to be the team leader, only the men who had spoken up with ideas received a boost.

“It didn’t matter whether women spoke up 1) almost never, 2) rarely, 3) sometimes, 4) often, or 5) almost always,” Kyle Emich, a professor at Alfred Lerner College of Business and Economics at the University of Delaware, and one of the authors, wrote in an email. “Women did not gain status for speaking up, and subsequently were less likely (much less) to be considered leaders.”

e) One of the IVs in this study has FIVE levels--that's unusual, but it can happen!  What is this IV and what are its five levels?  Is this IV manipulated or measured? 

f) What is the second IV? How many levels does it have and what are they? Is it manipulated or measured? 

g) The DV is not explicitly stated in this summary. What do you think the DV was, based on this description? 

h) Sketch a well-labeled graph of the results of this study, using the DV of "being nominated by peers as being the leader". I suggest that you put the 5-level IVs  on the x-axis and the other IV as a line color or bar color. 

i) The first example (With Eric and Erica) was a 2x2 factorial design. How would you state the design of this West Point example? 

 

 

Human intelligence over time

When people knew they could look up facts online, they didn't remember them as well.

Here's an article on how intelligence might be changing over time, from Australia's Sydney Morning Herald. It's a great opportunity to practice classifying claims and identifying studies. 

One point the article makes is that humans have been getting more intelligent in general. For example:

For the last century, we’ve been getting smarter - fast. The average IQ of the population has been increasing by about three points per decade.

Even Mensa has noticed. “Our membership is growing at about 10 per cent a year,” says Teresa Wong, the organisation’s recruitment officer. “And about a third of them are under 18.” [...]

The same increase in IQs over the last century has happened in nearly every other developed nation. It’s known as the Flynn effect after James Flynn, the New Zealand researcher who discovered it.

There are several theories for why this has happened: we’re better educated, modern education systems mould our brain in ways IQ tests tend to like, and many of us will do cognitively-taxing things like play video games during our leisure hours.

a. What kind of claim is it to say that "the average IQ of the population has been increasing by about 3 points per decade"? (Frequency, association, or cause?)

 b. What kind of data would you need to support this claim?

The journalist then shifts the focus: 

But something strange and troubling appears to be happening. In several countries – Norway, Denmark, Finland, Britain and France – IQ scores that rose for so long now seem to be dropping, at a rate of about 2.4 points per decade in recent decades.

c. If you were to sketch the average IQ scores of these nations over several decades, what would the graph look like? 

The journalist asks, 

Could it be that our smartphones are making us dumber? We can look up any fact in an instant. We never need to remember facts or directions or friends’ phone numbers. Is that making us more stupid?

d. What kind of claim is the journalist making, above?

e. Below are three studies the journalist describes. Two are probably correlational and one is probably experimental. Which is which?  (and, what are the two variables in each study?):

i. Studies have linked smartphone use with a decreased ability to exert high levels of focus and poorer attention control – although video games seem linked to better multitasking abilities.

ii. In one influential study, volunteers were asked to type newly-learned trivia facts into a computer. Those who were told the computer would remember the facts for them performed much worse on a later memory-recall test. The researchers dubbed it the Google Effect.

iii. In another study, volunteers were given a series of questions that pushed the limits of their brain. Those who said they were heavy smartphone users tended to perform worse and be less analytical. 

f. What kind of data would it take to convince you that the Internet is responsible for the recent declines in IQ around the world? 

You can read one of the original "Google Effect" studies here. 

Which brand of car is the most reliable?

How might advertisers operationalize the concept of "reliability" in ways favorable to their own brand?  Credit: BNFWork/Shutterstock

You may have seen these ads on TV: Real people stand in a warehouse and are asked which cars are "most reliable." A Honda driver might predict, "Hondas are the most reliable." But then the cover is whisked off in a flourish: The most reliable car is actually the Chevy! In one recent ad (viewable here), the narrator reports: 

"Based on a recent nationwide survey, Chevy is more reliable than Toyota, Honda, and Ford."

Should statements like these convince you? The website Jalopnik publishes journalism about cars. In this column, a journalist walks readers through how they might think critically about  reliability claims in the Chevy commercials. For example, the journalist writes that the reliability claim was based on a survey.

...the survey was sent in 2018 to owners of 2015 model year cars, and the reporting of the “repairs in the past 12 months” relates to those 2015 cars in their third year of service. Chevy reiterated that “Independent statisticians reviewed the materials and concluded with 95% confidence that Chevy’s percentage of no parts repaired or replaced is better than Toyota, Honda, Ford or 23 other brands.”

The journalist adds:

Anyone who has studied research and statistics will tell you that it’s not difficult to form the conclusions you want to based on your use of what is called an “operational definition.” For this particular study, Ipsos is operationally defining reliability as the repairs a car has had within a 12 month period once the car has reached three years old.

a) How did Chevy operationalize "reliability" in their study? 

b) What are some alternative ways to operationalize a car brand's "reliability?"

c) Can you think of downsides to the operationalization that Chevy used? 

The journalist pointed out that, other than the operationalization of reliability, the survey's number of respondents might matter. To wit:

Setting aside the fact that self-reported surveys can be flawed, fewer than 49,000 respondents completed them out of almost 840,000. What Chevrolet did not provide is a breakdown of those respondents by brand, because if we were looking at the percentage of repairs made by various brand owners this could make a difference.

For example, if only 1,000 of those 49,000 respondents were Honda owners and 100 of them reported repairs within that 12 month period, that means that Honda would have a 10 percent repair rate. But if 15,000 of the respondents were Chevrolet owners and 1,200 of them reported repairs, that is an 8 percent repair rate. Yet this distribution was not provided so it’s difficult to ascertain exactly how those percentages compare within each sample size of brand owners.

d) What do you think of the journalist's reasoning, above? 

In another argument, the journalist compares the self-report survey used by Chevy with two other sources of reliability data, Consumer Reports and J.D. Power: 

...in a 2018 ranking of brand reliability, Consumer Reports ranked both Honda and Toyota well above Chevrolet, [which] occupied the lower part of the list along with several other domestic brands.

But like Chevrolet’s survey, this data is also self-reported based on the ownership experiences of Consumer Reports subscribers.

I also reached out to J.D. Power, as they are a respected reference when it comes to these kinds of “quality” rankings and asked them to comment on the methodology behind Chevrolet’s study. They declined. (Recall that Chevy has touted J.D. Power awards in this very same series of ads.)

e) Why might the journalist extol Consumer Reports or J.D. Power over Chevy's Ipsos survey? What might these organizations doing differently--if anything--in their measurement of reliability? 

f) Reflect on how the availability heuristic (the pop-up principle), cherry picking, bias blind spots, or the present-present bias (all from Chapter 2) might lead individuals to make faulty conclusions about which car brands are most reliable.

 

Thanks to Marianne Lloyd for sharing this idea!