Everyday Research Methods

Human intelligence over time

When people knew they could look up facts online, they didn't remember them as well.

Here's an article on how intelligence might be changing over time, from Australia's Sydney Morning Herald. It's a great opportunity to practice classifying claims and identifying studies. 

One point the article makes is that humans have been getting more intelligent in general. For example:

For the last century, we’ve been getting smarter - fast. The average IQ of the population has been increasing by about three points per decade.

Even Mensa has noticed. “Our membership is growing at about 10 per cent a year,” says Teresa Wong, the organisation’s recruitment officer. “And about a third of them are under 18.” [...]

The same increase in IQs over the last century has happened in nearly every other developed nation. It’s known as the Flynn effect after James Flynn, the New Zealand researcher who discovered it.

There are several theories for why this has happened: we’re better educated, modern education systems mould our brain in ways IQ tests tend to like, and many of us will do cognitively-taxing things like play video games during our leisure hours.

a. What kind of claim is it to say that "the average IQ of the population has been increasing by about 3 points per decade"? (Frequency, association, or cause?)

 b. What kind of data would you need to support this claim?

The journalist then shifts the focus: 

But something strange and troubling appears to be happening. In several countries – Norway, Denmark, Finland, Britain and France – IQ scores that rose for so long now seem to be dropping, at a rate of about 2.4 points per decade in recent decades.

c. If you were to sketch the average IQ scores of these nations over several decades, what would the graph look like? 

The journalist asks, 

Could it be that our smartphones are making us dumber? We can look up any fact in an instant. We never need to remember facts or directions or friends’ phone numbers. Is that making us more stupid?

d. What kind of claim is the journalist making, above?

e. Below are three studies the journalist describes. Two are probably correlational and one is probably experimental. Which is which?  (and, what are the two variables in each study?):

i. Studies have linked smartphone use with a decreased ability to exert high levels of focus and poorer attention control – although video games seem linked to better multitasking abilities.

ii. In one influential study, volunteers were asked to type newly-learned trivia facts into a computer. Those who were told the computer would remember the facts for them performed much worse on a later memory-recall test. The researchers dubbed it the Google Effect.

iii. In another study, volunteers were given a series of questions that pushed the limits of their brain. Those who said they were heavy smartphone users tended to perform worse and be less analytical. 

f. What kind of data would it take to convince you that the Internet is responsible for the recent declines in IQ around the world? 

You can read one of the original "Google Effect" studies here. 

Which brand of car is the most reliable?

How might advertisers operationalize the concept of "reliability" in ways favorable to their own brand?  Credit: BNFWork/Shutterstock

You may have seen these ads on TV: Real people stand in a warehouse and are asked which cars are "most reliable." A Honda driver might predict, "Hondas are the most reliable." But then the cover is whisked off in a flourish: The most reliable car is actually the Chevy! In one recent ad (viewable here), the narrator reports: 

"Based on a recent nationwide survey, Chevy is more reliable than Toyota, Honda, and Ford."

Should statements like these convince you? The website Jalopnik publishes journalism about cars. In this column, a journalist walks readers through how they might think critically about  reliability claims in the Chevy commercials. For example, the journalist writes that the reliability claim was based on a survey.

...the survey was sent in 2018 to owners of 2015 model year cars, and the reporting of the “repairs in the past 12 months” relates to those 2015 cars in their third year of service. Chevy reiterated that “Independent statisticians reviewed the materials and concluded with 95% confidence that Chevy’s percentage of no parts repaired or replaced is better than Toyota, Honda, Ford or 23 other brands.”

The journalist adds:

Anyone who has studied research and statistics will tell you that it’s not difficult to form the conclusions you want to based on your use of what is called an “operational definition.” For this particular study, Ipsos is operationally defining reliability as the repairs a car has had within a 12 month period once the car has reached three years old.

a) How did Chevy operationalize "reliability" in their study? 

b) What are some alternative ways to operationalize a car brand's "reliability?"

c) Can you think of downsides to the operationalization that Chevy used? 

The journalist pointed out that, other than the operationalization of reliability, the survey's number of respondents might matter. To wit:

Setting aside the fact that self-reported surveys can be flawed, fewer than 49,000 respondents completed them out of almost 840,000. What Chevrolet did not provide is a breakdown of those respondents by brand, because if we were looking at the percentage of repairs made by various brand owners this could make a difference.

For example, if only 1,000 of those 49,000 respondents were Honda owners and 100 of them reported repairs within that 12 month period, that means that Honda would have a 10 percent repair rate. But if 15,000 of the respondents were Chevrolet owners and 1,200 of them reported repairs, that is an 8 percent repair rate. Yet this distribution was not provided so it’s difficult to ascertain exactly how those percentages compare within each sample size of brand owners.

d) What do you think of the journalist's reasoning, above? 

In another argument, the journalist compares the self-report survey used by Chevy with two other sources of reliability data, Consumer Reports and J.D. Power: 

...in a 2018 ranking of brand reliability, Consumer Reports ranked both Honda and Toyota well above Chevrolet, [which] occupied the lower part of the list along with several other domestic brands.

But like Chevrolet’s survey, this data is also self-reported based on the ownership experiences of Consumer Reports subscribers.

I also reached out to J.D. Power, as they are a respected reference when it comes to these kinds of “quality” rankings and asked them to comment on the methodology behind Chevrolet’s study. They declined. (Recall that Chevy has touted J.D. Power awards in this very same series of ads.)

e) Why might the journalist extol Consumer Reports or J.D. Power over Chevy's Ipsos survey? What might these organizations doing differently--if anything--in their measurement of reliability? 

f) Reflect on how the availability heuristic (the pop-up principle), cherry picking, bias blind spots, or the present-present bias (all from Chapter 2) might lead individuals to make faulty conclusions about which car brands are most reliable.

 

Thanks to Marianne Lloyd for sharing this idea!