Everyday Research Methods

Who's more likely to claim expertise?

The journalist's article was illustrated with a photo of a businessman. Is that the best choice, given the study? Photo credit: Yuri Arcurs/Getty Images

Are you familiar with proper numbers? How about declarative fractions? 

If you claimed that you knew these concepts, you're probably fooling yourself....because declarative fractions and proper numbers are not real math things. 

A Washington Post journalist summarized a study on who is more likely to falsely claim expertise, or as some folks might put it: to "BS." The article's headline reads, "Rich guys are most likely to have no idea what they are talking about, study suggests." 

What did the study do?

Study participants were asked to assess their knowledge of 16 math topics on a five-point scale ranging from “never heard of it” to “know it well, understand the concept.” Crucially, three of those topics were complete fabrications: “proper numbers,” “subjunctive scaling” and “declarative fractions.” Those who said they were knowledgeable about the fictitious topics were categorized as BSers.

The sample for the study was international:

The study drew from the Program for International Student Assessment, which is administered to tens of thousands of 15-year-olds worldwide. The test included a background questionnaire that captures demographic information, along with students’ attitudes toward the subjects they study in school. That section of the test included the questions about math knowledge.

Here is the pattern of results for gender (You can see tables of the data in the journalist's article).  

The data revealed that boys across all nine countries were significantly more likely than girls to pretend expertise, with the difference between the two working out to nearly half a standard deviation in some countries — a big gap, statistically speaking.

And here's the description of the results for income:

There’s also a significant class-based difference, with respondents from the wealthiest households showing a greater proclivity toward overstatement than those from the poorest.  

a) Here's a statistical validity question. When the authors say that the boys are about a half a standard deviation higher than girls in some countries and report it's "a big gap, statistically speaking," they are referring to effect size. In fact, if two groups are half a standard deviation apart, the effect size is probably about d = 0.5. Is that considered a small, medium, or large effect according to Cohen's conventions? 

b) Here's an external validity question. What were the age and income levels of the students in the study? Reflect on the headline, which refers to "rich guys," and the photo accompanying the story, which shows an adult man. What do you think--what external validity issues are raised here? 

c) Here's a construct validity question. The construct the journalist claims is being measured here is the "tendency to BS." How was that construct operationalized in this study? What do you think of that operationalization? Can you think of other ways to operationalize it? 

d) You might think we're ready for an internal validity question. However, internal validity doesn't seem relevant here, because nobody seems to be making a causal claim....or are they? Could you make a causal claim here? 

e) Advanced question: You might wish to investigate the table of data in the article that depicts the "BS gaps between boys and girls." Do you think that this table really needed separate columns for boys, girls, and "gap"? Why was it formatted this way? How else might you present these data? 

f) At the end of the article, the scientists are quoted about the study's implications and limitations. One of the study's original authors says, 

The authors also point out that their study was narrowly defined and restricted to the realm of mathematics. “Ideally,” they wrote, “future research should try to include a greater number of fake constructs in order to maximize precision of the bulls--- scale.”

Which of the four big validities seems to be addressed in this quote? 

How we spot leadership potential in men and women

Which one of these people will benefit most from sharing good ideas with the group? Credit: Dean Drobot/Shutterstock

Imagine you're working at a company and you'd like people to view you as a potential leader. How might you go about it? Perhaps you'd share your good ideas at meetings, hoping they'll notice your talent. Perhaps you'd pursue higher level training to improve your credentials. Maybe you'd point out ways your group's processes could improve. Would these strategies work? The answer may depend on your gender. 

The trouble is that people's prototype of a "leader" is a man. As this journalist describes, when people are asked to draw a picture of a leader, they almost always draw a male figure (by the way, that's a frequency claim). How might people's default assumption--that leaders are male--affect their ability to spot leadership potential in their colleagues? 

According to the journalist,

...getting noticed as a leader in the workplace is more difficult for women than for men. Even when a man and a woman were reading the same words off a script, only the man’s leadership potential was recognized.

Here is a study described in the piece. It's a factorial design (that is, it has two independent variables). As you read, decide what the two IVs are and what their levels are: 

In one experiment, participants were asked to call into a monthly sales team meeting of a fictional insurance company, during which they would hear from either an Eric or an Erica. Later they were asked to rate the speaker on the degree to which he or she had “exhibited leadership,” “influenced the team” or “assumed a leadership role.”

The Erics who spoke up with change-oriented ideas were far more likely to be identified as leaders [rated higher on the leadership questions] than Erics who simply critiqued their team’s performance. But Ericas did not receive a boost in status from sharing ideas even though they were exactly the same as the Erics’.

a) What is the first IV? How many levels does it have and what are they? Is it manipulated or measured? 

b) What is the second IV? How many levels does it have and what are they? Is it manipulated or measured? 

c) What are the three DVs? 

d) Sketch a well-labeled graph of the results of this study, using the DV of "exhibited leadership". Remember to put one of the IVs on the x-axis and the other IV as a line color or bar color. 

Here's another Factorial design described in the journalist's article. 

...in a second experiment, the researchers attempted to investigate the benefits of speaking up in the “wild.” The researchers asked participants in a 36-team skills competition at the U.S. Military Academy at West Point to fill out a series of questionnaires before, during and after their competition.

The authors acknowledged that there were limitations in studying this atypically male-dominated place, but the nature of the competition allowed them to control for athletic ability, cognitive ability and other factors.

Still, after the competition, when participants ranked who they wanted to be the team leader, only the men who had spoken up with ideas received a boost.

“It didn’t matter whether women spoke up 1) almost never, 2) rarely, 3) sometimes, 4) often, or 5) almost always,” Kyle Emich, a professor at Alfred Lerner College of Business and Economics at the University of Delaware, and one of the authors, wrote in an email. “Women did not gain status for speaking up, and subsequently were less likely (much less) to be considered leaders.”

e) One of the IVs in this study has FIVE levels--that's unusual, but it can happen!  What is this IV and what are its five levels?  Is this IV manipulated or measured? 

f) What is the second IV? How many levels does it have and what are they? Is it manipulated or measured? 

g) The DV is not explicitly stated in this summary. What do you think the DV was, based on this description? 

h) Sketch a well-labeled graph of the results of this study, using the DV of "being nominated by peers as being the leader". I suggest that you put the 5-level IVs  on the x-axis and the other IV as a line color or bar color. 

i) The first example (With Eric and Erica) was a 2x2 factorial design. How would you state the design of this West Point example? 

 

 

Human intelligence over time

When people knew they could look up facts online, they didn't remember them as well.

Here's an article on how intelligence might be changing over time, from Australia's Sydney Morning Herald. It's a great opportunity to practice classifying claims and identifying studies. 

One point the article makes is that humans have been getting more intelligent in general. For example:

For the last century, we’ve been getting smarter - fast. The average IQ of the population has been increasing by about three points per decade.

Even Mensa has noticed. “Our membership is growing at about 10 per cent a year,” says Teresa Wong, the organisation’s recruitment officer. “And about a third of them are under 18.” [...]

The same increase in IQs over the last century has happened in nearly every other developed nation. It’s known as the Flynn effect after James Flynn, the New Zealand researcher who discovered it.

There are several theories for why this has happened: we’re better educated, modern education systems mould our brain in ways IQ tests tend to like, and many of us will do cognitively-taxing things like play video games during our leisure hours.

a. What kind of claim is it to say that "the average IQ of the population has been increasing by about 3 points per decade"? (Frequency, association, or cause?)

 b. What kind of data would you need to support this claim?

The journalist then shifts the focus: 

But something strange and troubling appears to be happening. In several countries – Norway, Denmark, Finland, Britain and France – IQ scores that rose for so long now seem to be dropping, at a rate of about 2.4 points per decade in recent decades.

c. If you were to sketch the average IQ scores of these nations over several decades, what would the graph look like? 

The journalist asks, 

Could it be that our smartphones are making us dumber? We can look up any fact in an instant. We never need to remember facts or directions or friends’ phone numbers. Is that making us more stupid?

d. What kind of claim is the journalist making, above?

e. Below are three studies the journalist describes. Two are probably correlational and one is probably experimental. Which is which?  (and, what are the two variables in each study?):

i. Studies have linked smartphone use with a decreased ability to exert high levels of focus and poorer attention control – although video games seem linked to better multitasking abilities.

ii. In one influential study, volunteers were asked to type newly-learned trivia facts into a computer. Those who were told the computer would remember the facts for them performed much worse on a later memory-recall test. The researchers dubbed it the Google Effect.

iii. In another study, volunteers were given a series of questions that pushed the limits of their brain. Those who said they were heavy smartphone users tended to perform worse and be less analytical. 

f. What kind of data would it take to convince you that the Internet is responsible for the recent declines in IQ around the world? 

You can read one of the original "Google Effect" studies here. 

Which brand of car is the most reliable?

How might advertisers operationalize the concept of "reliability" in ways favorable to their own brand?  Credit: BNFWork/Shutterstock

You may have seen these ads on TV: Real people stand in a warehouse and are asked which cars are "most reliable." A Honda driver might predict, "Hondas are the most reliable." But then the cover is whisked off in a flourish: The most reliable car is actually the Chevy! In one recent ad (viewable here), the narrator reports: 

"Based on a recent nationwide survey, Chevy is more reliable than Toyota, Honda, and Ford."

Should statements like these convince you? The website Jalopnik publishes journalism about cars. In this column, a journalist walks readers through how they might think critically about  reliability claims in the Chevy commercials. For example, the journalist writes that the reliability claim was based on a survey.

...the survey was sent in 2018 to owners of 2015 model year cars, and the reporting of the “repairs in the past 12 months” relates to those 2015 cars in their third year of service. Chevy reiterated that “Independent statisticians reviewed the materials and concluded with 95% confidence that Chevy’s percentage of no parts repaired or replaced is better than Toyota, Honda, Ford or 23 other brands.”

The journalist adds:

Anyone who has studied research and statistics will tell you that it’s not difficult to form the conclusions you want to based on your use of what is called an “operational definition.” For this particular study, Ipsos is operationally defining reliability as the repairs a car has had within a 12 month period once the car has reached three years old.

a) How did Chevy operationalize "reliability" in their study? 

b) What are some alternative ways to operationalize a car brand's "reliability?"

c) Can you think of downsides to the operationalization that Chevy used? 

The journalist pointed out that, other than the operationalization of reliability, the survey's number of respondents might matter. To wit:

Setting aside the fact that self-reported surveys can be flawed, fewer than 49,000 respondents completed them out of almost 840,000. What Chevrolet did not provide is a breakdown of those respondents by brand, because if we were looking at the percentage of repairs made by various brand owners this could make a difference.

For example, if only 1,000 of those 49,000 respondents were Honda owners and 100 of them reported repairs within that 12 month period, that means that Honda would have a 10 percent repair rate. But if 15,000 of the respondents were Chevrolet owners and 1,200 of them reported repairs, that is an 8 percent repair rate. Yet this distribution was not provided so it’s difficult to ascertain exactly how those percentages compare within each sample size of brand owners.

d) What do you think of the journalist's reasoning, above? 

In another argument, the journalist compares the self-report survey used by Chevy with two other sources of reliability data, Consumer Reports and J.D. Power: 

...in a 2018 ranking of brand reliability, Consumer Reports ranked both Honda and Toyota well above Chevrolet, [which] occupied the lower part of the list along with several other domestic brands.

But like Chevrolet’s survey, this data is also self-reported based on the ownership experiences of Consumer Reports subscribers.

I also reached out to J.D. Power, as they are a respected reference when it comes to these kinds of “quality” rankings and asked them to comment on the methodology behind Chevrolet’s study. They declined. (Recall that Chevy has touted J.D. Power awards in this very same series of ads.)

e) Why might the journalist extol Consumer Reports or J.D. Power over Chevy's Ipsos survey? What might these organizations doing differently--if anything--in their measurement of reliability? 

f) Reflect on how the availability heuristic (the pop-up principle), cherry picking, bias blind spots, or the present-present bias (all from Chapter 2) might lead individuals to make faulty conclusions about which car brands are most reliable.

 

Thanks to Marianne Lloyd for sharing this idea! 

 

Claim: Mothers' friendships are good for babies' brains

The study showed that moms with larger social support networks had kids with higher cognitive test scores. Photo: Noriko Cooper / Alamy Stock Photo

The  Washington Post headline reads, "Mothers' friendships may be good for babies' brains."  The story is about an empirical study on mothers' social networks. Although it says "babies' brains", the study measured the babies' cognitive skill (there were no baby brain scans). The empirical article came out in JAMA Network Open. Does the original study support the claim? Let's find out.Before reading on, answer these questions:

 

a) What are the two variables in the Washington Post headline claim? What makes the claim a causal one? 

b) Of course, we need to conduct an experiment to definitively support a causal claim. Would an experiment be possible in this particular case?  

Here are some details about the study, as summarized by the journalist (not the researcher):

For the current study, researchers examined data on 1,082 mother-child pairs. They questioned women about their family structure, friendships and relationships in their communities and also looked at test results from cognitive assessments done when children were 2 years old.

Overall, mothers had an average of 3.5 friends in their social support networks. The kids of mothers with more than that tended to have higher cognitive test scores than the kids of those who had fewer, suggesting “network conditions were significantly associated with early cognitive development in children,” the study authors wrote.

c) Sketch a  simple scatterplot of the result described above. What will you put on the two axes? How many dots should your scatterplot have? What does one dot represent? 

Read the following details if you've studied Chapter 9: 

The researchers accounted for some factors that might influence child cognitive development, including mother’s age and IQ, father’s education level, and the child’s birth weight. 

In addition, "researchers also accounted for poverty levels"

d) The paragraph above is describing a multiple regression analysis. What would be the dependent (criterion variable) in this analysis? What are the predictor variables? (Hint: There are six predictor variables).

e) Imagine that someone proposes income as a possible internal validity problem. Given what you know about the study, can we say that "Perhaps friendship network size and child cognitive skills are correlated because of the third variable of income: That is, higher-income mothers may have more friends and more cognitively skilled children"?  Would this criticism be reasonable? Why or why not?

f) The journalist writes: "...they did not have data to assess other personal differences, such as history of depression, between mothers with large networks of friends and those with fewer."
This means history of depression could be a possible third variable explaining the relationship. How could the researchers rule out this third variable? 

The journalist writes: 

All of the families in the study lived in the area around Memphis. It is possible that results might be different elsewhere.

g) Which big validity is the journalist addressing in the statement above? 

h) Do you think the correlation between friendship and cognitive test scores would actually come out differently in a different city or country? Why or why not? 

 

Suggested answers:

a) The two variables are Amount of mothers' friendships and Level of health of babies' brains. The language of "may be good for" makes this a causal headline (remember we ignore adverbs such as "may" when classifying claims). 

b) Probably not; you'd have to randomly assign some mothers to have more friends and others to have fewer friends.

c)  One axis should have "number of friends" and the other should have "cognitive test score". There should be about 1082 dots on the scatterplot (better count your dots now...), and each dot represents a mother-child dyad.

d) The criterion variable is cognitive test score. The predictor variables are number of friends, poverty levels, mother’s age, mother's IQ, father’s education level and the child’s birth weight. 

e) Since the researchers did control for poverty levels, we can actually rule out wealth as a possible third variable. 

f) the researchers could obtain history of depression data on each mother and then add this variable as a predictor variable to the regression analysis. If the beta associated with "number of friends" is still significantly predicting cognitive test score, then we can rule out this possible confound (/third variable/ internal validity problem).

g) external validity

h) In my opinion, it seems unlikely that such a relationship is limited only to Memphis residents. 

Note: You can read the original empirical article in JAMA, here. Note to savvy instructors: The p-curve on the results in this paper suggests that there may be little evidentiary value. 

More butter, happier citizenry

What internal validity problems prevent us from claiming that butter leads to happiness? Photo: Fatty's place/Shutterstock

A writer for the Big Think wrote about how "Butter supply and life-satisfaction are linked".  It's a dairy interesting story with perfect scatterplots for practicing correlational designs.

Writer Frank Jacobs used the following bullet-points to open his story: 

• Haiti and other countries with low butter supply report low life satisfaction.
• The reverse is true for countries such as Germany, which score high in both categories.
• As the graph below shows, a curious pattern emerges across the globe. But is it causation or correlation?

You should click on the story and scroll down to the first scatterplot. 

a) What does one dot represent on this scatterplot?

b) What kind of correlation is being represented here: positive, negative, or zero?

c) Find your own home country. Is your country above the median or below the median on butter supply? Is your country above the median or below the median on life satisfaction? 

d) What would happen to the scatterplot if it were drawn with butter supply on the y-axis and happiness on the x-axis? Would it have the same shape and slope? 

The writer states:

Yet as the map shows, an abundance of butter does make people happy. Or could it be a case of correlation instead of causation? In that case, something else influences both life satisfaction and the availability of butter to go up and down together. 

d) In the curious paragraph above, the writer makes a clear causal claim. What is it?

e) After making the causal claim, the writer backs off, stating that "something else" could be associated with both life satisfaction and the availability of butter. He's talking about third variables here--or internal validity. What possible third variables might plausibly be correlated with both butter supply and life satisfaction in the world's countries? 

To help you consider possible third variables, zoom in on the countries that are in the lower left and upper right quadrants of the butter scatterplot. What dimension seems to separate the countries in those two quadrants? 

f) Take a look back at the column in Big Think. Is this data (and scatterplot result) from a published, peer reviewed study in an empirical journal? 

I butter thank Sue Frantz of Highline College for sharing this story with me. 

The science behind sleep

The science on sleep suggests that most of us need more of it!
Photo: Carlos Caetano/Shutterstock

A Washington Post article recently proposed that lack of sleep might be a "public health crisis." The journalist's story covered several types of empirical studies and made a series of frequency, association, and causal claims. 

Here are a series of quotes from the article. As you read each quote, ask yourself, a) How many variables seem to be involved in this claim? b) What is/are the variable(s)?  and then c) Is this claim frequency, association, or cause? How do you know? (Note: the links in each claim take you to the empirical articles on which they are based.)

Quote 1:

"Preschoolers who skip naps are worse at a memory game than those who snooze."

Quote 2:

"...a single night of sleep deprivation boosts brain levels of the proteins that form toxic clumps in Alzheimer’s patients." 

Quote 3:

"In 2016, the Centers for Disease Control and Prevention reported that a third of adults fail to get the recommended seven hours."

Quote 4:

"...even modest sleep reductions are linked to increased feelings of social isolation and loneliness."

Quote 5:

"A small study presented at the Society for Neuroscience conference last year found early evidence that playing “pink” noise — similar to white noise but with different frequencies — in a specific pattern could increase the proportion of time people spent in restorative deep sleep."

Quote 6:

"Adults over 50 with lots of insomnia symptoms were more likely to fall than those without, according to one study.:

Quote 7

"Studies have linked pain to poor sleep.."

 

Suggested answers to odd-numbered quotes.

Quote 1.

1. There are two variables:
2. Skipping a nap (or not) and level of skill at a memory game
3. This is an association claim, because the verb “are” is simply linking one type of preschooler (those who skip naps) with having lower skill at the memory game.

 

Quote 3.

1. There is one variable:
2. Getting 7 hours of sleep a night (or not)
3. This is a frequency claim. There is one variable and the sentence is mainly describing the number of U.S. adults who get that amount of sleep.

 

Quote 5.

1. There are two variables:
2. Playing “pink” noise (or not) and amount of time spent in restorative sleep
3. This is a causal claim because the verb used was “increase”.

 

Quote 7.

1. There are two variables:
2. Level of pain and quality of sleep.
3. The verb, “linked” is a verb signaling an association claim.

Which comes first? Cross-lag study on depression and social media use

Which comes first, social media use or depression? Photo:Africa Studio/Shutterstock

We have seen multiple studies that have tried to estimate how strong the link is between social media use and depression among young adults. We've covered some of these studies on this blog, including the effect size of the correlation and even a brain imaging study that manipulated "likes."

One challenge with studying real-world links between social media use and depression is that most use correlational designs. Therefore, we can't be sure whether any link between social media use and depression means that social media leads to more depression, or if depression leads to more social media use. And of course, in correlational studies there may even be a third variable, such as age, which is associated with both variables and causing them to appear correlated. 

The latest study used a longitudinal, cross-lag panel design, such as those described in Chapter 9, to study this question. Here's a writeup of the study by a journalist at Earth.com.

First, here are some comments about the design: 

The researchers point out that previous studies that found links between depression and social media use among tweens are limited in their results because they only captured a snapshot of teen social media use.

“You have to follow the same people over time in order to draw the conclusion that social media use predicts greater depressive symptoms,” said Taylor Heffer, the lead author of the study. “By using two large longitudinal samples, we were able to empirically test that assumption.”

Here are some more details about the method. They actually used two longitudinal samples: 

For this study, the researchers conducted a two-year longitudinal study surveying 6th, 7th, and 8th graders in Canada. The team also surveyed a series of undergraduate students once a year for six years as they progressed through college.

What were the results? 

Higher depression symptoms predicted social media use among teen girls which indicates that unhealthy social media use may be a symptom depression, not a catalyst for it.

“This finding contrasts with the idea that people who use a lot of social media become more depressed over time. Instead, adolescent girls who are feeling down may turn to social media to try and make themselves feel better,” said Heffer.

Questions

a) In a cross-lag panel design, researchers measure the same two variables in the same sample of people over time. What are the two variables they focused on here?

b) Sketch a series of boxes depicting the design of the study, similar to the figures in Chapter 9's cross-lag panel section. How many time periods would you need to depict for the undergraduate sample? 

c) Focus only on the girls. What do you think the autocorrelations might have been in this design? Fill those in on your sketch.

d) Still focusing only on the girls, given the description of the results, estimate what the two cross-lag correlations might have been. Which correlation should be larger--the one from Time 1 depression to Time 2 social media, or the one from Time 1 social media to Time 2 depression?

 

How much do chimpanzees cooperate?

Which behaviors were used to operationalize cooperation in chimps?
Photo credit: apple2499/Shutterstock

Human beings are a fundamentally prosocial and cooperative species, but what about one of our close relatives, the chimpanzee?  A science writer summarized a series of studies on chimpanzees who live in the Tchimpounga Chimpanzee Sanctuary, in the Republic of the Congo. The summary is here and you can read the empirical journal article here.

The studies illustrate how researchers can operationalize a variable like "cooperativeness" in other species. It provides a clear example of observational research, too. 

Researcher Alexandra Rosati conducted three observational tasks on the sanctuary, all meant to operationalize the construct cooperativeness. The first task took place with chimp volunteers who were alone in a dormitory area. One experimenter grabbed another one's stick and threw it out of reach into the dormitory. The second researcher made a show of trying to get the stick back through the bars of the dormitory. Would the chimp help her out by bringing her the stick? The researcher conducted 10 trials with each chimp. You can see the task in action at this link--watch Video 2. The journalist writes:

On average, the chimpanzees, who received no reward for cooperating, chose to hand the out-of-reach object to the researcher just over 50% of the time – but the rate of cooperation varied significantly among the individuals, with some helping in all 10 trials and others not at all. Chimps who chose to help more often also chose to help more quickly, supporting the conclusion that more cooperative individuals tend to make decisions faster than more selfish individuals do, the authors wrote.

Notice that there are two findings here. The first is the simple result of how many chimps helped (that's a frequency claim result).The second result is a correlational pattern. 

a.) What is the frequency claim you can make here? 

b.) Reread the description of the "stick" study (above) and decide what two variables are correlated.

c.) Placing these two variables on the x and y axis, sketch a scatterplot of the result that is described. (Compare your sketch to Figure 2a in the empirical journal article.) 

d.) Comment on the operationalization of the construct "Cooperation". Why might the researchers have chosen this operationalization? 

Here's a second way they studied chimp cooperation. This operationalization asks whether the chimps would punish individuals who don't follow rules:

When they had an opportunity to punish a thief who stole food from them, the chimps often chose to collapse the experimenter’s table [by pulling out the table's false leg] to prevent the thief from consuming the ill-gotten treats. 

e.) You can watch the "stealing" task here, under Video 3. What behavior (by the chimps) was used to operationalize "cooperativeness" here?  Why might this behavior be considered as an operationalization of Cooperation?

Finally, another task used in the study presented chimpanzees with two options for food. This task is viewable in Video 1 here: 

When presented with two sets of plates, one that contained a snack for both the chimpanzee and a familiar caretaker and another that had a treat only for the chimp, the apes were basically indifferent to whether the experimenter got something to eat. 

f.) The three operationalizations of cooperation in chimps found variable results. Two operationalizations showed some "cooperation," and one apparently did not. What does this pattern of results suggest to you about the importance of how well we operationalize variables? 

 

 

Reading vs. listening

Photo: Ladida/Getty Images

Audiobooks are more popular than ever. Have you ever wondered if there are cognitive differences between reading print books and listening to audiobooks? In this editorial, psychologist Dan Willingham explains some of the research comparing the two formats. Dr. Willingham describes several research findings, most of which come from experimental designs. 

Here's one point Willingham makes: 

...there are differences between print and audio, notably prosody. That’s the pitch, tempo and stress of spoken words. “What a great party” can be a sincere compliment or sarcastic put-down, but they look identical on the page. Although writing lacks symbols for prosody, experienced readers infer it as they go. In one experiment, subjects listened to a recording of someone’s voice who either spoke quickly or slowly. Next, everyone silently read the same text, purportedly written by the person whose voice they had just heard. Those hearing the quick talker read the text faster than those hearing the slow talker.

a. In the study above, what is the independent variable?  Was it manipulated independent groups or within groups (and what keywords in the description help you decide?)

b. What is the dependent variable? 

c. Sketch a small graph of the results described, labeling your axes mindfully.

 

Here is another study from Willingham's summary:

It sounds as if comprehension should be easier when listening than reading, but that’s not always true. For example, one study compared how well students learned about a scientific subject from a 22-minute podcast versus a printed article. Although students spent equivalent time with each format, on a written quiz two days later the readers scored 81 percent and the listeners 59 percent.

d. In this study, what is the independent variable?  Was it manipulated independent groups or within groups (and what aspects if the description helped you decide?)

e. What is the dependent variable? 

f. Sketch a small graph of the results described, labeling your axes mindfully.

g. Do you think you could have conducted this study as a repeated-measures design? Why or why not? 

h. What questions do you have about the internal validity of this study? 

 

Suggested answers

a. The IV is reading speed of the narrator during the first task; its levels are Slow and Fast. The phrase, "either spoke quickly or slowly" tells you this was manipulated as independent groups.

b. The dependent variable was how fast people read the text passage. 

d. Here the IV is reading a text or listening to a podcast. The results phrase refers to "readers" and "listeners," implying that the IV was manipulated as between groups (independent groups).

e. The DV is score on a comprehension quiz.

g. It might be difficult to conduct this exact study as a repeated-measures design, because people would have to process the same content aurally and via reading. Once they've processed the content once, it would undoubtedly be easier to process it a second time. 

h. You'd want to be sure that i) people were randomly assigned to the two conditions. This would control for selection threats; and ii) people in the two conditions read or listened to the exact same text (for example, a podcast and its transcript). This would help avoid a design confound.