Everyday Research Methods

Who's more likely to claim expertise?

The journalist's article was illustrated with a photo of a businessman. Is that the best choice, given the study? Photo credit: Yuri Arcurs/Getty Images

Are you familiar with proper numbers? How about declarative fractions? 

If you claimed that you knew these concepts, you're probably fooling yourself....because declarative fractions and proper numbers are not real math things. 

A Washington Post journalist summarized a study on who is more likely to falsely claim expertise, or as some folks might put it: to "BS." The article's headline reads, "Rich guys are most likely to have no idea what they are talking about, study suggests." 

What did the study do?

Study participants were asked to assess their knowledge of 16 math topics on a five-point scale ranging from “never heard of it” to “know it well, understand the concept.” Crucially, three of those topics were complete fabrications: “proper numbers,” “subjunctive scaling” and “declarative fractions.” Those who said they were knowledgeable about the fictitious topics were categorized as BSers.

The sample for the study was international:

The study drew from the Program for International Student Assessment, which is administered to tens of thousands of 15-year-olds worldwide. The test included a background questionnaire that captures demographic information, along with students’ attitudes toward the subjects they study in school. That section of the test included the questions about math knowledge.

Here is the pattern of results for gender (You can see tables of the data in the journalist's article).  

The data revealed that boys across all nine countries were significantly more likely than girls to pretend expertise, with the difference between the two working out to nearly half a standard deviation in some countries — a big gap, statistically speaking.

And here's the description of the results for income:

There’s also a significant class-based difference, with respondents from the wealthiest households showing a greater proclivity toward overstatement than those from the poorest.  

a) Here's a statistical validity question. When the authors say that the boys are about a half a standard deviation higher than girls in some countries and report it's "a big gap, statistically speaking," they are referring to effect size. In fact, if two groups are half a standard deviation apart, the effect size is probably about d = 0.5. Is that considered a small, medium, or large effect according to Cohen's conventions? 

b) Here's an external validity question. What were the age and income levels of the students in the study? Reflect on the headline, which refers to "rich guys," and the photo accompanying the story, which shows an adult man. What do you think--what external validity issues are raised here? 

c) Here's a construct validity question. The construct the journalist claims is being measured here is the "tendency to BS." How was that construct operationalized in this study? What do you think of that operationalization? Can you think of other ways to operationalize it? 

d) You might think we're ready for an internal validity question. However, internal validity doesn't seem relevant here, because nobody seems to be making a causal claim....or are they? Could you make a causal claim here? 

e) Advanced question: You might wish to investigate the table of data in the article that depicts the "BS gaps between boys and girls." Do you think that this table really needed separate columns for boys, girls, and "gap"? Why was it formatted this way? How else might you present these data? 

f) At the end of the article, the scientists are quoted about the study's implications and limitations. One of the study's original authors says, 

The authors also point out that their study was narrowly defined and restricted to the realm of mathematics. “Ideally,” they wrote, “future research should try to include a greater number of fake constructs in order to maximize precision of the bulls--- scale.”

Which of the four big validities seems to be addressed in this quote? 

How we spot leadership potential in men and women

Which one of these people will benefit most from sharing good ideas with the group? Credit: Dean Drobot/Shutterstock

Imagine you're working at a company and you'd like people to view you as a potential leader. How might you go about it? Perhaps you'd share your good ideas at meetings, hoping they'll notice your talent. Perhaps you'd pursue higher level training to improve your credentials. Maybe you'd point out ways your group's processes could improve. Would these strategies work? The answer may depend on your gender. 

The trouble is that people's prototype of a "leader" is a man. As this journalist describes, when people are asked to draw a picture of a leader, they almost always draw a male figure (by the way, that's a frequency claim). How might people's default assumption--that leaders are male--affect their ability to spot leadership potential in their colleagues? 

According to the journalist,

...getting noticed as a leader in the workplace is more difficult for women than for men. Even when a man and a woman were reading the same words off a script, only the man’s leadership potential was recognized.

Here is a study described in the piece. It's a factorial design (that is, it has two independent variables). As you read, decide what the two IVs are and what their levels are: 

In one experiment, participants were asked to call into a monthly sales team meeting of a fictional insurance company, during which they would hear from either an Eric or an Erica. Later they were asked to rate the speaker on the degree to which he or she had “exhibited leadership,” “influenced the team” or “assumed a leadership role.”

The Erics who spoke up with change-oriented ideas were far more likely to be identified as leaders [rated higher on the leadership questions] than Erics who simply critiqued their team’s performance. But Ericas did not receive a boost in status from sharing ideas even though they were exactly the same as the Erics’.

a) What is the first IV? How many levels does it have and what are they? Is it manipulated or measured? 

b) What is the second IV? How many levels does it have and what are they? Is it manipulated or measured? 

c) What are the three DVs? 

d) Sketch a well-labeled graph of the results of this study, using the DV of "exhibited leadership". Remember to put one of the IVs on the x-axis and the other IV as a line color or bar color. 

Here's another Factorial design described in the journalist's article. 

...in a second experiment, the researchers attempted to investigate the benefits of speaking up in the “wild.” The researchers asked participants in a 36-team skills competition at the U.S. Military Academy at West Point to fill out a series of questionnaires before, during and after their competition.

The authors acknowledged that there were limitations in studying this atypically male-dominated place, but the nature of the competition allowed them to control for athletic ability, cognitive ability and other factors.

Still, after the competition, when participants ranked who they wanted to be the team leader, only the men who had spoken up with ideas received a boost.

“It didn’t matter whether women spoke up 1) almost never, 2) rarely, 3) sometimes, 4) often, or 5) almost always,” Kyle Emich, a professor at Alfred Lerner College of Business and Economics at the University of Delaware, and one of the authors, wrote in an email. “Women did not gain status for speaking up, and subsequently were less likely (much less) to be considered leaders.”

e) One of the IVs in this study has FIVE levels--that's unusual, but it can happen!  What is this IV and what are its five levels?  Is this IV manipulated or measured? 

f) What is the second IV? How many levels does it have and what are they? Is it manipulated or measured? 

g) The DV is not explicitly stated in this summary. What do you think the DV was, based on this description? 

h) Sketch a well-labeled graph of the results of this study, using the DV of "being nominated by peers as being the leader". I suggest that you put the 5-level IVs  on the x-axis and the other IV as a line color or bar color. 

i) The first example (With Eric and Erica) was a 2x2 factorial design. How would you state the design of this West Point example?