Everyday Research Methods

Teaching with 100 Humans

In 2020, Netflix released an 8-episode series called "100 Humans" in which 100 (American) humans from a variety of ages and social groups participate in a series of studies. It's a gold mine of examples for critical thinking in research methods. You'll find examples of posttest-only, pretest-posttest designs, HARKing, ethics, and more. Most of the studies have flaws, but that makes them more fun to critique. 

I've cataloged ways you can use the series in your teaching: www.tinyurl.com/teachingwith100humans.

Reach out if you use it! Let me know how it works for you. 

 

Evidence-based tips for well-being during a pandemic

Dr. Santos provides evidence-based ways to enhance well-being during this pandemic. Credit: Yale University

How can people enhance their wellbeing during the Coronovirus pandemic? 

Dr. Laurie Santos (pictured) has been meeting with several news outlets, sharing evidence-based tips from her psychology of well-being class at Yale University. A version of the course is also offered on Coursera.

In the Twitter-embedded video here, produced by the World Economic Forum, Dr. Santos covers five tips for coping during the pandemic. In this space, I'm going to quote moments from the video, and you can practice classifying the claims and describing how the studies would have been conducted. The first three of these tips are also covered in this story from the World Economic Forum.

Tip 1: Keep social, even when socially distancing

Dr. Santos states in her segment that "Happy people spend more time with other people in general. They also prioritize time with their friends and family members"

a) What kind of claim is it to say that "happy people spend more time with other people in general"--frequency, association, or cause?  What are the variables in this claim? 

b) What kind of study was likely conducted to support this claim--correlational or experimental? 

c) Sketch a scatterplot with well-labeled axes that could depict this result.

Tip 2: Help others

To explain this tip, Dr. Santos mentions, "The research shows that doing random acts of kindness...can be incredibly powerful." For example, happy people are giving money to charity and volunteering in their communities. 

d) What kind of claim is it to say that "doing random acts of kindness...can make you happy"--frequency, association, or cause?  What are the variables in this claim? 

e) What kind of study would need to be conducted to support this claim--correlational or experimental? 

f) Can you imagine how you  might manipulate the variable, "doing random acts of kindness?"

Tip 3: Move--even if you can't leave your house

In this section, she says, "Exercise and moving your body is really correlated with improved well-being"

g) The verb, "correlated" gives this one away: It's an association claim. Dr. Santos specifies that they are "really correlated," which seems to suggest a strong effect size. What factors do we consider when we're trying to decide if a correlation (an effect size) is strong versus weak?

Tip 4: Savor everyday pleasures

"Research suggests that happy people tend to be more mindful. They tend to savor the present moment; they tend to be in the present moment"  Here Dr. Santos also strongly recommends taking deep breaths to calm the vagus nerve and connect to the present. 

h) How might you design an experiment to test the hypothesis that deep breathing can improve people's well-being during quarantine? Whom might you recruit for such a study? How would you operationalize each of the two variables? What results would you expect to find? 

Tip 5: Think about how you want to look back on this time

Finally, she shares one of my favorite tips for emotional regulation, psychological distancing: "Psychological distancing is the idea that if we think about ourselves and how we will have dealt with this crisis in the future...Think about how you want to tell your grandkids you lived through COVID-19, you don't want to be like, 'I panicked-bought and hoarded toilet paper..no, you want the story to be positive...'"

Experimental research has supported the causal claim that self-distancing--seeing yourself as though you are a fly on the wall, or even imagining looking back on yourself from the future--is associated with improved emotional regulation. It helps us take a cooler, less impulsive view of our emotions and process them instead of ruminating. For example, self-distancing helps people manage feelings of anger in constructive ways (here's a nice summary). 

i) Review the term translational research in your textbook (Ch 1) and explain how we might conduct translational research on psychological distancing during the pandemic. What is the basic research it is based on? What is the real-world context it's being translated to? And what kind of study could translate--that is, bridge the gap--between the two? 

 

 

 

 

 

 

Does this font make me look conservative?

Of these three candidates, Yang is considered more liberal than Klobuchar. Do the fonts used in their posters convey that? Photo credit: Kevin McGovern/Shutterstock

Here's something I never considered--different fonts lean conservative or liberal. 

Serif fonts have little curves and edges (like the "Amy" in the green poster here). Sans-serif fonts are cleaner, without these extra flourishes. 

A study showed multiple fonts to a sample of people and found that people rated the serif fonts as looking more conservative and sans-serif fonts as more liberal.

This research was covered by CNN.com. They summarized the results like this: 

For example, study participants saw Times New Roman as more conservative than Gill Sans. Blackletter, which looks like it belongs on a newspaper masthead, was seen as the most conservative font, while Sunrise, a cartoonish-looking script, was seen as the most liberal.

When they interviewed the researcher, she suggested, 

"If you think about serifs being used in more formal types of print or communications, maybe they're viewed as more traditional and sans serifs are viewed as more modern," Katherine Haenschen, an assistant professor of communications at Virginia Tech and the lead author of the study, told CNN. "There's a small but significant difference in how people perceive these fonts."

The researchers showed each font using a simple sentence, "The quick brown fox jumped over the lazy dog" and asked online participants to rate the font's conservative (vs. liberal) nature. For example:

The quick brown fox jumped over the lazy dog

The quick brown fox jumped over the lazy dog

The quick brown fox jumped over the lazy dog

a) In this study, what was the independent variable (IV)? What were its levels? What was the dependent variable? 

Here is CNN's description:

...987 participants were shown the phrase "The quick brown fox jumped over the lazy dog" in one of six fonts and styles: Times New Roman regular, Times New Roman bold, Times New Roman italic, Gill Sans regular, Gill Sans bold or Gill Sans italic.

b) Based on this description, was the IV manipulated as independent groups or within-groups? 

c) You could do the study either way. What are the advantages and disadvantages of manipulating the font variable as independent groups?What are the advantages and disadvantages of manipulating the font variable as within-groups?

d) What does it mean when the author says "There's a small but significant difference in how people perceive these fonts."? 

At the end of the story, the CNN journalist wrote about the potential applications: 

For example, if Bernie Sanders changed the font on his campaign materials, there probably wouldn't be much of a difference in how people see him because he's already a widely known figure.

The choice of font could, however, make a difference for a new candidate, like someone running for school board, town council or state legislature, Haenschen said. While researchers don't know for sure whether a font would change people's perceptions of a candidate, that's another question that could be explored in the future.

e) How might you conduct an experimental test of the journalist's speculation, above? 

You can see the empirical journal article here. Their Figure 1 shows ratings of conservatism and the 95% CIs for six fonts. 

 

 

Research on Native American mascots

The College of William and Mary has phased out its Indian-themed mascot over the years. In 2010, the College replaced its former mascot with "Griffin", above. Photo: Mitchell Layton/Getty Images

Since 2005, the NCAA has required colleges and universities to reflect upon, and if necessary, change sports team mascots that feature depictions of Native-Americans.  Now a team of psychology researchers have published an investigation about support for such mascots among different groups of people. The set of studies  includes both correlational and experimental designs, including one testing whether the mascots can affect people's donations to the universities. 

The lead researcher, Dr. Michael Kraus, told a journalist what led to his interest in the research question. 

“I was at the University of Illinois at the time, and despite their Native American mascot being officially discontinued in 2007, it was surprising to see the mascot on campus in 2013. The continued presence said something about the norms on campus, and what kinds of students feel safe and welcome there,” Kraus said.

The journalist summarized two studies conducted by Kraus and his team. Here's the first:

In a survey of 201 students, the researchers found that those with lower levels of explicit prejudice toward Native Americans reported lower levels of belonging at the University of Illinois.

In other words, students who disagreed with statements such as “It is now unnecessary for the U.S. government to honor their treaty obligations to Native tribes” also tended to disagree with statements such as “I identify with the university” and “I show school spirit regularly.”

In addition, students with higher levels of explicit prejudice, as well as white students, were more likely to have positive views of the retired “Chief” mascot.

Questions

a) This is a bivariate correlational study. What are its two main variables?

b) The journalist gives some clues about how each of these two variables was measured. What kinds of measurements were used--self-report? physiological? observational? What do you think of the items? 

c) Sketch a scatterplot of the results that are described. Label your axes carefully.

d) Can this study support the causal claim? Why or why not? (Apply the three causal criteria; when you come to internal validity, specify at least one third variable that might be associated with both measured variables.)

The empirical article contains several studies on this topic. One of them tested the impact of  the University of Illinois' retired mascot, Fighting Illini, on donations to the school. 

The experiments showed that stereotypic Native American imagery “reduced actual donations to the institution by 5.5%, and was particularly likely to decrease (increase) donations among those low (high) in prejudice,” the researchers wrote.

In the study, researchers had online participants read about different universities, one of which was the University of Illinois. Some participants  were randomly assigned to see the Native American mascot, Chief Illini, next to the school name, while some did not.  Then participants had the chance to distribute $2 in donations to the four universities. When people had been exposed to the  Native American mascot, donations to the University of Illinois decreased. 

But like all research, the study includes some caveats. “Probably the biggest one is that we really only studied these dynamics at the University of Illinois, but we expect these norms to operate similarly everywhere. Someone, maybe us, needs to do that additional work,” Kraus said.

More Questions

e) Is the donation study an experiment or a correlational study? What are its two main variables?

f) When Kraus states that "we expect these norms to operate similarly everywhere," to which of the four big validities is he referring?

You can access the empirical research here.

Instructors: The complete package of studies is a complete set with a frequency claim (via an observational study of mascot use on campus) , a correlational study, and an experimental study.

 

 

 

 

Interested students might learn from Dr. Stephanie Fryberg's work, showing how American Indian high school students respond to being reminded of professional sports mascots that feature American Indians.

Fidget spinners seem to cause worse memory and attention

The fidget spinner IV was manipulated as independent groups in the first study and as within groups in the second. Photo: Marcel Jancovic/Shutterstock

Fidget spinners were extremely popular a couple of years ago. When people promoted them, they often suggested that the toys could help people with ADHD focus and learn more. At the time, such claims were not scrutinized empirically (here's a nice summary of this lack of evidence). It takes time for empirical research to be conducted, analyzed, peer-reviewed, and published, and even though fidget spinners were hot in 2017, we now have a study that tested the effect of using a fidget spinner during a learning task. At least one press outlet covered this study. Here's their summary:

In the first experiment, 98 undergraduate students watched a 15-minute educational video lecture on baking bread while either using a fidget spinner or not using a fidget spinner [randomly assigned]. In addition, half of the participants not using a fidget spinner were randomly assigned to watch the lecture while sitting near someone using a fidget spinner.

A fill-in-the blank memory test showed that participants who used fidget spinners answered significantly fewer questions correctly about the material covered in the video. Participants who used fidget spinners were also more likely to report “zoning out” and having difficulty staying on task while watching the video.

Participants who were sitting near someone using a fidget spinner, however, did not appear to be affected.

Questions

a) This was an experiment. How do you know?

b) What were the IV and DV? How many levels were in the IV and what were they? Was the IV manipulated as independent groups or within groups? 

c) Which of the four simple experiments is this: postttest-only? Pretest-posttest? Repeated-measures? Concurrent measures? 

d) The journalist attached a causal claim to these results, saying "study finds using a fidget spinner can impair memory and attention." Apply the three criteria for causation to this claim to explain why the causal claim is probably valid. How does random assignment assist this study's internal validity?

e) Ask one question each about the study's statistical, construct, and external validities. 

The researchers replicated the effect in a second study. The wondered if fidget spinners might only work among people who believed in their powers: 

The researchers then conducted the experiment again. However, this time around they specifically recruited 48 undergraduates who believed that fidget spinners could help them focus in class.

But the results were the same.

According to the empirical journal article, the second study on 48 students was conducted as a repeated measures design, using two videos. Students used a fidget spinner during one video and no spinner during the other one. Students were tested on their recall for the videos after each one.

f) What are the IV and DV in the second study? How many levels were there in the IV?

g) How do you think the researchers used counterbalancing in this second study?  Why would that have been important? 

You can access the original empirical article here, though you may need to log in via your university library. 

 

 

 

 

 

Racial stereotypes behind how we explain school shootings

Participants and newspapers were more likely to blame video games when a school shooter was white than when he was black. Photo: Richard Levine / Alamy Stock Photo

What causes mass shootings? Of course, we have no experimental evidence on this question. Correlational evidence (both within the U.S. and evidence that compares the U.S. to other countries) is consistent with the argument that gun availability is a factor. 

However, this post isn't about what really causes school shootings. It's about what people think causes them. And it turns out that the race of a school shooter influences what people believe about the shooter. 

After some school shootings (a subset of mass shootings), people have sometimes looked for answers not in the availability of guns, but in video games. Now, there is actually no evidence linking video games to mass shootings. However, this ABC news story talks about when people are more likely to think that video games are responsible: 

There's little evidence showing a relationship between violent video games and school shootings, but video games are often blamed for overall violence in our communities. A recent study published in the Psychology of Popular Media Culture set out to find if we look for an external explanation for violence by whites more often than we do for African Americans.

"We try harder to make excuses for white perpetrators," said investigator Dr. James Ivory, a professor in the Department of Communications at Virginia Tech, in an interview with ABC News.

The first study gave people a mock news story about a school shooting. When asked what caused the shooting, people were more likely to blame video games if the shooter was white than if the shooter was black.

It turns out that in the original study (linked here;  if it's paywalled you may be able to access via your university library), participants read the mock news story and were randomly assigned to one of two photos of the perpetrator--a mugshot showing either a Black or a White man. 

a)  There were two variables in this study. Name each one. For each, was it manipulated or measured?

b) What kind of study is this, correlational or experimental?

c) Sketch a graph of the results of this study. What will you put on the y-axis? 

d) In the study above, what role does random assignment play? Why did the researchers randomly assign people to see either a Black or White perpetrator?  

 

Here is how ABC news described the second study:

The second study looked at thousands of news articles -- video games were mentioned more often in stories of white shooters [compared to black shooters]. 

e) What are the variables in the study above? Name each one; for each, was it manipulated or measured?

f) What kind of study is this, correlational or experimental?

g) Sketch a graph of the results of this study.

h) In the study above, the researchers could not randomly assign news article authors to write about a black vs. a white perpetrator. How does that change the impact of this study? 

i) Which of the two studies do you think provides the strongest evidence that race influences people's willingness to think video games are a cause of the shooting?

 

Suggested answers

a) One variable is race of the perpetrator (levels are Black or White), this was manipulated. The other variable is "likelihood to blame video games". This was measured.

b) This was an experiment because one variable was manipulated. 

d) Random assignment contributes to the study's internal validity. They can be sure that the same kinds of participants were in each group (reading about Black vs. White perpetrators). Therefore, people who play a lot of video games themselves were equally likely to be in the two groups. You might be interested to know that in the original study, the researchers also pretested the Black and White photos they used to be sure that the faces differed on race but not on perceived attractiveness or dangerousness. This pretesting also contributed to the experiment's internal validity.

e) The first variable was race of the perpetrator in the article (Black or White), which was measured. The second variable was whether or not the article mentioned video games (yes or no).

f) This was a correlational study because all variables were measured. 

h) Because this study used naturally-occurring crimes, the researchers may have an internal validity problem. It is possible that the Black and White perpetrated shootings differed not just on race but also on number of victims, age of victims, geography, or some other variable. This introduces an alternative explanation for the result--perhaps people blamed video games for White perpetrated shootings because they had more victims, had older victims, and so on.

i) You might have selected the first study as the strongest because it was an experiment and allows us to make a causal inference. Or you might have selected the second study because it was conducted on "real world" stimuli such as news stories--therefore it appears to have good ecological validity or external validity. 

Thanks to Marianne Lloyd for this idea!

Can Probiotics Actually Treat Obesity?

 

Should you be suspicious of a study conducted by a company that produces probiotics products? Photo: Science History Images / Alamy Stock Photo

Guest post by Dr. Dina Gohar, University of Michigan

 

Hippocrates, considered to be the Father of Medicine, reportedly said, “all disease begins in the gut. ” It took doctors centuries to start studying the connection between our gut microbiota and our health, which is now well-supported by research. We can improve the health and diversity of our gut microbiota with active strategies, such as exercising regularly, eating a healthy diet, and getting enough sleep. We might also be able to improve our gut microbiota by simply consuming live “good” bacteria and microorganisms called probiotics. Probiotics are now sold as capsules, pills, and powders, added to fruit juices, granola bars, and candy, and even put into cosmetics. Fermented foods, such as cultured yogurt, sauerkraut, and kombucha, contain them too. A cloud of hype and hope surrounds probiotics, which have been touted as a treatment (and means of prevention) for everything from intestinal disorders, mood and anxiety disorders, belly fat and obesity-related disorders. 

A recent article in the New York Times suggested that a probiotic of heat-killed bacteria can treat metabolic disorders like obesity. This suggestion was based on a study in Nature Medicine.The  New York Times journalist summarized the study as follows:

Researchers tested the bacterium, Akkermansia muciniphila, in 32 men and women who met the criteria for metabolic syndrome by having at least three of five conditions: high fasting blood sugar, high blood pressure, high triglycerides, low HDL (the “good” cholesterol) or excessive waist circumference.

…In a three-month trial, volunteers were randomized to one of three groups: daily tablets containing live bacteria, pasteurized bacteria [A. muciniphila] or a placebo. Compared with the placebo group, those who took pasteurized A. muciniphila had significantly improved insulin sensitivity and total cholesterol, and decreases in several blood markers of inflammation and liver dysfunction. They also had decreased body weight, fat mass and waist circumference, though those differences were not statistically significant. The live bacteria were largely ineffective.

1. This study was an experiment. What is the independent variable (IV) and what are its levels? What are the dependent variables (DV)? What do you think of the outcome measures chosen for this study?
   
   
2. How was the independent variable manipulated:  between or within groups? Which of the four basic experiments is it probably an example of: Pretest/postttest design? Posttest-only? Repeated measures? or Concurrent measures?
   
   
3. Can the results of this study be used to support the causal claim that “Taking pasteurized bacteria can improve people’s insulin sensitivity and total cholesterol?” Why or why not? (Apply the three criteria for causality.)
4. Look at the sample size of this study; do you think it’s a problem? Why or why not? Which of the four big validities does the sample size best fit into? 
5. Do the results above support the claim that probiotics like A. muciniphila can treat obesity? Why or why not? (Is internal validity relevant here? What about construct validity?) Can you conclude that probiotics can help you lose weight? Why or why not?
6. The New York Times article critically failed to mention that the researchers who conducted this study have a patent on the probiotic treatment and even started a company based on it. What do you think about this potential conflict of interest? 
7. Do you think it matters that this study’s somewhat promising findings were only with dead bacteria, whereas probiotics are generally defined as being live bacteria? Why or why not? (Which of the four big validities is being asked about in this question, if any?)
   
   

You might agree that the probiotic study described by this journalist has a few serious problems. For an example of probiotic research done right, check out this NPR news story on a current clinical trial examining if a specific probiotic could eliminate specific GI problems in children on antibiotics (courtesy of Dr. Jess Hartnett’s statistics blog).

Tilt your head down to appear more intimidating

According to the study, Captain Marvel's eyebrows have something to do with her dominant pose. Credit: Collection Christophel/Alamy Stock Photo

Captain Marvel looks tough. Perhaps one reason is the tilt of her head, which enhances the appearance of dominance, according to a series of studies that appears in Psychological Science. The studies were summarized by journalist Alice Walton for Forbes. 

Here are some excerpts from the journalist's description of the study:

...there’s a way to communicate dominance that doesn’t involve moving the facial muscles at all: just tilt your head downward slightly and maintain eye contact.

In one experiment, participants in the University of British Columbia study rated [several] avatars who were tilted upward 10 degrees, tilted downward 10 degrees, or level (neutral). Then they rated statements designed to gauge how dominant the person appeared, including “This person would enjoy having control over others,” and “This person would often try to get his way regardless of what people may want.”

Participants rated the downward-tilting heads significantly more dominant than either of the other two positions. When the experiment was repeated using photos of actual people rather than avatars, the same pattern was found.

a) The journalist labels this an experiment, which it really is. What is the independent variable? What are its levels? Is the IV manipulated as independent groups or within-groups? (By the way, the Forbes article shows photos of the three levels.)

b) What is the dependent variable in this study?

c) What's the experimental design: Posttest only? Pretest-posttest? Repeated measures? Concurrent measures? 

d) Sketch a bar graph of the results described here: "Participants rated the downward-tilting heads significantly more dominant than either of the other two positions."

e) What do you think of the construct validity of the dependent variable? Does this operationalization really get at "dominance?" Why or why not? What questions from Chapter 5 might you ask here? 

f) The journalist mentions that "the experiment was repeated using photos of actual people rather than avatars". It's an example of replication. Which type of replication study would you call it: direct replication? conceptual replication, or replication plus extension? (Chapter 14 defines these terms.)

g) By replicating the avatar study on actual people, which validity (or validities)  are these researchers enhancing? 

More Advanced Questions

After establishing the basic effect, the researchers conducted some additional studies to test the mechanism, or mediator. Read this paragraph (written by the journalist) to decide what the mechanism was, and think about how they tested it. 

In another experiment, the team found that the key to the effect was the eyes and eyebrows: when they showed participants faces with the eyes/eyebrows occluded, the tilted head/dominance effect disappeared. But then they showed participants pictures of just the eyes and brow portion of the face, the effect was present, which suggests that this part of the face is both necessary and sufficient—meaning it's not only required but is all that's necessary to elicit the effect.

The above argument suggests that the researchers' theory goes like this:

    Tilting head downward ---> changes in eyebrow position --> dominant appearance

h) To find evidence for their theory, the authors tested four conditions in a 2x2 factorial design. Downward tilt (vs. neutral), and face only vs. eyes/eyebrows only. You can see examples of the stimuli they used in Figure 4 of the original article in Psychological Science. Sketch the predictions of these two studies. Like this:

If changes in eyebrow position are important for tilted faces to appear dominant, then the results should show [sketch a result].

If changes in eyebrow position are NOT important for tilted faces to appear dominant, then the results should show [sketch a result].

 

Best toys for 2-year olds

In one of the conditions of this experiment, parents interacted with their young children while playing a shapes game on an ipad. What was the other condition? Photo credit: Gettyimages

For young children, play is school: Children can learn math, physics, language, social skills, and more. And when caregivers interact with them, children at play can learn even more about their world. Caregivers can direct children's attention and teach them new words and concepts. Research from the laboratory of Temple University professor Dr. Nora Newcombe studies children's cognitive development. In particular, this group have studied what children learn about shapes and spatial information from playing with toys with their caregivers. 

Here's a one-minute video clip that portrays an experiment conducted in their laboratory. The study tested how caregivers and children interact when playing with different kinds of spatial toys. Watch the video with the following questions in mind. 

a) The video describes an experiment. What is the independent variable (IV)? How many levels of this IV were there, and what were they? 

b) What are the dependent variables (DVs)? (There are at least two DVs). 

c) Can you tell, based just on this description, if the IV was manipulated as independent groups or within groups?

d) Sketch a graph of the study's result. Would you use a bar graph or line graph?  

e) Sometimes a journalist pairs pre-existing stock video (i.e., video filmed for different purposes) with a story about a research study. Using other videos may be necessary because scientists do not release videos of their research participants for ethical reasons. How well do you think the videos that accompanied the story matched the study's design? 

f) What questions should you ask about this study's construct validity? internal validity? statistical validity? external validity? 

Who's more likely to claim expertise?

The journalist's article was illustrated with a photo of a businessman. Is that the best choice, given the study? Photo credit: Yuri Arcurs/Getty Images

Are you familiar with proper numbers? How about declarative fractions? 

If you claimed that you knew these concepts, you're probably fooling yourself....because declarative fractions and proper numbers are not real math things. 

A Washington Post journalist summarized a study on who is more likely to falsely claim expertise, or as some folks might put it: to "BS." The article's headline reads, "Rich guys are most likely to have no idea what they are talking about, study suggests." 

What did the study do?

Study participants were asked to assess their knowledge of 16 math topics on a five-point scale ranging from “never heard of it” to “know it well, understand the concept.” Crucially, three of those topics were complete fabrications: “proper numbers,” “subjunctive scaling” and “declarative fractions.” Those who said they were knowledgeable about the fictitious topics were categorized as BSers.

The sample for the study was international:

The study drew from the Program for International Student Assessment, which is administered to tens of thousands of 15-year-olds worldwide. The test included a background questionnaire that captures demographic information, along with students’ attitudes toward the subjects they study in school. That section of the test included the questions about math knowledge.

Here is the pattern of results for gender (You can see tables of the data in the journalist's article).  

The data revealed that boys across all nine countries were significantly more likely than girls to pretend expertise, with the difference between the two working out to nearly half a standard deviation in some countries — a big gap, statistically speaking.

And here's the description of the results for income:

There’s also a significant class-based difference, with respondents from the wealthiest households showing a greater proclivity toward overstatement than those from the poorest.  

a) Here's a statistical validity question. When the authors say that the boys are about a half a standard deviation higher than girls in some countries and report it's "a big gap, statistically speaking," they are referring to effect size. In fact, if two groups are half a standard deviation apart, the effect size is probably about d = 0.5. Is that considered a small, medium, or large effect according to Cohen's conventions? 

b) Here's an external validity question. What were the age and income levels of the students in the study? Reflect on the headline, which refers to "rich guys," and the photo accompanying the story, which shows an adult man. What do you think--what external validity issues are raised here? 

c) Here's a construct validity question. The construct the journalist claims is being measured here is the "tendency to BS." How was that construct operationalized in this study? What do you think of that operationalization? Can you think of other ways to operationalize it? 

d) You might think we're ready for an internal validity question. However, internal validity doesn't seem relevant here, because nobody seems to be making a causal claim....or are they? Could you make a causal claim here? 

e) Advanced question: You might wish to investigate the table of data in the article that depicts the "BS gaps between boys and girls." Do you think that this table really needed separate columns for boys, girls, and "gap"? Why was it formatted this way? How else might you present these data? 

f) At the end of the article, the scientists are quoted about the study's implications and limitations. One of the study's original authors says, 

The authors also point out that their study was narrowly defined and restricted to the realm of mathematics. “Ideally,” they wrote, “future research should try to include a greater number of fake constructs in order to maximize precision of the bulls--- scale.”

Which of the four big validities seems to be addressed in this quote?