Everyday Research Methods

Does "virtue signaling" change minds?

If a lot of us send virtue signals, it can communicate to others that norms are changing. Credit: SEEphotos7171/Shutterstock"

Among people fighting for racial justice, "virtue signaling" may sometimes be derided as empty action. "Virtue signals" are public messages of support for a social justice cause. They could take the form of a yard sign, a t-shirt, or a social media post. Critics sometimes argue that such messages are easy and cheap. It's easy for a company to put a message on their home page that supports Black Lives Matter. It's not hard to wear a t-shirt that supports a social justice cause. But are these expressions of support all talk and no action?

An article in Time magazine argues that virtue signaling can be more powerful than we think. The authors, psychological scientists Jamil Zaki and Mina Cikara, integrate several studies to make their argument. Here's part of their introduction: 

It’s critical to question people whose most radical act is temporarily changing their profile picture. It’s critical they follow their words with action; the time for talk has passed. But focusing solely on virtue signalers obscures their most important role. Like radio waves, the signals they send are received by someone, and those receivers matter, too.

Zaki and Cikara argue that virtue signaling is an important way of influencing others. If a lot of people change their profile pictures to support BLM, for example, more people will start to believe that this movement is popular and they may adopt it, too:

Opinions can work this way: when a particular viewpoint gets a lot of attention, people assume it’s popular, and shift towards it. We receive signals, and are changed by them.

What studies support their argument? Here's one that the authors summarize: 

As social norms shift, individuals shift with them: adopting popular opinions and behaviors, and dropping ones that fall out of style. Norms are especially powerful as they gain steam. In one set of studies, Gregg Sparkman and Greg Walton presented diners in a cafeteria [either] with evidence that 30% of the U.S. population was vegetarian, or that 30% of the population was newly vegetarian. People who learned about this second, “dynamic social norm” were twice as likely to order a meatless lunch themselves. They saw not just where the crowd was, but where it was going, and didn’t want to be left behind.

Here are some questions about the study summarized above:

a) Was this an experiment or a correlational study? How do you know? (If it's an experiment, what kind of study is it?)

b) What were the key variables in the study? Were they manipulated or measured? 

c) Sketch a small graph of the results being described. Would a bar graph or a scatterplot make more sense?

Here's another study Zaki and Cikara introduce:

Signaling and conformity are not inherently positive. In 2016, the U.S. elected a president whose campaign was jammed with racist talking points. Prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants). 

d) This summary doesn't really help you figure out how the study was designed. But here's a chance to think carefully: What kind of study would we need to conduct in order to test the claim that "prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants)."?  There are three variables you would need to include in your study. What are they? 

 

Suggested answers to selected questions

a) This was probably a posttest-only experiment, because it sounds like they manipulated the message about vegetarianism and measured what foods people took in the cafeteria. Here is a link to the original empirical article. 

d) The key dependent variable in this study is prejudice levels, so that's the first variable. Then, in order to establish that "prejudice increased", you'd need to measure prejudice both before and after the election, so that's another variable. Finally, in order to show that it increased specifically toward groups he targeted, you'd need to measure different types of groups (Muslims, immigrants, as well as other potential groups that Trump did not deride in his campaign). You can review the Crandall article here. Do their results really support what Zaki and Cikara are saying about it? (Pay attention to the difference between people's actual prejudice and the normativity of prejudice.) 

 

 

 

 

Racial bias in psychological science journals

A recent study found that in psychology, the authors, participants, topics, and editors of empirical journal articles are mostly White.
Photo credit: Africa Studio/Shutterstock

Is science fair?

You learned in Chapter 2 that empirical journals are the primary place where psychological scientists make their research public. Each empirical journal has an editorial board-- a group of scientists who have power to select the peer reviewers for each article and make the final decision about publication. 

Each step of the way, the stated goals of the publication process are to be fair, to focus on the quality of the science, and to publish the best science possible. But is the process fair?  In light of the renewed energy around anti-racism, a timely new study sheds light on racial diversity in the psychology publication process. The researchers asked, Is race an important variable in the scientific publication process?

The team coded all empirical journal articles that appeared in the most prestigious journals in psychology (over 26,000 articles). There were two journals each from social, cognitive, and developmental psychology. The authors measured:

• how often race was studied as a subject in these articles
• the race of the participants in the studies
• the proportion of the authors of scientific articles were White vs. Non-white
• the proportion of the editors of the journals themselves were White vs. Non-white

Here are some of the results of this published study of published studies, as summarized by the Association for Psychological Science's trade publication:

The results suggest that there is a hierarchy in psychology research that determines who and what gets published. “We are not saying anyone has bad intentions,” said Roberts [one of the study's authors]. “It’s a systemic problem across many fields, and psychological science is no exception.”

First, across the past five decades, articles in psychological journals that highlight race have been rare, and although developmental and social psychology journals have published a growing number of these articles, they have remained virtually nonexistent in cognitive psychology.

Second, most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism.

Third, many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color.

Let's parse these claims. 

Questions

a) What kind of a claim is it (frequency, association, or cause) to state that "across the past five decades, articles in psychological journals that highlight race have been rare"

[And what's "rare"? The published report says that "only 5% of publications highlighted race" (p. 4). This 5% figure is an average across cognitive journals (less than 1%) and social (5%) and developmental journals (8%)]

b) Let's take the second claim, "most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism"

The quoted text above provides two claims in one sentence. What is the frequency claim in the quoted text? What is the association claim in the quoted text? What are the two variables in the association claim?

c) Now for the third claim, "many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color."

As with b), the quoted test provides two claims in one sentence, a frequency claim and an association claim. What are these? 

d) In Chapter 1 of the 4th Edition (available Fall 2020), you will read about Merton's norms of science--a set of values that the community of scientists usually endorses and strives  to follow. Here they are, with short descriptions (for a full review, see Chapter 1 or this summary). 

Universalism: Anyone can do science as long as they follow the accepted criteria for science, which are established in advance, are objective, and are accessible to all.
Communality: Share! The process of science, including data, methods, and products, should be free and open for all to use. 
Disinterestedness: The quality of the science should be the only thing that matters, not the prestige of the person or place it was conducted. 
Organized Skepticism: Verify everything. Critical peer review matters, every study should be replicated, and criticism of one's evidence is expected and welcome. 

Roberts and his coauthors argued that psychology is too White. Which of Merton's norms does this result seem to violate, and why? 

Suggested answers:

a) This is a frequency claim--a statement about the frequency of the variable, "highlighting race" across 26,000 journal articles.

b) The frequency claim is about the proportion of journal editors who are White vs. a Person of Color (POC). The association claim is about a link between two variables: whether a journal has a White editor or not, and whether that journal published articles highlighting race or not.

c)  The frequency claim is about the proportion of article authors who are White vs. POC. The association claim is about a link between two variables: whether the article was written by a White author or not, and whether that article included POC as participants or not.

d) Open to discussion!

Here is a summary of this work.

Testing a COVID drug: Design flaws and ethical lapses

Photo credit: Alissa Eckert, MS; Dan Higgins, MAMS/CDC

Here's a good example of a bad study. It's related to the COVID pandemic. In this story by NPR  journalist Vanessa Romo, you can read about a doctor in Texas who claims to be conducting research on the drug hydroxychloroquine, using COVID patients in a nursing home as participants. The doctor's research includes both design problems and ethical problems.

According to the journalist, 

...a doctor at a Texas nursing home started giving the anti-malaria drug hydroxychloroquine to dozens of elderly patients diagnosed with COVID-19 and tracking the outcomes in what he's calling an "observational study".

According to the doctor conducting this study, 

"It's actually going well. People are getting better," Armstrong told NPR, adding that after just a handful of days, some of the 39 patients on the medication are showing signs of improvement.

Armstrong admits it is difficult to quantify how much of his elderly patients' improvement is due to the malaria drug or how they would have fared without it. Nor can he explain why other patients are not responding to the tablet doses, though he notes many are only halfway through the five-day cycle.

Design questions

a) Why can't the doctor decide "how they would have fared without it"? Your answer should discuss the study's need for a comparison group. 

b) The doctor's study includes one group--all of the patients in the "study" are taking the drug. What kind of comparison group will be needed to conduct a true clinical trial?

c) If you're as far as Chapter 11, you'll recognize the design of this doctor's study. What is it?

Here's another quote from the article:

[The doctor] said he started by screening those he believed would benefit most and added more people each day. He monitored their blood oxygen saturation, temperatures and how well they were breathing. "The people who [I decided to put] on it were getting sicker but were not so sick that they had to go the hospital," Armstrong explained.

d) What design problems, if any, to you see with the doctor selecting only this type of patient for being in his "observational study?" And, what might have happened if he had included more severely ill patients for his study? (How might the results change?)

Ethical questions

According to federal guidelines and the ethical principles of the Belmont Report, when a person claims to be doing "research," they should obtain ethical approval from an IRB. Ethical approval would require such procedures as obtaining informed consent (respect for persons), minimizing harm (beneficence), and selecting participants fairly (justice). Read these next quotes from the story:

He acknowledged that some families were not aware their relatives were put on the drug, saying that "for the most part," he consulted with each nursing home resident prior to giving them on the tablets.

While the "overwhelming majority of them are awake and alert and can actually have a conversation," Armstrong said some suffer from middle stages of dementia. In some cases, he did not discuss prescribing the tablets with anyone at all before doing so. He said it is common for physicians to prescribe new medications to patients without explicit consent from the patient or family members. "It's not required," he said.

e) Review the sections of Chapter 4 on "respect for persons," "beneficence," and "justice," and apply details from these concepts to this example. 

f) Do you think study design flaws such as the ones you've noted in a), b) and c) should be considered ethical violations, too? Why or why not? 

Safety questions

A potential issue with this doctor's decision concerns the safety of the drug hydroxychloroquine: According to the journalist:

The Food and Drug Administration has not approved the drug for the treatment of COVID-19. 
and
[the drug] is known to have serious negative health impacts. That is why so many in the medical community worry about prescribing it without such proof.

g) The safety of the drug is relevant to the ethical principle of beneficence. Can you explain why?

Finally...

Another wider ethical issue (perhaps outside the scope of a research methods course) concerns how the doctor obtained the drug, which has been in short supply:

[The doctor] began making calls to track down a source for the medicine, which is in short supply.That's when his political connections proved useful.

Armstrong, who is a prominent GOP activist, called Republican Lt. Gov. Dan Patrick. He says Patrick reached out to Texas state Sen. Bryan Hughes, also a Republican, who knew someone on the board of the New Jersey-based company Amneal Pharmaceuticals. The company, which makes and distributes the drug, has donated more than a million tablets nationwide, including to the statesof Texas and Louisiana.Two days later, Armstrong had received more than enough medication to begin giving it to patients. 

h) Do you think it is reasonable for a doctor to use political connections to obtain medications that are in restricted supply? Why or why not? 

Suggested answers to selected questions

a) and b) and c) Because the doctor's "study" has only a group of patients who are taking the drug, he doesn't have a comparison group. In Chapter 11 this is called a one-group, pretest-posttest design, otherwise known as the Really Bad Experiment. The design is flawed. If patients improve on the drug, the doctor will not know whether their improvement was caused by the drug itself, or simply because their bodies were healing or because they were getting excellent nursing care, or even some other reason. 

The doctor's behavior in this COVID study reminds me of the Halstead radical mastectomy, which for decades was the dominant treatment for breast cancer in the U.S.. Surgeons at the time were so convinced that this treatment was the best one for breast cancer that they never bothered to compare it to another treatment--they never tested its efficacy. (When a randomized control study was finally conducted in the 1970s, it found that the surgery was no more effective than other, less invasive treatments.)

d) By selecting only relatively less sick patients, the doctor is stacking the deck in favor of finding evidence that the drug is effective. Many of the patients are likely to get better, so the doctor may conclude that the drug has worked. But of course, without a comparison group of equally healthy people who are not getting the drug, we cannot know if the patients would have recovered anyway.

e) f) g) The lack of informed consent--especially for vulnerable patients who have Alzheimer's--is a clear violation of the respect for persons principle. And the use of a risky treatment, one that is known to be dangerous to people with heart conditions, is likely a violation of the beneficence principle. I would argue that when a researcher conducts a flawed study like this, it also a violation of beneficence--the study puts participants at risk, while the benefit of the potential information gained from the research is negligible. We might also argue that the study violates justice: by studying only people at a nursing home, these patients bear the burden of being research participation when anyone can benefit from this research (assuming, of course, that the research were actually well-designed).

Many thanks to Dr. Stephen Chew for pointing me to this example!

Post script: Looking for more examples of bad studies, as described by science writers? Here are two:

Here's a critique of the many problems with another early published study on hydroxychloroquine.

Here's critique of studies supposedly investigating the death rate of COVID-19. 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers. 

 

Dog people live longer

The study found an association between dog ownership and longevity. Which third variables were controlled for? Which third variables might still be responsible for the relationship? Photo: Shutterstock

I can't resist blogging about research on dogs. Here's a story on NPR that summarized several studies that have tested the correlation between longevity and dog ownership. The journalist introduces the topic this way: 

Dogs and humans that interact with one another get a jolt of oxytocin, the so-called "cuddle hormone." And, if you get to look at dogs and hug them every day, you just might live longer than people who don't have to clean animal hair off their clothes, according to a pair of studies out this month.

Here is a summary of one of the studies:

[researchers] looked over a 12-year period at dog owners in Sweden who have cardiovascular disease. They adjusted for the kinds of things we know affect cardiovascular health — age, demographics, socioeconomic status, marital status, number of children at home — and even after adjusting for all of that, they found a benefit of dog ownership [on longevity].

....they followed over 100,000 people and found a 21% reduction in deaths from any cause.

The study above used multiple regression. 

a) What was the dependent (criterion) variable in this study?  What were the predictor variables? 

b) Sketch a little regression table with results that would be consistent with what is mentioned above. Put the criterion variable at the top, and list the predictor variables. Which betas would be significant here? 

c) The researchers controlled for age, demographics, SES, marital status, and children--but regression doesn't control for all third variables. What else could be associated with both dog ownership and longevity? 

d) They mention a "21% reduction in deaths from any cause."  Which of the four big validities is this concerning? 

One of the experts who was interviewed for this story said:

In particular, I found it very convincing and striking that the benefits seems to be larger among individuals who live alone compared to multiperson households. That suggests the companionship of a dog is possibly very important to their heart health.

e) This is a moderator! Identify the variables in this relationship and complete this sentence: ____ moderates the relationship between ____ and _____.

 

In part of the article, the journalist presents quotes from an expert, who seems to equivocate on the causality issue:

"When you look at the big picture and look at all the evidence around dog ownership and cardiovascular health, it's pretty clear the signal is real and likely causal," he says.

In other words, he's convinced getting a dog improves your health — especially for those with heart disease.

The research is not definitive, though, Kazi adds. "They're not randomized trials, the gold standard for what we would do to evaluate a new drug," he says.

f) It is frustrating to read that the expert believes the "signal is real and likely causal" and then admits that "it's not definitive." You can set this expert straight. Why can't this correlational study support causation?  Apply covariance, temporal precedence, and internal validity. When you address internal validity, check out answer c) above!

 

 

Suggested answers

a) The criterion variable is longevity (being alive at the end of the study). The predictor variables are dog ownership, age, demographics, SES, marital status, and children living at home. 

b) The table might look like this:

DV: Longevity (or being alive after 12 years)

IV                            Beta (sig)
Dog ownership         .12*
Age                            -.11
SES                              .13*
Married                        .08*
Children at home        .07*

d) Current health and mobility might be associated with both dog ownership and longevity. After all, you  may need to have a certain level of health and mobility to own a dog, and healthier folks are more likely to be alive at the end of the study. 

e)  Living status (being alone or married) moderates the relationship between dog ownership and longevity.  

f) There is covariance--there is a link between owning a dog and living longer. 

There is temporal precedence-- the study took place over several years and dog ownership was measured first, and death rates later. 

There is not full internal validity. They controlled for age, demographics, SES, marital status, and children living at home, so they can rule those out. But they did not control for everything. One possibility is that current health and mobility might be associated with both--after all, you  may need to be at a certain level of health and mobility to own a dog, and those folks are more likely to be alive at the end of the study. 

 

 

 

Racial stereotypes behind how we explain school shootings

Participants and newspapers were more likely to blame video games when a school shooter was white than when he was black. Photo: Richard Levine / Alamy Stock Photo

What causes mass shootings? Of course, we have no experimental evidence on this question. Correlational evidence (both within the U.S. and evidence that compares the U.S. to other countries) is consistent with the argument that gun availability is a factor. 

However, this post isn't about what really causes school shootings. It's about what people think causes them. And it turns out that the race of a school shooter influences what people believe about the shooter. 

After some school shootings (a subset of mass shootings), people have sometimes looked for answers not in the availability of guns, but in video games. Now, there is actually no evidence linking video games to mass shootings. However, this ABC news story talks about when people are more likely to think that video games are responsible: 

There's little evidence showing a relationship between violent video games and school shootings, but video games are often blamed for overall violence in our communities. A recent study published in the Psychology of Popular Media Culture set out to find if we look for an external explanation for violence by whites more often than we do for African Americans.

"We try harder to make excuses for white perpetrators," said investigator Dr. James Ivory, a professor in the Department of Communications at Virginia Tech, in an interview with ABC News.

The first study gave people a mock news story about a school shooting. When asked what caused the shooting, people were more likely to blame video games if the shooter was white than if the shooter was black.

It turns out that in the original study (linked here;  if it's paywalled you may be able to access via your university library), participants read the mock news story and were randomly assigned to one of two photos of the perpetrator--a mugshot showing either a Black or a White man. 

a)  There were two variables in this study. Name each one. For each, was it manipulated or measured?

b) What kind of study is this, correlational or experimental?

c) Sketch a graph of the results of this study. What will you put on the y-axis? 

d) In the study above, what role does random assignment play? Why did the researchers randomly assign people to see either a Black or White perpetrator?  

 

Here is how ABC news described the second study:

The second study looked at thousands of news articles -- video games were mentioned more often in stories of white shooters [compared to black shooters]. 

e) What are the variables in the study above? Name each one; for each, was it manipulated or measured?

f) What kind of study is this, correlational or experimental?

g) Sketch a graph of the results of this study.

h) In the study above, the researchers could not randomly assign news article authors to write about a black vs. a white perpetrator. How does that change the impact of this study? 

i) Which of the two studies do you think provides the strongest evidence that race influences people's willingness to think video games are a cause of the shooting?

 

Suggested answers

a) One variable is race of the perpetrator (levels are Black or White), this was manipulated. The other variable is "likelihood to blame video games". This was measured.

b) This was an experiment because one variable was manipulated. 

d) Random assignment contributes to the study's internal validity. They can be sure that the same kinds of participants were in each group (reading about Black vs. White perpetrators). Therefore, people who play a lot of video games themselves were equally likely to be in the two groups. You might be interested to know that in the original study, the researchers also pretested the Black and White photos they used to be sure that the faces differed on race but not on perceived attractiveness or dangerousness. This pretesting also contributed to the experiment's internal validity.

e) The first variable was race of the perpetrator in the article (Black or White), which was measured. The second variable was whether or not the article mentioned video games (yes or no).

f) This was a correlational study because all variables were measured. 

h) Because this study used naturally-occurring crimes, the researchers may have an internal validity problem. It is possible that the Black and White perpetrated shootings differed not just on race but also on number of victims, age of victims, geography, or some other variable. This introduces an alternative explanation for the result--perhaps people blamed video games for White perpetrated shootings because they had more victims, had older victims, and so on.

i) You might have selected the first study as the strongest because it was an experiment and allows us to make a causal inference. Or you might have selected the second study because it was conducted on "real world" stimuli such as news stories--therefore it appears to have good ecological validity or external validity. 

Thanks to Marianne Lloyd for this idea!

Which Ice Cream Flavor Are You?

Many flavors, many personalities: How could we test whether an ice cream personality test is reliable and valid? Photo credit: Okanmetin/Getty Images

At the height of summer, we all start considering the same, timeless question: Can your personality can be accurately summarized by an ice cream flavor? This Buzzfeed poll asks you to answer a series of summer-activity-related questions and then delivers a verdict. Go take it yourself! I'll wait. 

The first time I took the poll, I answered the questions seriously and was told that I am characterized as "Coffee," along with this description: 

You’re a go-getter, a hard worker who doesn’t pause to reflect on what you have accomplished in the past because you are so focused on what you want to accomplish in the future. You seek to inspire the people around you and help to propel them to greater achievements. At the same time, you are approachable and kind rather than snotty and entitled. It’s basically the best of all worlds.

I sure felt good reading that personality description! The second time I took the quiz, I clicked on questions randomly haphazardly, just to see what other feedback there might be. This time I got "Chocolate", with this description:  

Mmm, you’ve got a truly rich and decadent side that you aren’t afraid to share. You make people feel good — really good. You’ve got a knack for excelling in social situations and know just how to be a crowd-pleaser. Keep doing your thing and living that luscious life.

Of course, nobody is going to mistake a Buzzfeed poll for a legitimate personality measure. But it does provide a fun way to consider issues raised in Chapters 5 and 6. Take the quiz yourself and then prepare these questions:

 

1.     What flavor ice cream did you get? What did the results say about you?

 

2.     Do you think this quiz is reliable? What kind of reliability is relevant for this measure? How would you empirically establish the quiz's reliability? Here's a hint: You'd start with a large sample of people and have them take this ice cream quiz, and then..... (what?)

3.  Now think about the measurement validity of this quiz. I'll narrow down the question for you here: Are people who score in the Coffee category actually more likely to be "hardworking," "inspiring," and "approachable," as the description suggests? How would you establish the criterion validity of this quiz? Here's a hint: You'd start with a large sample of people and have them take the ice cream quiz. You'd separate them into people who got the Coffee category and those who did not get Coffee. How should the Coffees be different from the non-coffees?

 

4.     What might we change about the quiz if we wanted to study ice cream preferences in a scientific way? 

 

5.  Read this description on the Barnum effect. Reflect on how the Barnum effect might be at play for people who take this Buzzfeed quiz. 

 

Many thanks for this idea to Josh Maxwell of the University of New Mexico!

 

Suggested answers to selected questions.

2. Test-retest reliability is the most relevant. To establish it, you'd have a large sample of people take the Buzzfeed quiz at one time point and record everyone's category (e.g., Coffee, Chocolate, etc.) Then you'd track down the same group of people again some time later (maybe 4 weeks later) and have them take the same quiz again. People who were scored Coffee at Time 1 should also score Coffee at Time 2, and people who were scored as Chocolate at Time 1 should also score as Chocolate at Time 2. (You'd compute the correlation between these two category assignments using Cohen's Kappa.)

3. To establish this criterion validity, you'd recruit a large sample of people and have them take the quiz, recording those who got Coffee and those who did not. Then you'd need to measure variables such as "hardworking" "inspiring" and "approachable." I'm not sure how to measure "inspiring" or "hardworking," but you could probably measure "approachable" using the Big Five trait of Agreeableness (here's one measure of that). If the Buzzfeed poll is valid, then a simple bar graph (and an independent groups t-test) should show that people who are assigned Coffee should score higher on agreeableness than people who were assigned some other flavor. 

 

 

Claim: Mothers' friendships are good for babies' brains

The study showed that moms with larger social support networks had kids with higher cognitive test scores. Photo: Noriko Cooper / Alamy Stock Photo

The  Washington Post headline reads, "Mothers' friendships may be good for babies' brains."  The story is about an empirical study on mothers' social networks. Although it says "babies' brains", the study measured the babies' cognitive skill (there were no baby brain scans). The empirical article came out in JAMA Network Open. Does the original study support the claim? Let's find out.Before reading on, answer these questions:

 

a) What are the two variables in the Washington Post headline claim? What makes the claim a causal one? 

b) Of course, we need to conduct an experiment to definitively support a causal claim. Would an experiment be possible in this particular case?  

Here are some details about the study, as summarized by the journalist (not the researcher):

For the current study, researchers examined data on 1,082 mother-child pairs. They questioned women about their family structure, friendships and relationships in their communities and also looked at test results from cognitive assessments done when children were 2 years old.

Overall, mothers had an average of 3.5 friends in their social support networks. The kids of mothers with more than that tended to have higher cognitive test scores than the kids of those who had fewer, suggesting “network conditions were significantly associated with early cognitive development in children,” the study authors wrote.

c) Sketch a  simple scatterplot of the result described above. What will you put on the two axes? How many dots should your scatterplot have? What does one dot represent? 

Read the following details if you've studied Chapter 9: 

The researchers accounted for some factors that might influence child cognitive development, including mother’s age and IQ, father’s education level, and the child’s birth weight. 

In addition, "researchers also accounted for poverty levels"

d) The paragraph above is describing a multiple regression analysis. What would be the dependent (criterion variable) in this analysis? What are the predictor variables? (Hint: There are six predictor variables).

e) Imagine that someone proposes income as a possible internal validity problem. Given what you know about the study, can we say that "Perhaps friendship network size and child cognitive skills are correlated because of the third variable of income: That is, higher-income mothers may have more friends and more cognitively skilled children"?  Would this criticism be reasonable? Why or why not?

f) The journalist writes: "...they did not have data to assess other personal differences, such as history of depression, between mothers with large networks of friends and those with fewer."
This means history of depression could be a possible third variable explaining the relationship. How could the researchers rule out this third variable? 

The journalist writes: 

All of the families in the study lived in the area around Memphis. It is possible that results might be different elsewhere.

g) Which big validity is the journalist addressing in the statement above? 

h) Do you think the correlation between friendship and cognitive test scores would actually come out differently in a different city or country? Why or why not? 

 

Suggested answers:

a) The two variables are Amount of mothers' friendships and Level of health of babies' brains. The language of "may be good for" makes this a causal headline (remember we ignore adverbs such as "may" when classifying claims). 

b) Probably not; you'd have to randomly assign some mothers to have more friends and others to have fewer friends.

c)  One axis should have "number of friends" and the other should have "cognitive test score". There should be about 1082 dots on the scatterplot (better count your dots now...), and each dot represents a mother-child dyad.

d) The criterion variable is cognitive test score. The predictor variables are number of friends, poverty levels, mother’s age, mother's IQ, father’s education level and the child’s birth weight. 

e) Since the researchers did control for poverty levels, we can actually rule out wealth as a possible third variable. 

f) the researchers could obtain history of depression data on each mother and then add this variable as a predictor variable to the regression analysis. If the beta associated with "number of friends" is still significantly predicting cognitive test score, then we can rule out this possible confound (/third variable/ internal validity problem).

g) external validity

h) In my opinion, it seems unlikely that such a relationship is limited only to Memphis residents. 

Note: You can read the original empirical article in JAMA, here. Note to savvy instructors: The p-curve on the results in this paper suggests that there may be little evidentiary value. 

The science behind sleep

The science on sleep suggests that most of us need more of it!
Photo: Carlos Caetano/Shutterstock

A Washington Post article recently proposed that lack of sleep might be a "public health crisis." The journalist's story covered several types of empirical studies and made a series of frequency, association, and causal claims. 

Here are a series of quotes from the article. As you read each quote, ask yourself, a) How many variables seem to be involved in this claim? b) What is/are the variable(s)?  and then c) Is this claim frequency, association, or cause? How do you know? (Note: the links in each claim take you to the empirical articles on which they are based.)

Quote 1:

"Preschoolers who skip naps are worse at a memory game than those who snooze."

Quote 2:

"...a single night of sleep deprivation boosts brain levels of the proteins that form toxic clumps in Alzheimer’s patients." 

Quote 3:

"In 2016, the Centers for Disease Control and Prevention reported that a third of adults fail to get the recommended seven hours."

Quote 4:

"...even modest sleep reductions are linked to increased feelings of social isolation and loneliness."

Quote 5:

"A small study presented at the Society for Neuroscience conference last year found early evidence that playing “pink” noise — similar to white noise but with different frequencies — in a specific pattern could increase the proportion of time people spent in restorative deep sleep."

Quote 6:

"Adults over 50 with lots of insomnia symptoms were more likely to fall than those without, according to one study.:

Quote 7

"Studies have linked pain to poor sleep.."

 

Suggested answers to odd-numbered quotes.

Quote 1.

1. There are two variables:
2. Skipping a nap (or not) and level of skill at a memory game
3. This is an association claim, because the verb “are” is simply linking one type of preschooler (those who skip naps) with having lower skill at the memory game.

 

Quote 3.

1. There is one variable:
2. Getting 7 hours of sleep a night (or not)
3. This is a frequency claim. There is one variable and the sentence is mainly describing the number of U.S. adults who get that amount of sleep.

 

Quote 5.

1. There are two variables:
2. Playing “pink” noise (or not) and amount of time spent in restorative sleep
3. This is a causal claim because the verb used was “increase”.

 

Quote 7.

1. There are two variables:
2. Level of pain and quality of sleep.
3. The verb, “linked” is a verb signaling an association claim.

Reading vs. listening

Photo: Ladida/Getty Images

Audiobooks are more popular than ever. Have you ever wondered if there are cognitive differences between reading print books and listening to audiobooks? In this editorial, psychologist Dan Willingham explains some of the research comparing the two formats. Dr. Willingham describes several research findings, most of which come from experimental designs. 

Here's one point Willingham makes: 

...there are differences between print and audio, notably prosody. That’s the pitch, tempo and stress of spoken words. “What a great party” can be a sincere compliment or sarcastic put-down, but they look identical on the page. Although writing lacks symbols for prosody, experienced readers infer it as they go. In one experiment, subjects listened to a recording of someone’s voice who either spoke quickly or slowly. Next, everyone silently read the same text, purportedly written by the person whose voice they had just heard. Those hearing the quick talker read the text faster than those hearing the slow talker.

a. In the study above, what is the independent variable?  Was it manipulated independent groups or within groups (and what keywords in the description help you decide?)

b. What is the dependent variable? 

c. Sketch a small graph of the results described, labeling your axes mindfully.

 

Here is another study from Willingham's summary:

It sounds as if comprehension should be easier when listening than reading, but that’s not always true. For example, one study compared how well students learned about a scientific subject from a 22-minute podcast versus a printed article. Although students spent equivalent time with each format, on a written quiz two days later the readers scored 81 percent and the listeners 59 percent.

d. In this study, what is the independent variable?  Was it manipulated independent groups or within groups (and what aspects if the description helped you decide?)

e. What is the dependent variable? 

f. Sketch a small graph of the results described, labeling your axes mindfully.

g. Do you think you could have conducted this study as a repeated-measures design? Why or why not? 

h. What questions do you have about the internal validity of this study? 

 

Suggested answers

a. The IV is reading speed of the narrator during the first task; its levels are Slow and Fast. The phrase, "either spoke quickly or slowly" tells you this was manipulated as independent groups.

b. The dependent variable was how fast people read the text passage. 

d. Here the IV is reading a text or listening to a podcast. The results phrase refers to "readers" and "listeners," implying that the IV was manipulated as between groups (independent groups).

e. The DV is score on a comprehension quiz.

g. It might be difficult to conduct this exact study as a repeated-measures design, because people would have to process the same content aurally and via reading. Once they've processed the content once, it would undoubtedly be easier to process it a second time. 

h. You'd want to be sure that i) people were randomly assigned to the two conditions. This would control for selection threats; and ii) people in the two conditions read or listened to the exact same text (for example, a podcast and its transcript). This would help avoid a design confound.