Everyday Research Methods

Conspiracy reasoning vs. Scientific reasoning (the case of QAnon)

Conspiracy theories, unlike scientific theories, are unfalsifiable. Credit: buecax/Depositphotos

Early in the research methods course you learn about the theory-data cycle (Chapter 1). You also learn about the benefits of scientific reasoning over informal, everyday reasoning such as intuition, experience, or authority (Chapter 2). The QAnon conspiracy theory is a good example of several principles in these two chapters.

The quotes from this post come from this excellent story in The Atlantic, by journalist Adrienne LaFrance. 

In case you have never heard of the QAnon conspiracy theory, it started from a series of posts by an anonymous author named Q, who posted occasionally on anonymous "image boards" such as 4chan. Here is a quick recap by LaFrance:

In its broadest contours, the QAnon belief system looks something like this: Q is an intelligence or military insider with proof that corrupt world leaders are secretly torturing children all over the world; the malefactors are embedded in the deep state; Donald Trump is working tirelessly to thwart them. (“These people need to ALL be ELIMINATED,” Q wrote in one post.) The eventual destruction of the global cabal is imminent, Q prophesies, but can be accomplished only with the support of patriots who search for meaning in Q’s clues. To believe Q requires rejecting mainstream institutions, ignoring government officials, battling apostates, and despising the press. 

Part I. Falsifiability
Let's practice describing one way that conspiracy thinking is different from scientific thinking. One of the defining features of scientific thinking is that scientific claims are falsifiable. That is, scientists make claims/hypotheses that can be falsified  (disproven) by collecting and analyzing evidence. 

Most conspiracy theories a cannot be falsified, in part because they are stated in squishy, ambiguous terms, and in part because they are so multifaceted. As LaFrance writes,

In the face of inconvenient facts, [QAnon theory] has the ambiguity and adaptability to sustain a movement of this kind over time. For QAnon, every contradiction can be explained away; no form of argument can prevail against it.

Here is an example of some QAnon reasoning: 

At one point, Harger [a believer she interviewed] told me I should look into what happened to John F. Kennedy Jr.—who died in 1999, when his airplane crashed into the Atlantic Ocean off Martha’s Vineyard—suggesting that Hillary Clinton had had him assassinated. (Alternatively, a contingent of QAnon believers say that JFK Jr. faked his death and that he’s a behind-the-scenes Trump supporter, and possibly even Q himself. Some anticipate his dramatic public return so that he can serve as Trump’s running mate in 2020.) When I asked Harger whether there’s any evidence to support the assassination claim, he flipped my question around: “Is there any evidence not to?”

a) Why is Harger's theory (above) about JFK Jr.'s assassination unfalsifiable?

b) And, How does the theory's unfalsifiable nature make it more likely that Harger can believe in QAnon? 

c) Think like a scientist now. We can write a falsifiable statement (i.e., one that could be demonstrated by evidence to be wrong) related to JFK Jr. For example, to falsify the theory that "J.F.K. Jr. will become Trump's running mate in 2020" we could predict this statement:

"J.F.K. Jr.'s name will appear on the ballot with Trump in the 2020 Presidential election." (This is falsifiable because if someone else's name appears on the ballot, then this theory is wrong.)

Now see if you can come up with a falsifiable statement derived from the theory (above) that J.F.K. Jr.'s plane crash death was an assassination. Your statement can take the form, If J.F.K. Jr. was assassinated, then we should see __________. 

Part II. Accepting evidence
One feature of QAnon theory that makes it impossible to falsify is that its believers do not accept any evidence from legitimate sources. That is, whereas scientists use independently verifiable data to support or falsify theories, QAnon believers do not accept the legitimacy of such data. Here's LaFrance again:

Q frequently rails against legitimate sources of information as fake. Shock and Harger [the two believers] rely on information they encounter on Facebook rather than news outlets run by journalists. They don’t read the local paper or watch any of the major television networks. “You can’t watch the news,” Shock said. “Your news channel ain’t gonna tell us [expletive].” Harger says he likes One America News Network. Not so long ago, he used to watch CNN, and couldn’t get enough of Wolf Blitzer. “We were glued to that; we always have been,” he said. “Until this man, Trump, really opened our eyes to what’s happening. And Q. Q is telling us beforehand the stuff that’s going to happen.” I asked Harger and Shock for examples of predictions that had come true. They could not provide specifics and instead encouraged me to do the research myself. When I asked them how they explained the events Q had predicted that never happened, such as Clinton’s arrest, they said that deception is part of Q’s plan. Shock added, “I think there were more things that were predicted that did happen.” Her tone was gentle rather than indignant.

d) Why does QAnon's rejection of legitimate sources of information help its followers maintain their beliefs? 

As an aside, in the passage above, pay attention to the believer's statement that "deception is part of Q's plan." This feature virtually ensures that the theory remains unfalsifiable: Either believers will point to evidence that supports the theory, or, when faced with evidence that does not support the theory, they will claim that "deception is part of Q's plan." The theory is supported no matter what. 

Part III.  Confirmation bias.
In Chapter 2, you learn about quirks of human reasoning that make our intuition inferior to the scientific method. One of these quirks is the confirmation bias, the tendency of people to notice and remember evidence that confirms their beliefs, and to ignore or discount evidence that disconfirms them. 

e) How might confirmation bias be at play in the passage above, about events that Q predicted but that never happened? 

Confirmation bias is probably more prevalent when situations and theories are ambiguous. Ambiguity is clearly at play in QAnon, because Q usually makes general (rather than specific) threats. LaFrance writes,

—this is a very welcoming belief system, warm in its tolerance for contradiction—and are also what makes it possible for a practical man like Hayes [who has a youtube channel "explaining" Q's theories] to play the role that he does. QAnon is complex and confusing. People from all over the internet seek guidance from someone who seems levelheaded. 

f) Explain how confirmation bias, rather than scientific thinking (which strives to objectively evaluate evidence both for and against a theory), can explain the following reasoning about who Q himself is:

Many QAnon adherents see significance in Trump tweets containing words that begin with the letter Q. Recent world events have rewarded them amply. “I am a great friend and admirer of the Queen & the United Kingdom,” Trump began one tweet on March 29. The day before, he had tweeted this: “I am giving consideration to a QUARANTINE.” The Q crowd seized on both tweets, arguing that if you ignore most of the letters in the messages, you’ll find a confession from Trump: “I am … Q.”

Coda.
QAnon might seem like a harmless academic example of conspiracy theories. And, as LaFrance suggests, some people follow it just for fun (it might have even started out as a parody). However, as LaFrance argues, it has actually incited violence in the past. For example, in 2017, one believer walked into a Washington, D.C. pizza shop with a rifle because he had believed QAnon message boards that children were being held hostage in the shop's basement (the building doesn't have one). (He was wrong: "'The intel on this wasn’t 100 percent,' Welch told The New York Times after his arrest.")  LaFrance also reports, "In 2018, Reddit banned QAnon groups from its platform for inciting violence."  In the United States, several congressional candidates have promoted QAnon's theories. Unfalsifiable conspiracy thinking might give our leaders a (false) sense of confidence, but it certainly doesn't belong in government discourse.  

If QAnon supporters call for violence, revolt, or collective action and people do not think critically about what they say, this fake conspiracy theory will become a real threat to our democracy. 

 

 

 

 

 

 

 

Lion attacks on cattle: A field experiment

Researchers painted eyes like these on some of the cattle in each herd. Credit: Cameron Radford

Here's an example of a field experiment that took place in actual fields. The empirical article appeared in Nature Communications, and the authors of the research wrote their own summary for a popular audience in The Conversation. The quotes below come from their popular audience summary. 

In this study, the authors worked with cattle ranchers in Botswana who were trying to reduce the number of lions and leopards who attacked their herds.

The introduction outlines the theory the researchers were testing:

Many big cats – including lions, leopards, and tigers – are ambush predators. This means that they rely on stalking their prey and retaining the element of surprise. In some cases, being seen by their prey can lead them to abandon the hunt. We tested whether we could hack into this response to reduce livestock losses to lions and leopards in Botswana’s Okavango delta region.

Below are some details of the method of their study. It sounds like an experiment: Can you see why?

Working with Botswana Predator Conservation and local herders, we painted cattle from 14 herds that had recently suffered lion attacks. Over four years, a total of 2,061 cattle were involved in the study.

Before release from their overnight enclosure, we painted about one-third of each herd with an artificial eye-spot design on the rump, one-third with simple cross-marks, and left the remaining third of the herd unmarked. We carried out 49 painting sessions and each of these lasted for 24 days.

The cattle were also collared and all foraged in the same area and moved similarly, suggesting they were exposed to similar risk. However the individuals painted with artificial eye-spots were significantly more likely to survive than unpainted or cross-painted control cattle within the same herd.

In fact, none of the 683 painted “eye-cows” were killed by ambush predators during the four-year study, while 15 (of 835) unpainted, and 4 (of 543) cross-painted cattle were killed.

Questions

a) What were the independent and dependent variables in this study? What were the levels of each variable?

b) What kind of experiment was this: Posttest-only? Prettest-posttest? Repeated measures? Or concurrent measures? (Challenge question: How might your answer change if you take the predator's perspective, as compared to the cattle's perspective?)

c) Sketch a bar graph of the results of the study. (Later you can compare your sketch to the figures in the Nature article).

d) The authors don't seem to specify here whether or not they used random assignment to the three painting conditions. Why would random assignment be important?

e) Ask a question about the external validity of this study. (Which term will your answer involve: random assignment or random sampling?) 

At the end of their popular press summary, the authors raise two important limitations:

First, it is important to realise that, in our experimental design, there were always unmarked cattle in the herd. Consequently, it is unclear whether painting would still be effective if these proverbial “sacrificial lambs” were not still on the menu. Further research could uncover this, but in the meantime applying artificial marks to the highest-value individuals within the herd may be most pragmatic.

Second, it is important to consider habituation, meaning that predators may get used to and eventually ignore the deterrent. This is a fundamental issue for nearly all non-lethal approaches. Whether the technique remains effective in the longer term is not yet known in this case.

f) How might you design a study to test the first limitation? That is, how could you manipulate the eye-paintings variable in order to test if this intervention still works if there are no "sacrificial lambs" in the herd? What would your different conditions look like?

g) How might the authors test the second limitation? What can they change about, or add to, their study to test whether "the technique remains effective in the longer term"?

 

 

 

Testing a stereotype about only children (null effect)

Are only children more narcissistic? Credit: stockimagesbank/Shutterstock

People hold all kinds of stereotypes about family structures. For example, lots of people believe that the eldest child in a household is the most responsible. People think that younger children are often spoiled or that middle children are ignored or neglected. Most research on birth order doesn't support these beliefs, though (here's one example, which found that oldest children are a bit smarter, but not any different in personality).

What about only children? Both laypeople and psychologists have sometimes claimed that only children are more likely to become narcissistic. Because they tend to be the focus of parental attention and don't have to negotiate with siblings, onlys are considered to have fewer opportunities to develop humility and social skills. But is the stereotype true?

Here's a journalist's summary of a study that tested this assumption. The study found "... no evidence that only children are more narcissistic than those with brothers or sisters": 

Past research on this topic has turned up conflicting findings. Those earlier studies, however, often had methodological limitations — such as small sample sizes — that made their results less than reliable.

The authors of the new study say their research addresses and corrects those problems. “We can now say with rather high confidence that only children are not substantially more narcissistic than people with siblings,” says Michael Dufner, the study’s lead author and a psychologist at the University of Leipzig

The published article included two studies. The first was meant to establish that people actually do hold the stereotype that only children are more narcissistic: 

... Dufner and his colleagues conducted an online survey of a representative sample of 556 German adults. Their average age was 46, and 105 of them (19 percent) were only children.

The survey was designed to determine the prevalence of the stereotype that people who are only children are more likely to be narcissistic than those with siblings.

The survey found that people do, indeed, tend to judge only children as being more narcissistic, both in terms of their need to be the center of attention (a characteristic known as narcissistic admiration) and in terms of their desire to demean or devalue others to build up their own inflated sense of importance (narcissistic rivalry).

The second study tested the stereotype's accuracy:

For the second part of the study, the researchers analyzed data collected from a representative sample of 1,810 German adults, including 233 without siblings, who answered questions designed to measure their level of narcissism.

The analysis debunked the stereotype. “We found no evidence for the claim that only children are more narcissistic than non-only children,” the researchers write.

Questions

a) What kind of claim is it to say, in the first study, that "people do, indeed, tend to judge only children as being more narcissistic" ? Frequency, association, or causal?

b) Was it important to use a representative sample in Study 1? Why or why not? 

c) The second study was a correlational design. Which two variables did they measure?

d) Why would it be important to use a representative sample for Study 2?

e) Sketch a bar graph of the result that would support the Study 2 statement, "We found no evidence for the claim that only children are more narcissistic than non-only children" 

f) Ask a question about the construct validity of Study 2. Ask a question about the external validity of Study 2.

g) To use terminology from the textbook, Study 2 found a null effect, or no difference. Chapter 11 reviews the potential reasons a study might find a null effect. One reason is a potential ceiling effect on the dependent variable, or DV. Do you think that could be a problem here, with the DV of narcissism? 

h) One way to ensure that a study has enough power and precision to accurately detect a true null effect is to be sure the sample size is large. What was the sample size in this study? 

i) Chapter 11's coverage of null effects focuses on how experiments can sometimes result in null effects, but this study is an example of a correlational study finding a null effect. That is, the authors measured both only-child status and narcissism, and found no relationship. Which of the questions about null effects don't really apply to a correlational study? See the table in Chapter 11 (3rd edition: Table 11.2, 4th edition: Table 11.3). 

To see the original empirical article, go here. 

Does "virtue signaling" change minds?

If a lot of us send virtue signals, it can communicate to others that norms are changing. Credit: SEEphotos7171/Shutterstock"

Among people fighting for racial justice, "virtue signaling" may sometimes be derided as empty action. "Virtue signals" are public messages of support for a social justice cause. They could take the form of a yard sign, a t-shirt, or a social media post. Critics sometimes argue that such messages are easy and cheap. It's easy for a company to put a message on their home page that supports Black Lives Matter. It's not hard to wear a t-shirt that supports a social justice cause. But are these expressions of support all talk and no action?

An article in Time magazine argues that virtue signaling can be more powerful than we think. The authors, psychological scientists Jamil Zaki and Mina Cikara, integrate several studies to make their argument. Here's part of their introduction: 

It’s critical to question people whose most radical act is temporarily changing their profile picture. It’s critical they follow their words with action; the time for talk has passed. But focusing solely on virtue signalers obscures their most important role. Like radio waves, the signals they send are received by someone, and those receivers matter, too.

Zaki and Cikara argue that virtue signaling is an important way of influencing others. If a lot of people change their profile pictures to support BLM, for example, more people will start to believe that this movement is popular and they may adopt it, too:

Opinions can work this way: when a particular viewpoint gets a lot of attention, people assume it’s popular, and shift towards it. We receive signals, and are changed by them.

What studies support their argument? Here's one that the authors summarize: 

As social norms shift, individuals shift with them: adopting popular opinions and behaviors, and dropping ones that fall out of style. Norms are especially powerful as they gain steam. In one set of studies, Gregg Sparkman and Greg Walton presented diners in a cafeteria [either] with evidence that 30% of the U.S. population was vegetarian, or that 30% of the population was newly vegetarian. People who learned about this second, “dynamic social norm” were twice as likely to order a meatless lunch themselves. They saw not just where the crowd was, but where it was going, and didn’t want to be left behind.

Here are some questions about the study summarized above:

a) Was this an experiment or a correlational study? How do you know? (If it's an experiment, what kind of study is it?)

b) What were the key variables in the study? Were they manipulated or measured? 

c) Sketch a small graph of the results being described. Would a bar graph or a scatterplot make more sense?

Here's another study Zaki and Cikara introduce:

Signaling and conformity are not inherently positive. In 2016, the U.S. elected a president whose campaign was jammed with racist talking points. Prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants). 

d) This summary doesn't really help you figure out how the study was designed. But here's a chance to think carefully: What kind of study would we need to conduct in order to test the claim that "prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants)."?  There are three variables you would need to include in your study. What are they? 

 

Suggested answers to selected questions

a) This was probably a posttest-only experiment, because it sounds like they manipulated the message about vegetarianism and measured what foods people took in the cafeteria. Here is a link to the original empirical article. 

d) The key dependent variable in this study is prejudice levels, so that's the first variable. Then, in order to establish that "prejudice increased", you'd need to measure prejudice both before and after the election, so that's another variable. Finally, in order to show that it increased specifically toward groups he targeted, you'd need to measure different types of groups (Muslims, immigrants, as well as other potential groups that Trump did not deride in his campaign). You can review the Crandall article here. Do their results really support what Zaki and Cikara are saying about it? (Pay attention to the difference between people's actual prejudice and the normativity of prejudice.) 

 

 

 

 

Racial bias in psychological science journals

A recent study found that in psychology, the authors, participants, topics, and editors of empirical journal articles are mostly White.
Photo credit: Africa Studio/Shutterstock

Is science fair?

You learned in Chapter 2 that empirical journals are the primary place where psychological scientists make their research public. Each empirical journal has an editorial board-- a group of scientists who have power to select the peer reviewers for each article and make the final decision about publication. 

Each step of the way, the stated goals of the publication process are to be fair, to focus on the quality of the science, and to publish the best science possible. But is the process fair?  In light of the renewed energy around anti-racism, a timely new study sheds light on racial diversity in the psychology publication process. The researchers asked, Is race an important variable in the scientific publication process?

The team coded all empirical journal articles that appeared in the most prestigious journals in psychology (over 26,000 articles). There were two journals each from social, cognitive, and developmental psychology. The authors measured:

• how often race was studied as a subject in these articles
• the race of the participants in the studies
• the proportion of the authors of scientific articles were White vs. Non-white
• the proportion of the editors of the journals themselves were White vs. Non-white

Here are some of the results of this published study of published studies, as summarized by the Association for Psychological Science's trade publication:

The results suggest that there is a hierarchy in psychology research that determines who and what gets published. “We are not saying anyone has bad intentions,” said Roberts [one of the study's authors]. “It’s a systemic problem across many fields, and psychological science is no exception.”

First, across the past five decades, articles in psychological journals that highlight race have been rare, and although developmental and social psychology journals have published a growing number of these articles, they have remained virtually nonexistent in cognitive psychology.

Second, most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism.

Third, many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color.

Let's parse these claims. 

Questions

a) What kind of a claim is it (frequency, association, or cause) to state that "across the past five decades, articles in psychological journals that highlight race have been rare"

[And what's "rare"? The published report says that "only 5% of publications highlighted race" (p. 4). This 5% figure is an average across cognitive journals (less than 1%) and social (5%) and developmental journals (8%)]

b) Let's take the second claim, "most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism"

The quoted text above provides two claims in one sentence. What is the frequency claim in the quoted text? What is the association claim in the quoted text? What are the two variables in the association claim?

c) Now for the third claim, "many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color."

As with b), the quoted test provides two claims in one sentence, a frequency claim and an association claim. What are these? 

d) In Chapter 1 of the 4th Edition (available Fall 2020), you will read about Merton's norms of science--a set of values that the community of scientists usually endorses and strives  to follow. Here they are, with short descriptions (for a full review, see Chapter 1 or this summary). 

Universalism: Anyone can do science as long as they follow the accepted criteria for science, which are established in advance, are objective, and are accessible to all.
Communality: Share! The process of science, including data, methods, and products, should be free and open for all to use. 
Disinterestedness: The quality of the science should be the only thing that matters, not the prestige of the person or place it was conducted. 
Organized Skepticism: Verify everything. Critical peer review matters, every study should be replicated, and criticism of one's evidence is expected and welcome. 

Roberts and his coauthors argued that psychology is too White. Which of Merton's norms does this result seem to violate, and why? 

Suggested answers:

a) This is a frequency claim--a statement about the frequency of the variable, "highlighting race" across 26,000 journal articles.

b) The frequency claim is about the proportion of journal editors who are White vs. a Person of Color (POC). The association claim is about a link between two variables: whether a journal has a White editor or not, and whether that journal published articles highlighting race or not.

c)  The frequency claim is about the proportion of article authors who are White vs. POC. The association claim is about a link between two variables: whether the article was written by a White author or not, and whether that article included POC as participants or not.

d) Open to discussion!

Here is a summary of this work.

Empathy and social justice

The more privilege people have, the less empathy they have for people lower in status. But that can change with increased perspective-taking. 
Photo credit: Clay Banks/Unsplash

The spring of 2020 has seen sustained, worldwide, democratic protests, led by Black Lives Matter and demanding social justice and police reform in the U.S.  After this spring, American support for Black Lives Matter and police reform are suddenly higher than it has ever been.  Psychologist Jamil Zaki asks in this opinion piece, Why now? Why did Americans react more strongly in 2020, after police killed George Floyd, compared to other highly publicized instances when police killed other unarmed Black people, including Breonna Taylor, Tamir Rice, Philando Castille, Michael Brown, and Stephon Clark?  

Dr. Zaki points to empathy as the answer. He writes, 

In the past few weeks, many people have opened their eyes to suffering they had previously ignored. Much credit for this should go to the activists and organizers who have made it harder to look away. 

Dr. Zaki supports his argument with evidence from psychological research on empathy, power, and structural change. For example, teams of psychological researchers have investigated how being in powerful positions is associated with being less able to understand others' suffering:

In one series of studies, the psychologist Michael Kraus and his colleagues measured people’s socio-economic status, as well as their ability to decipher emotions in pictures and in-person interactions. People higher in status were less accurate about other people’s feelings. More recent work has replicated these results and also found that high-status individuals make more errors when trying to take other people’s perspective.

a) What are the variables in Dr. Kraus's research? Were these variables measured or manipulated? (Hint: There are two studies--take one at a time.)

b) Given your answer to question a), were these studies correlational or experimental?

Here are two more brief research descriptions (the links go to the original empirical studies): 

Higher-status individuals display less interest when talking with strangers, and report less concern for the suffering of others. These gaps play out in racial contexts as well. In another study, Kraus found that high-income white Americans overestimate racial economic equality more than black Americans or low-income white Americans.

c) What about the studies above--what are the variables? Are the studies correlational or experimental? (Hint: There are two studies--decide the variables in each study separately.)

Zaki argues that one reason higher status people have less empathy is because the powerful are less motivated to think about others. In contrast, less-powerful people need to pay attention to others in order to function well in society. He writes, "This is one way privilege works its way into our minds. Not only are privileged people exempt from material struggles, they can comfortably ignore everyone else’s."

Here are two more brief study descriptions that Zaki uses in support of his argument:

[Power] expands the change a person could make while narrowing the aperture of who they truly see. But this is not inevitable. When powerful people choose to empathize, they become more cooperative and more invested in justice. In one particularly relevant series of studies, Emile Bruneau and his colleagues asked members of low-power groups to “perspective give,” sharing their stories, and high-power individuals to perspective take, paraphrasing what they’d heard. These dialogues increased connection and positive regard between groups — not by ignoring existing power structures, but by reversing them.

d) In the study by Bruneau and colleagues, what were the variables? Can you tell if the variables were manipulated or measured?  

e) What do you think--was Bruneau's study an experiment or a correlational study? Can it support the causal claim that "these dialogues increased connection and positive regard between groups?"

f) Dr. Zaki's descriptions of the studies are short on detail, likely the result of working under a word-limit and writing for a popular audience. Try following the links to one of the studies linked in Dr. Zaki's article. Are you able to access the full text of the study easily? If not, can you access the journal using your university's library? When you read the full text of the original study, were you correct about its correlational or experimental nature? 

 

 

Saturated fat can impair concentration

Breakfast of (concentration) champions? Maybe not....Photo credit: Kyndall Ramirez/Unsplash

A headline at Psych Central reads, "High fat meal may impair concentration." Does the study behind the headline support that causal claim?

At first, you might assume that this came from a correlational study on diet and concentration. However, it turns out the research summarized in the article was experimental: It manipulated the saturated fat content of one specific meal. In the study, researchers created two versions of a breakfast--one made with saturated fat and the other made with sunflower oil, which is also high in dietary fat but is not the saturated type. 

The journalist describes the method of the study this way: 

Women in the study [n = 52] completed a baseline assessment of their attention during a morning visit to the lab. The tool, called a continuous performance test, is a measure of sustained attention, concentration and reaction time based on 10 minutes of computer-based activities.

The high-fat meal followed: eggs, biscuits, turkey sausage and gravy containing 60 grams of fat, either a palmitic acid-based oil high in saturated fat or the lower-saturated-fat sunflower oil. Both meals totaled 930 calories and were designed to mimic the contents of various fast-food meals such as a Burger King double whopper with cheese or a McDonald’s Big Mac and medium fries.

Five hours later, the women took the continuous performance test again.

Between one and four weeks later, they repeated these steps, eating the opposite meal of what they had eaten on the first visit.

After eating the meal high in saturated fat, all of the participating women were, on average, 11 percent less able to detect target stimuli in the attention assessment.

(The original empirical journal article is here; it might be paywalled)

Questions:

a) What was the independent variable (IV) in this study and what were its levels?

b) What was the dependent variable?

c) Was the IV manipulated as independent groups or within-groups?

d) This design is difficult to classify because it has a pretest and posttest, but the women also experienced both levels of the IV. Therefore it shares features with two of the four designs in Chapter 10. What are those two designs it shares features with? (Note to students: As you can see in this example, the four designs you learned about in Chapter 10 are not the only types of experiments there are, and this study provides an example of alternative designs.) 

e) Sketch the results of this study--using "reduction in concentration" (i.e., a difference score) as the dependent variable. 

Next let's work through the four big validities. 

f) Here's some text related to external validity of this study:

For this work, Madison [the lead author] conducted a secondary analysis of data from Kiecolt-Glaser’s study assessing whether high-fat meals increased fatigue and inflammation among cancer survivors.

and

[the experimental meals]  were designed to mimic the contents of various fast-food meals such as a Burger King double whopper with cheese or a McDonald’s Big Mac and medium fries.

Given the information above, to what kinds of people and what kinds of situations might this research be generalized? How do you know? 

g) The researchers found an 11 percent drop in concentration after the meal high in saturated-fat. What further information would you like to know in order to evaluate the study's statistical conclusions? 

h) Can this study support the causal claim that "A high-fat meal may impair concentration"? Apply the three criteria to the study: Covariance (what did the results show?), temporal precedence (did the method ensure that the meal came before the drop in concentration?), and internal validity (What potential confounds did they control for in the study?) 

i) Here's some information about the study's construct validity: 

Women in the study completed a baseline assessment of their attention during a morning visit to the lab. The tool, called a continuous performance test, is a measure of sustained attention, concentration and reaction time based on 10 minutes of computer-based activities.

What questions would you need to ask to determine if this test is a good measure of people's ability to concentrate? 

Instructors: If you teach Chapter 12, you can also use this example as a mixed factorial design, because the study also included a participant variable of endotoxemia, a condition called "leaky gut." From the journalist's summary: 

After eating the meal high in saturated fat, all of the participating women were, on average, 11 percent less able to detect target stimuli in the attention assessment. Concentration lapses were also apparent in the women with signs of leaky gut: Their response times were more erratic and they were less able to sustain their attention during the 10-minute test.

“If the women had high levels of endotoxemia, it also wiped out the between-meal differences. They were performing poorly no matter what type of fat they ate,” Madison said.

Students could identify the IV, PV, and DV and practice graphing the results described above. 

Thanks to my colleague Stephen Chew, who shared this example.

Teaching with 100 Humans

In 2020, Netflix released an 8-episode series called "100 Humans" in which 100 (American) humans from a variety of ages and social groups participate in a series of studies. It's a gold mine of examples for critical thinking in research methods. You'll find examples of posttest-only, pretest-posttest designs, HARKing, ethics, and more. Most of the studies have flaws, but that makes them more fun to critique. 

I've cataloged ways you can use the series in your teaching: www.tinyurl.com/teachingwith100humans.

Reach out if you use it! Let me know how it works for you. 

 

What's the deal with predatory journals?

The published study claimed to have observed one parrot, one penguin, and one woodpecker. Photo Credit: Zdenek Machacek/Unsplash

Today's the first time that the National Audubon Society has been featured on this blog! The story is ornithological (sort of) but it's mainly a parable about predatory journals. The journalist for the Audubon Society tells the story of ornithologist Daniel Baldassarre, who published an article in the legitimate-sounding Scientific Journal of Research and Reviews (SJJR). The Audubon journalist explains that there are several odd features to this empirical article. 

The published article, “What’s the Deal with Birds?”, includes the following abstract: 

Many people wonder: what’s the deal with birds? This is a common query. Birds are pretty weird. I mean, they have feathers. WTF? Most other animals don’t have feathers. To investigate this issue, I looked at some birds. I looked at a woodpecker, a parrot, and a penguin. They were all pretty weird! In conclusion, we may never know the deal with birds, but further study is warranted.

The journalist explains:

...the sample size—a woodpecker, a parrot, and a penguin—... seems suspiciously small. The main figure, a graph plotted along an x-axis ranging from “weird beak” to “looks like a fish,” with a red line labeled “the deal,” is a textbook example of dadaist absurdity. The prose veers wildly between academic terminology and gormless observation. “This is the first study I am aware of to attempt to quantify the deal with birds,” Baldassarre writes. “Unfortunately, the results were ambiguous, although Bayesian approaches may prove useful in the future...When presented with weird behavior, birds exhibited a multimodal response including physical aggression and duetting, both of which were repeatable across highly variable contexts.” 

How did such a ridiculous article get published? Turns out that Dr. Baldasarre was trying to prove a point--some scientific journals are not legitimate--they are predatory, and will accept anything.

...the SJRR is a predatory journal, designed to bilk unwary academics out of money, and Baldassarre’s paper is a joke—quite literally—at their expense. 

[...] Baldasarre first had the idea to submit a joke paper in early February, when the latest in a long line of emails from one of these scam journals landed in his inbox. He slapped together the first iteration of “What’s the Deal With Birds”—a couple of paragraphs in the cursory format of a manuscript—only to see it rejected, perhaps as an obvious parody. Undaunted, he inserted some selections from an earlier legitimate paper to pad it out and resubmitted it to SJRR. 

[...]While SJRR initially demanded a $1,700 publication fee, Baldassarre was eventually able to bargain them down to nothing.

You can access the original paper, including its full-color figure, in its predatory-journal-glory here. (Oh no! It looks like SJRR might have taken it down, given all the negative publicity. Here is an independent link.) 

Questions

a) What is a predatory journal and how is it different from a legitimate journal? (Hint: Use the terms peer review and high quality science in your answer.)

b) If you're doing your own library research for a literature review, how can you know if an article you'd like to use is in a predatory journal or not? One way is to ask your librarian. If you are a student, go to your university's library home page, and find where you can access online help from a university librarian. It might be an online chat, a help desk, or an email consultation service. 

c) One potential way to check if a journal is predatory is to use Beall's List. Go there and try typing in the name of the journal (Scientific Journal of Research and Reviews) in this bird story, or try the name of the journal's publisher, Iris Publishers. Does Beall's List consider this journal or its publisher to be predatory? 

d) Visit the homepage of SJRR's publisher, Iris Publishers. Now visit the homepage of two legitimate publishers, Sage Journals and PLOS, both of which publish legitimate science journals. Just by looking at the websites, can you tell which publishers are legitimate?  

e) Predatory journals make money by charging authors (as much as $2000) to publish their articles. But importantly, just because a journal charges the authors for publication does not mean it is predatory. For example, the journal PlosONE is a legitimate journal that charges authors to publish their work. PlosONE is open access, meaning that anybody can access PlosONE articles for free--you don't have to have a subscription via a university library (i.e., the expenses of publication are transferred from libraries to researchers).  Visit the PlosONE webpage and find an example of a (presumably legitimate) psychology article there. 

f) Draw a Venn diagram or concept map depicting the relationships among the following concepts: predatory publishers, legitimate publishers, predatory journals, legitimate journals, authors pay to publish, open-access, peer reviewed. 

Evidence-based tips for well-being during a pandemic

Dr. Santos provides evidence-based ways to enhance well-being during this pandemic. Credit: Yale University

How can people enhance their wellbeing during the Coronovirus pandemic? 

Dr. Laurie Santos (pictured) has been meeting with several news outlets, sharing evidence-based tips from her psychology of well-being class at Yale University. A version of the course is also offered on Coursera.

In the Twitter-embedded video here, produced by the World Economic Forum, Dr. Santos covers five tips for coping during the pandemic. In this space, I'm going to quote moments from the video, and you can practice classifying the claims and describing how the studies would have been conducted. The first three of these tips are also covered in this story from the World Economic Forum.

Tip 1: Keep social, even when socially distancing

Dr. Santos states in her segment that "Happy people spend more time with other people in general. They also prioritize time with their friends and family members"

a) What kind of claim is it to say that "happy people spend more time with other people in general"--frequency, association, or cause?  What are the variables in this claim? 

b) What kind of study was likely conducted to support this claim--correlational or experimental? 

c) Sketch a scatterplot with well-labeled axes that could depict this result.

Tip 2: Help others

To explain this tip, Dr. Santos mentions, "The research shows that doing random acts of kindness...can be incredibly powerful." For example, happy people are giving money to charity and volunteering in their communities. 

d) What kind of claim is it to say that "doing random acts of kindness...can make you happy"--frequency, association, or cause?  What are the variables in this claim? 

e) What kind of study would need to be conducted to support this claim--correlational or experimental? 

f) Can you imagine how you  might manipulate the variable, "doing random acts of kindness?"

Tip 3: Move--even if you can't leave your house

In this section, she says, "Exercise and moving your body is really correlated with improved well-being"

g) The verb, "correlated" gives this one away: It's an association claim. Dr. Santos specifies that they are "really correlated," which seems to suggest a strong effect size. What factors do we consider when we're trying to decide if a correlation (an effect size) is strong versus weak?

Tip 4: Savor everyday pleasures

"Research suggests that happy people tend to be more mindful. They tend to savor the present moment; they tend to be in the present moment"  Here Dr. Santos also strongly recommends taking deep breaths to calm the vagus nerve and connect to the present. 

h) How might you design an experiment to test the hypothesis that deep breathing can improve people's well-being during quarantine? Whom might you recruit for such a study? How would you operationalize each of the two variables? What results would you expect to find? 

Tip 5: Think about how you want to look back on this time

Finally, she shares one of my favorite tips for emotional regulation, psychological distancing: "Psychological distancing is the idea that if we think about ourselves and how we will have dealt with this crisis in the future...Think about how you want to tell your grandkids you lived through COVID-19, you don't want to be like, 'I panicked-bought and hoarded toilet paper..no, you want the story to be positive...'"

Experimental research has supported the causal claim that self-distancing--seeing yourself as though you are a fly on the wall, or even imagining looking back on yourself from the future--is associated with improved emotional regulation. It helps us take a cooler, less impulsive view of our emotions and process them instead of ruminating. For example, self-distancing helps people manage feelings of anger in constructive ways (here's a nice summary). 

i) Review the term translational research in your textbook (Ch 1) and explain how we might conduct translational research on psychological distancing during the pandemic. What is the basic research it is based on? What is the real-world context it's being translated to? And what kind of study could translate--that is, bridge the gap--between the two?