Everyday Research Methods

Really? Wearing high heels can improve your walking

Should we trust a study that tested only 8 people? Photo: sirtravelalot/Shutterstock

Yahoo news picked up a story from the Washington Post , which, in turn, had covered an empirical journal article about high heels and walking (did you follow that?). The Yahoo headline read, "Can wearing high heels actually improve your walking?"  In this post, we'll check out the weaknesses of this study--and think about why these two news sources thought the study would be worthy of coverage. 

Yahoo's story started out like this: 
Can wearing high heels actually be good for you? A new study published in the Journal of Applied Physiology says that wearing high heels can help you improve your walking overall. This could come as a surprise since there are many warnings against wearing high heels constantly.

Then the Yahoo journalist says:

Researchers wanted to find out what changes occur in the human leg when someone starts wearing heels for the first time. Researchers had participants wear custom heels that were, at most, 3 inches for 14 weeks, per The Washington Post.

The participants that consistently wore the heels over the 14 weeks had become efficient walkers, specifically using less energy to walk than before they started wearing the heels,

a) Based on what you've read so far, what is the independent variable in this study? What is the dependent variable?

Now, read a few more details about the study from the Washington Post article (linked here). 

For the new study, published this month in the Journal of Applied Physiology, Beck and his colleagues looked for healthy young men, as well as women, who had never or almost never worn high heels. By recruiting heel virgins, the researchers hoped to be able to track changes inside people’s legs associated with starting to wear the higher footwear.

They wound up with five male and three female volunteers.

The scientists then created high-heeled shoes acceptable to all recruits by attaching foam wedges to the soles of flat Chuck Taylor All-Star Low Top sneakers. Each pair was custom fitted so that a participant’s foot would be flexed downward at a 14-degree angle. In practice, this meant the shoes sported about a 2½- to 3-inch wedge heel.

This summary above should alert you to one of the potential problems with this study: its small sample size.

b) Why might it be a problem to have a sample of only eight people in any study? Your answer should talk not about generalization, but about how a small sample might result in results that are unusual and cannot be replicated. 

The study was a pretest-posttest design. The Post article explains that the researchers tested each of the participants on their walking efficiency (as a pretest), and then gave them their high heeled Chuck Taylors to wear for 14 weeks. After the 14 weeks, everybody's walking efficiency was tested again. 

As you know from Chapter 10, in a pretest-posttest design,  participants are tested on the dependent variable (here, walking efficiency) twice, once before and once after exposure to one level of the independent variable. So let's read more about the independent variable. Some people wore high heels and some did not, but....

[The researchers] asked them to go forth into the world and wear their new heels every day for 14 weeks. Not everyone did. Some of the volunteers “were kind of embarrassed” by the makeshift stacked sneakers, Beck said. Others complained of squished toes or other discomforts, quibbles familiar to any flat-shoe habitue who dons heels on occasion.

But more than half of the volunteers, male and female, wore the shoes most of the time. After 14 weeks, everyone returned to the lab and repeated the earlier tests. Those volunteers who had mostly given up early on wearing heels showed no changes in their legs or walking. But those who wore the heels fairly consistently tended to have shorter calf muscles and stiffer Achilles tendons than before.

c) Check that paragraph again, and ask yourself, Did the researchers randomly assign people to the "high heels" and "no high heels" conditions? Why or why not? 

d) Looking back at c), think about how this poses an internal validity problem for the study. Which of the internal validity threats (from Chapter 10 or 11) does it seem to be? 

e) Why do you think this particular study (which had a tiny sample, put only 5 people in one group and 3 in the other, not randomly assigned) was picked up by the Washington Post and by Yahoo news?  Do you think the study should have been covered by the press? 

 

You can find the original empirical journal article here. 

 

Really? "Cutting out meat might help prevent snoring"

Does the study support the causal claim the journalist attached to it? Photo: Dusan Petkovic/Shutterstock

Guest post by Carli Fine, University of Delaware

A recent news article for the general public was headlined, “Cutting Out Meat Might Help Prevent
Snoring: Study”. The journalistic source was Health Day: News for Healthier Living, which makes recommendations for the general public and has news for medical professionals. The original empirical journal article  (open access) was entitled “Plant-based and vegetarian diets are associated with reduced obstructive sleep apnoea risk”.

This study offers a great opportunity to discuss how association claims differ from causal claims and how empirical journals differ from popular journalism.

In the empirical study (Melaku et al., 2024), researchers investigated the variables of adherence level to a general plant-based diet (PDI) and risk level for obstructive sleep apnoea (OSA). PDI was measured with a 24-hour recall method, as participants reported the foods that they had
consumed in the past day.  OSA was measured using a tool called the STOP-BANG (which combines data about people's Snoring, Tiredness, Observed snorting, Pressure (blood pressure), Body mass index (BMI), Age, Neck size, and Gender). The researchers found a negative relationship between PDI adherence and OSA risk (STOP-BANG scores), such that higher PDI adherence went with lower OSA risk.

a) The journalist's headline reads, "Cutting Out Meat Might Help Prevent
 Snoring". What's the verb in this headline? To what category of claim does this verb belong: Frequency, association, or cause?  (Use Table 3.4 below for a hint.)

Table 3.4

b) What were the study's variables?  

Name of the variable (Conceptual level)	What are the variable's possible levels?	Is this manipulated or measured?

c) How do you know this was a correlational study (and not an experiment)?

d)  Sketch a well-labeled graph of the result they describe. It’s okay if you don’t know the exact
data for each group. Before you start: Will your graph be a bar graph or a scatterplot? 

 

e) Does the study support a causal claim/headline? Name, and then apply, each of the three causal criteria to this situation. When you get to the third one, you should specify a plausible third variable that could explain the relationship.

Name of causal criterion	Application to this example

f) At the end of the popular article you'll see a “What This Means For You” box, copied below. Comment on this takeaway: Does the study support the claim in the box? 

g) Scholarly sources are sometimes referred to as primary source literature, while journalistic sources may be referred to as secondary source literature. Fill out the table below identifying who the authors are for each type of article. Which type of source should you use for a literature review?  You'll find Table 2.3 helpful. 

Type of Source	“Degree” of Source	Authors of Source	Intended Audience of Source	Examples of this Type of Source
Scholarly	Primary
Journalistic	Secondary

h) Journalistic sources tend to analyze, interpret, or summarize the content of primary source literature. Besides the headline, here are a few more examples of quotes that differ between the two articles. What are some of the differences in meaning? Why do you think that the journalist made these choices?

Journalistic Source	Scholarly Source
“People with diets highest in plant-based foods were 19% less likely to have sleep apnea compared to those with the lowest amounts of plant-based foods in their diet.”	“Participants in the fifth (highest) quintile of PDI had 19% lower odds (OR=0.81; 95% CI: 0.66–1.00) of sleep apnea risk compared to those in the first quintile (p-value for trend=0.008).”
“A plant-based diet worked more powerfully to lower sleep apnea risk in men”	“Overall PDI is primarily associated with reduced odds of OSA risk in males, not females.”

 

 

 

 

Suggested Answers:

a) The popular article title is “Cutting Out Meat Might Help Prevent Snoring: Study”. The verb is “prevent”, which is a causal claim verb. The empirical article title is “Plant-based and vegetarian diets are associated with reduced obstructive sleep apnoea risk”. The verb is “associated with” which is an association claim verb.

b)

Name of the variable (Conceptual level)	What are the possible levels of this variable?	Is this variable measured or manipulated?
Level of Adherence to a General/Healthy Plant-Based Diet (PDI)	Low to High	Measured
Level of Risk for Sleep Apnoea (OSA)	Low to High	Measured

c) This was a correlational study because all the variables were measured. In an experiment, at least one variable is manipulated.

d)

e)

Name of causal criterion	Application to this example
Covariance	The results show that level of adherence to a general/healthy plant-based diet was associated with level of sleep apnoea. Yes, covariance is achieved!
Temporal Precedence	This one is tricky! The researchers asked about both variables in the same questionnaire, at the same time. However, PDI was assessed based on foods consumed in the past 24 hours, which therefore took place before the STOP-BANG tool was administered. Yes, temporal precedence is achieved!
Internal Validity	This is a correlational study, so we can’t rule out any third variables that might be responsible for the underlying relationship. For example, people who adhere more to a plant-based diet may also choose to engage more in exercise and healthy behaviors that reduce risk for sleep apnoea. Alternatively, people who have a higher socioeconomic status (SES) might have better access to and means to afford fresh fruits and vegetables and also have better access to healthcare services leading to better overall sleep hygiene. No, internal validity Is not achieved!

f) The key takeaway uses the verb “reduce” which is a causal claim. The causal claim made by the popular article cannot be supported because the study did not manipulate a variable to establish internal validity. In fact, the researchers even write, “The cross-sectional nature of the study prevents us from inferring causality between dietary indices and OSA risk.”

g) 

Type of Source	“Degree” of Source	Authors of Source	Intended Audience of Source	Examples of this Type of Source
Scholarly	Primary	Researchers who conducted a scientific study	Other scientists, researchers, readers of this blog and students who have taken research methods!	European Respiratory Society, Nature, Science, Cell, Journal of the American Medical Association
Journalistic	Secondary	Journalists or reporters, typically employed by a news organization	General population	Health Day: News for Healthier Living, The New York Times, The Washington Post, BBC News, CNN

g) You should only use empirical/scholarly/primary sources for a literature review.

h) Journalists write for a general audience that might not have specialized knowledge in a particular field. They might oversimplify the research by avoiding technical jargon and statistical terms to make it less precise but more accessible. Journalists typically want their stories to be impactful for people’s daily lives. They might make causal interpretations of correlational studies to be more actionable.

What's your love language?

As it's nearly Valentine's Day, it's a good time to feature this new story on so-called "love languages." According to a new review by relationships researchers, it looks like we should lump this idea, along with the Myers Briggs Type Inventory and learning styles, into the category of "things that make research psychologists cringe."

Many journalists have covered an empirical review article recently published in the journal, Current Directions in Psychological Science. by psychologists. This post features an story for the website medical express.

Even if you don't know your love language, you've probably heard of the concept. The theory's pervasiveness in pop culture has only increased in the 30-odd years since Baptist pastor Gary Chapman published his book "The Five Love Languages: The Secret to Love That Lasts." [...]

"We were very skeptical about the love languages idea, so we decided to review the existing studies on it," says Emily Impett, a professor in the UTM department of psychology who collaborated with UTM graduate student Gideon Park and York University Assistant Professor Amy Muise. "None of the 10 studies supported Chapman's claims."

Chapman's book argues that there are five love languages: physical touch, words of affirmation, acts of service, quality time and gifts, and that each person has a primary preference for one of them. Furthermore, he argues in his book that when two partners have the same love language (for example, both of them prefer physical touch) that their relationship will be better.

The journalist explains that Impett and her team tested each of Chapman's ideas. One was that people only have one primary love language. In fact, Chapman's unscientific "quiz" forces people to choose one. When people are provided a likert scale for each type, they report multiple preferences of showing and expressing love. Second, Chapman's original thinking was based on a restricted sample (mostly White, religious, middle-class Americans), and in other cultural contexts, there may be different ways of expressing love. 

But let's focus on this claim:

Most importantly, Impett and her team found no scientific evidence for Chapman's central contention that people who choose partners that speak their love language, or learn to speak it, will have more successful relationships. "There's no support for this matching effect," says Impett. "People are basically happier in relationships when they receive any of these expressions of love." 

The researchers reviewed the literature and found

no scientific evidence for Chapman's central contention that people who choose partners that speak their love language, or learn to speak it, will have more successful relationships. "There's no support for this matching effect," says Impett. "People are basically happier in relationships when they receive any of these expressions of love." 

This conclusion is a great example of the theory-data cycle. Let's walk through it. I've included Figure 1 for your reference.

a) The theory-data cycle starts with a theory. What is Chapman's theory about love languages? Remember that a theory is a series of general statements about how variables are related to each other. 

b) A researcher can then design a study to test the theory. In one study that Impett and her team reviewed, researchers recruited a sample of couples and measured each person's love language, and also measured their relationship satisfaction. Then they divided the couples into two types: those who had the same love language and those that did not. 

c) The next step in the theory-data cycle is to state a hypothesis for the study--what should the results of this specific study be if the theory is true? Sketch a graph of the predicted outcome, or state specifically what the results should be. 

d) The next step is to evaluate the results in light of the theory. What did the study actually find? Do these results support the theory, or not? 

 

 

Most American adults feel awkward

The journalist's uncritical acceptance of this frequency
claim's data is also awkward.
Photo credit:JRP Studio/Shutterstock

Here's an interesting set of frequency claims about what percentage of U.S. adults feel awkward. The data were presented at the content farm StudyFinds, under the headline, "That's awkward! 68% still often feel as insecure as they did as teens" 

Here's a snippet from the journalist's summary:

A survey of 2,000 U.S. respondents ages 25 to 45 found that as teens, people were most self-conscious about their body shape (65%), hairstyle (61%), and smile (61%). Many continue to feel this way, despite the average person feeling at their most awkward at age 17.

As adults, over half are still uncomfortable about their body shape (55%), hairstyle (53%), and smile (52%). More men than women are currently self-conscious about their glasses/contact lenses (51% vs. 39%), height (54% vs. 42%), hairstyle (57% vs. 48%), and smile (55% vs. 49%).

You might be wondering why a researcher might collect such specific data about what people feel awkward about. Turns out this study was  "Conducted by OnePoll on behalf of Smile Express, a doctor-monitored at-home aligner treatment for adults"--in other words, the poll was commissioned by a private company that sells ways people can improve their appearance. 

Questions

a) How might the commercial nature of this survey affect how the study was conducted? For example, what results might Smile Express be hoping to find? How might that affect the question writing strategy and sampling strategy? 

b) Which of the four big validities should we prioritize when reading about a frequency claim such as "over half are still uncomfortable about their... smile (52%)"? 

c) To help you consider external validity, here's some information about the study's sample. Based on this quote, do you know anything about the representativeness of the study's sample? 

This random double-opt-in survey of 2,000 U.S. adults ages 25–45 was commissioned by Smile Express between Aug. 7 and Aug. 10, 2023. It was conducted by market research company OnePoll, whose team members are members of the Market Research Society and have corporate membership to the American Association for Public Opinion Research (AAPOR) and the European Society for Opinion and Marketing Research (ESOMAR).

Discussion of question c:

To understand the paragraph above, you need to know what a "double-opt-in" survey is. As the name implies, an "opt-in" survey is one people volunteer for. For example, people might volunteer to participate in a study in exchange for cash, gift cards, or frequent flyer miles. A double-opt-in study is one in which people opt in twice; first they volunteer to be contacted by the commercial polling company, and then they opt in to a particular survey or poll. 

To test the effectiveness of opt-in polls, Pew Research once compared three commercial opt-in polls to three Interent panel samples which had used random sampling of addresses. They found that the opt-in polls resulted in larger errors compared to known benchmarks. For example, people in the opt-in polls were more likely to be receiving social security benefits than the general population and were less likely to have served in the military. In other words, opt-in surveys are not particularly representative, especially when compared to Internet panels, which use probability samples.

Given that opt-in polls are not particularly representative, why, then, might the StudyFinds journalist have described this sample as "random"? Maybe she misinterpreted the study's sampling strategy?

Advanced students will enjoy critically reading the website of the market research company OnePoll and identifying ways that it describes its research (its FAQ includes the question, "How do you ensure that data is statistically significant"?)

Music festivals "make people more generous, socially connected"

Credit: halfpoint/Deposit Photos

Summer is the time for music festivals like Burning Man, Burning Nest, Firefly, and Bonnaroo. If you are planning to attend one this year, this post might interest you.

The science journalism clearinghouse called Study Finds recently reported on a study conducted at Yale University. The journalists' report included the causal headline that Music festivals like Burning Man make people more generous, socially connected to others". 

a) What are the variables in this causal headline, and what makes it a causal claim?

b) In general, researchers need to conduct an experiment in order to support a causal claim. Do you see any barriers to conducting an experiment on the causal effects of music festivals?

It will be interesting to read about whether the researchers' study can feasibly support the causal claim that the journalist attached to it. Here's how Study Finds described the study's method: 

The Yale researchers studied the behavior and experience of over 1,200 people attending festivals such as Burning Man, Burning Nest, Lightning in a Bottle, Dirty Bird, and Latitude. All of the festivals took place in the United States or the United Kingdom and emphasized art, music, and self-expression. The team set up booths at all the events, which they called “Play Games for Science.”

People ... self-reported their experiences at the event and whether they were willing to share their resources with friends and strangers. For example, week-long festivals such as Burning Man do not sell food, and people who find themselves short must rely on the generosity of others.

[...] Overall, 63.2 percent of people said they had a transformative experience and felt themselves changing. ... One of the takeaways from the transformative experience was feeling more connected to other humans, which made them more likely to share food and resources with other attendees.

c) So far, do the method and results of this study (as reported by the journalist) support the conclusion that attending the festivals caused a transformative experience? Why or why not? (Hint: Think about the need for a comparison group.)

d) Based on what you've read, was the variable of "transformative experience" operationalized as a physiological, observational, or self-report measure?

In this case, the journalist's coverage of the study doesn't do the original study justice, as I discovered when I read the original empirical article, which appeared in the journal Nature Communication (you can access it here).There are several important omissions in the journalist's report. One is that the researchers themselves did not use causal language in their work. Instead, their article is titled "Prosocial correlates of transformative experiences at secular multi-day mass gatherings" (my emphasis).

The Study Finds journalist also omitted key methods and variables. For example, one important variable in the original study was that the researchers graphed people's self-reports of having a transformative experience on each day of a 7-day festival (see Figure 2,  here). Using this method, the researchers found that the later the day (going from Day 1 to Day 7), the more likely respondents were to report having a transformative experience. 

e) Why is the above method and result important, in terms of supporting the claim that the music festivals can increase people's tendency to have a transformative experience? (Hint: think about temporal precedence.)

Another important detail omitted by the Study Finds story, is that in addition to asking people to self-report their transformative experiences, researchers also asked participants to imagine playing a "dictator game". In this game, participants are given 10 valuable tickets. Participants can then donate some number of their tickets to randomly selected stranger. The dictator game is an observational measure of generosity to strangers. The researchers found that festival participants chose to donate between 50 and 75% of their tickets to the stranger. This was much higher than the level of donations typically found, as reported in a meta-analysis of all previous studies using the dictator game (which found an average of 26%). 

f)  This dictator game result is a type of quasi-experiment--we might call it a non-equivalent control group, posttest-only design. What is the quasi-independent variable--that is, what are the two levels of the non-equivalent control group (and why are they "non-equivalent")? What is the dependent variable? 

g) The result suggests that attending the music festival might cause people to be more generous in the dictator game, but there might be a selection effect. Why? 

 

Conspiracy reasoning vs. Scientific reasoning (the case of QAnon)

Conspiracy theories, unlike scientific theories, are unfalsifiable. Credit: buecax/Depositphotos

Early in the research methods course you learn about the theory-data cycle (Chapter 1). You also learn about the benefits of scientific reasoning over informal, everyday reasoning such as intuition, experience, or authority (Chapter 2). The QAnon conspiracy theory is a good example of several principles in these two chapters.

The quotes from this post come from this excellent story in The Atlantic, by journalist Adrienne LaFrance. 

In case you have never heard of the QAnon conspiracy theory, it started from a series of posts by an anonymous author named Q, who posted occasionally on anonymous "image boards" such as 4chan. Here is a quick recap by LaFrance:

In its broadest contours, the QAnon belief system looks something like this: Q is an intelligence or military insider with proof that corrupt world leaders are secretly torturing children all over the world; the malefactors are embedded in the deep state; Donald Trump is working tirelessly to thwart them. (“These people need to ALL be ELIMINATED,” Q wrote in one post.) The eventual destruction of the global cabal is imminent, Q prophesies, but can be accomplished only with the support of patriots who search for meaning in Q’s clues. To believe Q requires rejecting mainstream institutions, ignoring government officials, battling apostates, and despising the press. 

Part I. Falsifiability
Let's practice describing one way that conspiracy thinking is different from scientific thinking. One of the defining features of scientific thinking is that scientific claims are falsifiable. That is, scientists make claims/hypotheses that can be falsified  (disproven) by collecting and analyzing evidence. 

Most conspiracy theories a cannot be falsified, in part because they are stated in squishy, ambiguous terms, and in part because they are so multifaceted. As LaFrance writes,

In the face of inconvenient facts, [QAnon theory] has the ambiguity and adaptability to sustain a movement of this kind over time. For QAnon, every contradiction can be explained away; no form of argument can prevail against it.

Here is an example of some QAnon reasoning: 

At one point, Harger [a believer she interviewed] told me I should look into what happened to John F. Kennedy Jr.—who died in 1999, when his airplane crashed into the Atlantic Ocean off Martha’s Vineyard—suggesting that Hillary Clinton had had him assassinated. (Alternatively, a contingent of QAnon believers say that JFK Jr. faked his death and that he’s a behind-the-scenes Trump supporter, and possibly even Q himself. Some anticipate his dramatic public return so that he can serve as Trump’s running mate in 2020.) When I asked Harger whether there’s any evidence to support the assassination claim, he flipped my question around: “Is there any evidence not to?”

a) Why is Harger's theory (above) about JFK Jr.'s assassination unfalsifiable?

b) And, How does the theory's unfalsifiable nature make it more likely that Harger can believe in QAnon? 

c) Think like a scientist now. We can write a falsifiable statement (i.e., one that could be demonstrated by evidence to be wrong) related to JFK Jr. For example, to falsify the theory that "J.F.K. Jr. will become Trump's running mate in 2020" we could predict this statement:

"J.F.K. Jr.'s name will appear on the ballot with Trump in the 2020 Presidential election." (This is falsifiable because if someone else's name appears on the ballot, then this theory is wrong.)

Now see if you can come up with a falsifiable statement derived from the theory (above) that J.F.K. Jr.'s plane crash death was an assassination. Your statement can take the form, If J.F.K. Jr. was assassinated, then we should see __________. 

Part II. Accepting evidence
One feature of QAnon theory that makes it impossible to falsify is that its believers do not accept any evidence from legitimate sources. That is, whereas scientists use independently verifiable data to support or falsify theories, QAnon believers do not accept the legitimacy of such data. Here's LaFrance again:

Q frequently rails against legitimate sources of information as fake. Shock and Harger [the two believers] rely on information they encounter on Facebook rather than news outlets run by journalists. They don’t read the local paper or watch any of the major television networks. “You can’t watch the news,” Shock said. “Your news channel ain’t gonna tell us [expletive].” Harger says he likes One America News Network. Not so long ago, he used to watch CNN, and couldn’t get enough of Wolf Blitzer. “We were glued to that; we always have been,” he said. “Until this man, Trump, really opened our eyes to what’s happening. And Q. Q is telling us beforehand the stuff that’s going to happen.” I asked Harger and Shock for examples of predictions that had come true. They could not provide specifics and instead encouraged me to do the research myself. When I asked them how they explained the events Q had predicted that never happened, such as Clinton’s arrest, they said that deception is part of Q’s plan. Shock added, “I think there were more things that were predicted that did happen.” Her tone was gentle rather than indignant.

d) Why does QAnon's rejection of legitimate sources of information help its followers maintain their beliefs? 

As an aside, in the passage above, pay attention to the believer's statement that "deception is part of Q's plan." This feature virtually ensures that the theory remains unfalsifiable: Either believers will point to evidence that supports the theory, or, when faced with evidence that does not support the theory, they will claim that "deception is part of Q's plan." The theory is supported no matter what. 

Part III.  Confirmation bias.
In Chapter 2, you learn about quirks of human reasoning that make our intuition inferior to the scientific method. One of these quirks is the confirmation bias, the tendency of people to notice and remember evidence that confirms their beliefs, and to ignore or discount evidence that disconfirms them. 

e) How might confirmation bias be at play in the passage above, about events that Q predicted but that never happened? 

Confirmation bias is probably more prevalent when situations and theories are ambiguous. Ambiguity is clearly at play in QAnon, because Q usually makes general (rather than specific) threats. LaFrance writes,

—this is a very welcoming belief system, warm in its tolerance for contradiction—and are also what makes it possible for a practical man like Hayes [who has a youtube channel "explaining" Q's theories] to play the role that he does. QAnon is complex and confusing. People from all over the internet seek guidance from someone who seems levelheaded. 

f) Explain how confirmation bias, rather than scientific thinking (which strives to objectively evaluate evidence both for and against a theory), can explain the following reasoning about who Q himself is:

Many QAnon adherents see significance in Trump tweets containing words that begin with the letter Q. Recent world events have rewarded them amply. “I am a great friend and admirer of the Queen & the United Kingdom,” Trump began one tweet on March 29. The day before, he had tweeted this: “I am giving consideration to a QUARANTINE.” The Q crowd seized on both tweets, arguing that if you ignore most of the letters in the messages, you’ll find a confession from Trump: “I am … Q.”

Coda.
QAnon might seem like a harmless academic example of conspiracy theories. And, as LaFrance suggests, some people follow it just for fun (it might have even started out as a parody). However, as LaFrance argues, it has actually incited violence in the past. For example, in 2017, one believer walked into a Washington, D.C. pizza shop with a rifle because he had believed QAnon message boards that children were being held hostage in the shop's basement (the building doesn't have one). (He was wrong: "'The intel on this wasn’t 100 percent,' Welch told The New York Times after his arrest.")  LaFrance also reports, "In 2018, Reddit banned QAnon groups from its platform for inciting violence."  In the United States, several congressional candidates have promoted QAnon's theories. Unfalsifiable conspiracy thinking might give our leaders a (false) sense of confidence, but it certainly doesn't belong in government discourse.  

If QAnon supporters call for violence, revolt, or collective action and people do not think critically about what they say, this fake conspiracy theory will become a real threat to our democracy. 

Here's a great story of a person who used to believe in QAnon and has subsequently changed his mind. 

 

 

 

 

 

 

 

Racial bias in psychological science journals

A recent study found that in psychology, the authors, participants, topics, and editors of empirical journal articles are mostly White.
Photo credit: Africa Studio/Shutterstock

Is science fair?

You learned in Chapter 2 that empirical journals are the primary place where psychological scientists make their research public. Each empirical journal has an editorial board-- a group of scientists who have power to select the peer reviewers for each article and make the final decision about publication. 

Each step of the way, the stated goals of the publication process are to be fair, to focus on the quality of the science, and to publish the best science possible. But is the process fair?  In light of the renewed energy around anti-racism, a timely new study sheds light on racial diversity in the psychology publication process. The researchers asked, Is race an important variable in the scientific publication process?

The team coded all empirical journal articles that appeared in the most prestigious journals in psychology (over 26,000 articles). There were two journals each from social, cognitive, and developmental psychology. The authors measured:

• how often race was studied as a subject in these articles
• the race of the participants in the studies
• the proportion of the authors of scientific articles were White vs. Non-white
• the proportion of the editors of the journals themselves were White vs. Non-white

Here are some of the results of this published study of published studies, as summarized by the Association for Psychological Science's trade publication:

The results suggest that there is a hierarchy in psychology research that determines who and what gets published. “We are not saying anyone has bad intentions,” said Roberts [one of the study's authors]. “It’s a systemic problem across many fields, and psychological science is no exception.”

First, across the past five decades, articles in psychological journals that highlight race have been rare, and although developmental and social psychology journals have published a growing number of these articles, they have remained virtually nonexistent in cognitive psychology.

Second, most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism.

Third, many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color.

Let's parse these claims. 

Questions

a) What kind of a claim is it (frequency, association, or cause) to state that "across the past five decades, articles in psychological journals that highlight race have been rare"

[And what's "rare"? The published report says that "only 5% of publications highlighted race" (p. 4). This 5% figure is an average across cognitive journals (less than 1%) and social (5%) and developmental journals (8%)]

b) Let's take the second claim, "most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism"

The quoted text above provides two claims in one sentence. What is the frequency claim in the quoted text? What is the association claim in the quoted text? What are the two variables in the association claim?

c) Now for the third claim, "many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color."

As with b), the quoted test provides two claims in one sentence, a frequency claim and an association claim. What are these? 

d) In Chapter 1 of the 4th Edition (available Fall 2020), you will read about Merton's norms of science--a set of values that the community of scientists usually endorses and strives  to follow. Here they are, with short descriptions (for a full review, see Chapter 1 or this summary). 

Universalism: Anyone can do science as long as they follow the accepted criteria for science, which are established in advance, are objective, and are accessible to all.
Communality: Share! The process of science, including data, methods, and products, should be free and open for all to use. 
Disinterestedness: The quality of the science should be the only thing that matters, not the prestige of the person or place it was conducted. 
Organized Skepticism: Verify everything. Critical peer review matters, every study should be replicated, and criticism of one's evidence is expected and welcome. 

Roberts and his coauthors argued that psychology is too White. Which of Merton's norms does this result seem to violate, and why? 

Suggested answers:

a) This is a frequency claim--a statement about the frequency of the variable, "highlighting race" across 26,000 journal articles.

b) The frequency claim is about the proportion of journal editors who are White vs. a Person of Color (POC). The association claim is about a link between two variables: whether a journal has a White editor or not, and whether that journal published articles highlighting race or not.

c)  The frequency claim is about the proportion of article authors who are White vs. POC. The association claim is about a link between two variables: whether the article was written by a White author or not, and whether that article included POC as participants or not.

d) Open to discussion!

Here is a summary of this work.

What's the deal with predatory journals?

The published study claimed to have observed one parrot, one penguin, and one woodpecker. Photo Credit: Zdenek Machacek/Unsplash

Today's the first time that the National Audubon Society has been featured on this blog! The story is ornithological (sort of) but it's mainly a parable about predatory journals. The journalist for the Audubon Society tells the story of ornithologist Daniel Baldassarre, who published an article in the legitimate-sounding Scientific Journal of Research and Reviews (SJJR). The Audubon journalist explains that there are several odd features to this empirical article. 

The published article, “What’s the Deal with Birds?”, includes the following abstract: 

Many people wonder: what’s the deal with birds? This is a common query. Birds are pretty weird. I mean, they have feathers. WTF? Most other animals don’t have feathers. To investigate this issue, I looked at some birds. I looked at a woodpecker, a parrot, and a penguin. They were all pretty weird! In conclusion, we may never know the deal with birds, but further study is warranted.

The journalist explains:

...the sample size—a woodpecker, a parrot, and a penguin—... seems suspiciously small. The main figure, a graph plotted along an x-axis ranging from “weird beak” to “looks like a fish,” with a red line labeled “the deal,” is a textbook example of dadaist absurdity. The prose veers wildly between academic terminology and gormless observation. “This is the first study I am aware of to attempt to quantify the deal with birds,” Baldassarre writes. “Unfortunately, the results were ambiguous, although Bayesian approaches may prove useful in the future...When presented with weird behavior, birds exhibited a multimodal response including physical aggression and duetting, both of which were repeatable across highly variable contexts.” 

How did such a ridiculous article get published? Turns out that Dr. Baldasarre was trying to prove a point--some scientific journals are not legitimate--they are predatory, and will accept anything.

...the SJRR is a predatory journal, designed to bilk unwary academics out of money, and Baldassarre’s paper is a joke—quite literally—at their expense. 

[...] Baldasarre first had the idea to submit a joke paper in early February, when the latest in a long line of emails from one of these scam journals landed in his inbox. He slapped together the first iteration of “What’s the Deal With Birds”—a couple of paragraphs in the cursory format of a manuscript—only to see it rejected, perhaps as an obvious parody. Undaunted, he inserted some selections from an earlier legitimate paper to pad it out and resubmitted it to SJRR. 

[...]While SJRR initially demanded a $1,700 publication fee, Baldassarre was eventually able to bargain them down to nothing.

You can access the original paper, including its full-color figure, in its predatory-journal-glory here. (Oh no! It looks like SJRR might have taken it down, given all the negative publicity. Here is an independent link.) 

Questions

a) What is a predatory journal and how is it different from a legitimate journal? (Hint: Use the terms peer review and high quality science in your answer.)

b) If you're doing your own library research for a literature review, how can you know if an article you'd like to use is in a predatory journal or not? One way is to ask your librarian. If you are a student, go to your university's library home page, and find where you can access online help from a university librarian. It might be an online chat, a help desk, or an email consultation service. 

c) One potential way to check if a journal is predatory is to use Beall's List. Go there and try typing in the name of the journal (Scientific Journal of Research and Reviews) in this bird story, or try the name of the journal's publisher, Iris Publishers. Does Beall's List consider this journal or its publisher to be predatory? 

d) Visit the homepage of SJRR's publisher, Iris Publishers. Now visit the homepage of two legitimate publishers, Sage Journals and PLOS, both of which publish legitimate science journals. Just by looking at the websites, can you tell which publishers are legitimate?  

e) Predatory journals make money by charging authors (as much as $2000) to publish their articles. But importantly, just because a journal charges the authors for publication does not mean it is predatory. For example, the journal PlosONE is a legitimate journal that charges authors to publish their work. PlosONE is open access, meaning that anybody can access PlosONE articles for free--you don't have to have a subscription via a university library (i.e., the expenses of publication are transferred from libraries to researchers).  Visit the PlosONE webpage and find an example of a (presumably legitimate) psychology article there. 

f) Draw a Venn diagram or concept map depicting the relationships among the following concepts: predatory publishers, legitimate publishers, predatory journals, legitimate journals, authors pay to publish, open-access, peer reviewed. 

Testing a COVID drug: Design flaws and ethical lapses

Photo credit: Alissa Eckert, MS; Dan Higgins, MAMS/CDC

Here's a good example of a bad study. It's related to the COVID pandemic. In this story by NPR  journalist Vanessa Romo, you can read about a doctor in Texas who claims to be conducting research on the drug hydroxychloroquine, using COVID patients in a nursing home as participants. The doctor's research includes both design problems and ethical problems.

According to the journalist, 

...a doctor at a Texas nursing home started giving the anti-malaria drug hydroxychloroquine to dozens of elderly patients diagnosed with COVID-19 and tracking the outcomes in what he's calling an "observational study".

According to the doctor conducting this study, 

"It's actually going well. People are getting better," Armstrong told NPR, adding that after just a handful of days, some of the 39 patients on the medication are showing signs of improvement.

Armstrong admits it is difficult to quantify how much of his elderly patients' improvement is due to the malaria drug or how they would have fared without it. Nor can he explain why other patients are not responding to the tablet doses, though he notes many are only halfway through the five-day cycle.

Design questions

a) Why can't the doctor decide "how they would have fared without it"? Your answer should discuss the study's need for a comparison group. 

b) The doctor's study includes one group--all of the patients in the "study" are taking the drug. What kind of comparison group will be needed to conduct a true clinical trial?

c) If you're as far as Chapter 11, you'll recognize the design of this doctor's study. What is it?

Here's another quote from the article:

[The doctor] said he started by screening those he believed would benefit most and added more people each day. He monitored their blood oxygen saturation, temperatures and how well they were breathing. "The people who [I decided to put] on it were getting sicker but were not so sick that they had to go the hospital," Armstrong explained.

d) What design problems, if any, to you see with the doctor selecting only this type of patient for being in his "observational study?" And, what might have happened if he had included more severely ill patients for his study? (How might the results change?)

Ethical questions

According to federal guidelines and the ethical principles of the Belmont Report, when a person claims to be doing "research," they should obtain ethical approval from an IRB. Ethical approval would require such procedures as obtaining informed consent (respect for persons), minimizing harm (beneficence), and selecting participants fairly (justice). Read these next quotes from the story:

He acknowledged that some families were not aware their relatives were put on the drug, saying that "for the most part," he consulted with each nursing home resident prior to giving them on the tablets.

While the "overwhelming majority of them are awake and alert and can actually have a conversation," Armstrong said some suffer from middle stages of dementia. In some cases, he did not discuss prescribing the tablets with anyone at all before doing so. He said it is common for physicians to prescribe new medications to patients without explicit consent from the patient or family members. "It's not required," he said.

e) Review the sections of Chapter 4 on "respect for persons," "beneficence," and "justice," and apply details from these concepts to this example. 

f) Do you think study design flaws such as the ones you've noted in a), b) and c) should be considered ethical violations, too? Why or why not? 

Safety questions

A potential issue with this doctor's decision concerns the safety of the drug hydroxychloroquine: According to the journalist:

The Food and Drug Administration has not approved the drug for the treatment of COVID-19. 
and
[the drug] is known to have serious negative health impacts. That is why so many in the medical community worry about prescribing it without such proof.

g) The safety of the drug is relevant to the ethical principle of beneficence. Can you explain why?

Finally...

Another wider ethical issue (perhaps outside the scope of a research methods course) concerns how the doctor obtained the drug, which has been in short supply:

[The doctor] began making calls to track down a source for the medicine, which is in short supply.That's when his political connections proved useful.

Armstrong, who is a prominent GOP activist, called Republican Lt. Gov. Dan Patrick. He says Patrick reached out to Texas state Sen. Bryan Hughes, also a Republican, who knew someone on the board of the New Jersey-based company Amneal Pharmaceuticals. The company, which makes and distributes the drug, has donated more than a million tablets nationwide, including to the statesof Texas and Louisiana.Two days later, Armstrong had received more than enough medication to begin giving it to patients. 

h) Do you think it is reasonable for a doctor to use political connections to obtain medications that are in restricted supply? Why or why not? 

Suggested answers to selected questions

a) and b) and c) Because the doctor's "study" has only a group of patients who are taking the drug, he doesn't have a comparison group. In Chapter 11 this is called a one-group, pretest-posttest design, otherwise known as the Really Bad Experiment. The design is flawed. If patients improve on the drug, the doctor will not know whether their improvement was caused by the drug itself, or simply because their bodies were healing or because they were getting excellent nursing care, or even some other reason. 

The doctor's behavior in this COVID study reminds me of the Halstead radical mastectomy, which for decades was the dominant treatment for breast cancer in the U.S.. Surgeons at the time were so convinced that this treatment was the best one for breast cancer that they never bothered to compare it to another treatment--they never tested its efficacy. (When a randomized control study was finally conducted in the 1970s, it found that the surgery was no more effective than other, less invasive treatments.)

d) By selecting only relatively less sick patients, the doctor is stacking the deck in favor of finding evidence that the drug is effective. Many of the patients are likely to get better, so the doctor may conclude that the drug has worked. But of course, without a comparison group of equally healthy people who are not getting the drug, we cannot know if the patients would have recovered anyway.

e) f) g) The lack of informed consent--especially for vulnerable patients who have Alzheimer's--is a clear violation of the respect for persons principle. And the use of a risky treatment, one that is known to be dangerous to people with heart conditions, is likely a violation of the beneficence principle. I would argue that when a researcher conducts a flawed study like this, it also a violation of beneficence--the study puts participants at risk, while the benefit of the potential information gained from the research is negligible. We might also argue that the study violates justice: by studying only people at a nursing home, these patients bear the burden of being research participation when anyone can benefit from this research (assuming, of course, that the research were actually well-designed).

Many thanks to Dr. Stephen Chew for pointing me to this example!

Post script: Looking for more examples of bad studies, as described by science writers? Here are two:

Here's a critique of the many problems with another early published study on hydroxychloroquine.

Here's critique of studies supposedly investigating the death rate of COVID-19. 

Is "sugar high" a thing?

 

Sugar doesn't make kids hyperactive; at least not according to a null effect study and a meta-analysis. Photo: Alamy

Here's something "everybody says":  Kids will become hyperactive if they eat too much sugar. This folk belief is as much a part of North American cultural wisdom as learning styles and the catharsis hypothesis. But this column in the New York Times explains that, like learning styles and catharsis, the idea of a "sugar high" is also a myth. 

 

The physician who wrote the column traced the modern idea of sugar causing hyperactivity to 1975. But then the author describes a randomized clinical trial on this question. Here's the description: 

An extraordinarily rigorous study settled the question in 1994. Writing in the New England Journal of Medicine, a group of scientists tested normal preschoolers and children whose parents described them as being sensitive to sugar. Neither the parents, the children nor the research staff knew which of the children were getting sugary foods and which were getting a diet sweetened with aspartame and other artificial sweeteners. Urine was tested to verify compliance with the diets. Nine different measures of cognitive and behavioral performance were assessed, with measurements taken at five-second intervals.

The original study was, indeed, ambitious. The authors provided all food to the homes of the children in the study so they could completely control what sweeteners were being used. The study took place over nine weeks in a repeated-measures design--the diets were based on sucrose (sugar) one week, aspartame another week, and saccharine during another week. The researchers measured the children's cognitive performance and mood throughout the study.

The study concluded that sugar does not affect children’s behavior or cognitive function. An editorial that accompanied the study put a fine point on that conclusion, stating “there is no evidence that sugar alone can turn a child with normal attention into a hyperactive child.” 

Questions

a) What were the independent and dependent variables in the study described here? What are the levels of the IV?

b) The IV was manipulated as within subjects, using a repeated-measures design. Describe why counterbalancing would be important here, and how you could counterbalance in this study. Which of the four big validities does counterbalancing support?

c) What questions could you ask about the other three validities for this study? 

d) This study found a null effect (no impact of sucrose on children's behavior). There were only 48 children in the diet study, and this might be considered a low number--there may not have been enough kids in the study to detect a small effect of sucrose. However, the repeated measures design means this study can make a stronger case for the conclusion that sugar doesn't affect kids' behavior. Review the information in Chapter 11 to explain why the repeated measures design helps support the conclusion. 

The author of the piece mentions this, too:

One year later, an analysis that gathered data from all published studies on the subject reached the same conclusion.

e) This sentence is describing a meta-analysis. In your own words, describe what steps they would have taken to conduct this meta-analysis.

f) Why is a meta-analysis stronger evidence for a null effect of sugar on hyperactivity? 

g) Given this information, what would you say to a relative or friend who believes that sugar causes hyperactivity in their own child?