Everyday Research Methods

Music festivals "make people more generous, socially connected"

Credit: halfpoint/Deposit Photos

Summer is the time for music festivals like Burning Man, Burning Nest, Firefly, and Bonnaroo. If you are planning to attend one this year, this post might interest you.

The science journalism clearinghouse called Study Finds recently reported on a study conducted at Yale University. The journalists' report included the causal headline that Music festivals like Burning Man make people more generous, socially connected to others". 

a) What are the variables in this causal headline, and what makes it a causal claim?

b) In general, researchers need to conduct an experiment in order to support a causal claim. Do you see any barriers to conducting an experiment on the causal effects of music festivals?

It will be interesting to read about whether the researchers' study can feasibly support the causal claim that the journalist attached to it. Here's how Study Finds described the study's method: 

The Yale researchers studied the behavior and experience of over 1,200 people attending festivals such as Burning Man, Burning Nest, Lightning in a Bottle, Dirty Bird, and Latitude. All of the festivals took place in the United States or the United Kingdom and emphasized art, music, and self-expression. The team set up booths at all the events, which they called “Play Games for Science.”

People ... self-reported their experiences at the event and whether they were willing to share their resources with friends and strangers. For example, week-long festivals such as Burning Man do not sell food, and people who find themselves short must rely on the generosity of others.

[...] Overall, 63.2 percent of people said they had a transformative experience and felt themselves changing. ... One of the takeaways from the transformative experience was feeling more connected to other humans, which made them more likely to share food and resources with other attendees.

c) So far, do the method and results of this study (as reported by the journalist) support the conclusion that attending the festivals caused a transformative experience? Why or why not? (Hint: Think about the need for a comparison group.)

d) Based on what you've read, was the variable of "transformative experience" operationalized as a physiological, observational, or self-report measure?

In this case, the journalist's coverage of the study doesn't do the original study justice, as I discovered when I read the original empirical article, which appeared in the journal Nature Communication (you can access it here).There are several important omissions in the journalist's report. One is that the researchers themselves did not use causal language in their work. Instead, their article is titled "Prosocial correlates of transformative experiences at secular multi-day mass gatherings" (my emphasis).

The Study Finds journalist also omitted key methods and variables. For example, one important variable in the original study was that the researchers graphed people's self-reports of having a transformative experience on each day of a 7-day festival (see Figure 2,  here). Using this method, the researchers found that the later the day (going from Day 1 to Day 7), the more likely respondents were to report having a transformative experience. 

e) Why is the above method and result important, in terms of supporting the claim that the music festivals can increase people's tendency to have a transformative experience? (Hint: think about temporal precedence.)

Another important detail omitted by the Study Finds story, is that in addition to asking people to self-report their transformative experiences, researchers also asked participants to imagine playing a "dictator game". In this game, participants are given 10 valuable tickets. Participants can then donate some number of their tickets to randomly selected stranger. The dictator game is an observational measure of generosity to strangers. The researchers found that festival participants chose to donate between 50 and 75% of their tickets to the stranger. This was much higher than the level of donations typically found, as reported in a meta-analysis of all previous studies using the dictator game (which found an average of 26%). 

f)  This dictator game result is a type of quasi-experiment--we might call it a non-equivalent control group, posttest-only design. What is the quasi-independent variable--that is, what are the two levels of the non-equivalent control group (and why are they "non-equivalent")? What is the dependent variable? 

g) The result suggests that attending the music festival might cause people to be more generous in the dictator game, but there might be a selection effect. Why? 

 

Conspiracy reasoning vs. Scientific reasoning (the case of QAnon)

Conspiracy theories, unlike scientific theories, are unfalsifiable. Credit: buecax/Depositphotos

Early in the research methods course you learn about the theory-data cycle (Chapter 1). You also learn about the benefits of scientific reasoning over informal, everyday reasoning such as intuition, experience, or authority (Chapter 2). The QAnon conspiracy theory is a good example of several principles in these two chapters.

The quotes from this post come from this excellent story in The Atlantic, by journalist Adrienne LaFrance. 

In case you have never heard of the QAnon conspiracy theory, it started from a series of posts by an anonymous author named Q, who posted occasionally on anonymous "image boards" such as 4chan. Here is a quick recap by LaFrance:

In its broadest contours, the QAnon belief system looks something like this: Q is an intelligence or military insider with proof that corrupt world leaders are secretly torturing children all over the world; the malefactors are embedded in the deep state; Donald Trump is working tirelessly to thwart them. (“These people need to ALL be ELIMINATED,” Q wrote in one post.) The eventual destruction of the global cabal is imminent, Q prophesies, but can be accomplished only with the support of patriots who search for meaning in Q’s clues. To believe Q requires rejecting mainstream institutions, ignoring government officials, battling apostates, and despising the press. 

Part I. Falsifiability
Let's practice describing one way that conspiracy thinking is different from scientific thinking. One of the defining features of scientific thinking is that scientific claims are falsifiable. That is, scientists make claims/hypotheses that can be falsified  (disproven) by collecting and analyzing evidence. 

Most conspiracy theories a cannot be falsified, in part because they are stated in squishy, ambiguous terms, and in part because they are so multifaceted. As LaFrance writes,

In the face of inconvenient facts, [QAnon theory] has the ambiguity and adaptability to sustain a movement of this kind over time. For QAnon, every contradiction can be explained away; no form of argument can prevail against it.

Here is an example of some QAnon reasoning: 

At one point, Harger [a believer she interviewed] told me I should look into what happened to John F. Kennedy Jr.—who died in 1999, when his airplane crashed into the Atlantic Ocean off Martha’s Vineyard—suggesting that Hillary Clinton had had him assassinated. (Alternatively, a contingent of QAnon believers say that JFK Jr. faked his death and that he’s a behind-the-scenes Trump supporter, and possibly even Q himself. Some anticipate his dramatic public return so that he can serve as Trump’s running mate in 2020.) When I asked Harger whether there’s any evidence to support the assassination claim, he flipped my question around: “Is there any evidence not to?”

a) Why is Harger's theory (above) about JFK Jr.'s assassination unfalsifiable?

b) And, How does the theory's unfalsifiable nature make it more likely that Harger can believe in QAnon? 

c) Think like a scientist now. We can write a falsifiable statement (i.e., one that could be demonstrated by evidence to be wrong) related to JFK Jr. For example, to falsify the theory that "J.F.K. Jr. will become Trump's running mate in 2020" we could predict this statement:

"J.F.K. Jr.'s name will appear on the ballot with Trump in the 2020 Presidential election." (This is falsifiable because if someone else's name appears on the ballot, then this theory is wrong.)

Now see if you can come up with a falsifiable statement derived from the theory (above) that J.F.K. Jr.'s plane crash death was an assassination. Your statement can take the form, If J.F.K. Jr. was assassinated, then we should see __________. 

Part II. Accepting evidence
One feature of QAnon theory that makes it impossible to falsify is that its believers do not accept any evidence from legitimate sources. That is, whereas scientists use independently verifiable data to support or falsify theories, QAnon believers do not accept the legitimacy of such data. Here's LaFrance again:

Q frequently rails against legitimate sources of information as fake. Shock and Harger [the two believers] rely on information they encounter on Facebook rather than news outlets run by journalists. They don’t read the local paper or watch any of the major television networks. “You can’t watch the news,” Shock said. “Your news channel ain’t gonna tell us [expletive].” Harger says he likes One America News Network. Not so long ago, he used to watch CNN, and couldn’t get enough of Wolf Blitzer. “We were glued to that; we always have been,” he said. “Until this man, Trump, really opened our eyes to what’s happening. And Q. Q is telling us beforehand the stuff that’s going to happen.” I asked Harger and Shock for examples of predictions that had come true. They could not provide specifics and instead encouraged me to do the research myself. When I asked them how they explained the events Q had predicted that never happened, such as Clinton’s arrest, they said that deception is part of Q’s plan. Shock added, “I think there were more things that were predicted that did happen.” Her tone was gentle rather than indignant.

d) Why does QAnon's rejection of legitimate sources of information help its followers maintain their beliefs? 

As an aside, in the passage above, pay attention to the believer's statement that "deception is part of Q's plan." This feature virtually ensures that the theory remains unfalsifiable: Either believers will point to evidence that supports the theory, or, when faced with evidence that does not support the theory, they will claim that "deception is part of Q's plan." The theory is supported no matter what. 

Part III.  Confirmation bias.
In Chapter 2, you learn about quirks of human reasoning that make our intuition inferior to the scientific method. One of these quirks is the confirmation bias, the tendency of people to notice and remember evidence that confirms their beliefs, and to ignore or discount evidence that disconfirms them. 

e) How might confirmation bias be at play in the passage above, about events that Q predicted but that never happened? 

Confirmation bias is probably more prevalent when situations and theories are ambiguous. Ambiguity is clearly at play in QAnon, because Q usually makes general (rather than specific) threats. LaFrance writes,

—this is a very welcoming belief system, warm in its tolerance for contradiction—and are also what makes it possible for a practical man like Hayes [who has a youtube channel "explaining" Q's theories] to play the role that he does. QAnon is complex and confusing. People from all over the internet seek guidance from someone who seems levelheaded. 

f) Explain how confirmation bias, rather than scientific thinking (which strives to objectively evaluate evidence both for and against a theory), can explain the following reasoning about who Q himself is:

Many QAnon adherents see significance in Trump tweets containing words that begin with the letter Q. Recent world events have rewarded them amply. “I am a great friend and admirer of the Queen & the United Kingdom,” Trump began one tweet on March 29. The day before, he had tweeted this: “I am giving consideration to a QUARANTINE.” The Q crowd seized on both tweets, arguing that if you ignore most of the letters in the messages, you’ll find a confession from Trump: “I am … Q.”

Coda.
QAnon might seem like a harmless academic example of conspiracy theories. And, as LaFrance suggests, some people follow it just for fun (it might have even started out as a parody). However, as LaFrance argues, it has actually incited violence in the past. For example, in 2017, one believer walked into a Washington, D.C. pizza shop with a rifle because he had believed QAnon message boards that children were being held hostage in the shop's basement (the building doesn't have one). (He was wrong: "'The intel on this wasn’t 100 percent,' Welch told The New York Times after his arrest.")  LaFrance also reports, "In 2018, Reddit banned QAnon groups from its platform for inciting violence."  In the United States, several congressional candidates have promoted QAnon's theories. Unfalsifiable conspiracy thinking might give our leaders a (false) sense of confidence, but it certainly doesn't belong in government discourse.  

If QAnon supporters call for violence, revolt, or collective action and people do not think critically about what they say, this fake conspiracy theory will become a real threat to our democracy. 

Here's a great story of a person who used to believe in QAnon and has subsequently changed his mind. 

 

 

 

 

 

 

 

Racial bias in psychological science journals

A recent study found that in psychology, the authors, participants, topics, and editors of empirical journal articles are mostly White.
Photo credit: Africa Studio/Shutterstock

Is science fair?

You learned in Chapter 2 that empirical journals are the primary place where psychological scientists make their research public. Each empirical journal has an editorial board-- a group of scientists who have power to select the peer reviewers for each article and make the final decision about publication. 

Each step of the way, the stated goals of the publication process are to be fair, to focus on the quality of the science, and to publish the best science possible. But is the process fair?  In light of the renewed energy around anti-racism, a timely new study sheds light on racial diversity in the psychology publication process. The researchers asked, Is race an important variable in the scientific publication process?

The team coded all empirical journal articles that appeared in the most prestigious journals in psychology (over 26,000 articles). There were two journals each from social, cognitive, and developmental psychology. The authors measured:

• how often race was studied as a subject in these articles
• the race of the participants in the studies
• the proportion of the authors of scientific articles were White vs. Non-white
• the proportion of the editors of the journals themselves were White vs. Non-white

Here are some of the results of this published study of published studies, as summarized by the Association for Psychological Science's trade publication:

The results suggest that there is a hierarchy in psychology research that determines who and what gets published. “We are not saying anyone has bad intentions,” said Roberts [one of the study's authors]. “It’s a systemic problem across many fields, and psychological science is no exception.”

First, across the past five decades, articles in psychological journals that highlight race have been rare, and although developmental and social psychology journals have published a growing number of these articles, they have remained virtually nonexistent in cognitive psychology.

Second, most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism.

Third, many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color.

Let's parse these claims. 

Questions

a) What kind of a claim is it (frequency, association, or cause) to state that "across the past five decades, articles in psychological journals that highlight race have been rare"

[And what's "rare"? The published report says that "only 5% of publications highlighted race" (p. 4). This 5% figure is an average across cognitive journals (less than 1%) and social (5%) and developmental journals (8%)]

b) Let's take the second claim, "most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism"

The quoted text above provides two claims in one sentence. What is the frequency claim in the quoted text? What is the association claim in the quoted text? What are the two variables in the association claim?

c) Now for the third claim, "many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color."

As with b), the quoted test provides two claims in one sentence, a frequency claim and an association claim. What are these? 

d) In Chapter 1 of the 4th Edition (available Fall 2020), you will read about Merton's norms of science--a set of values that the community of scientists usually endorses and strives  to follow. Here they are, with short descriptions (for a full review, see Chapter 1 or this summary). 

Universalism: Anyone can do science as long as they follow the accepted criteria for science, which are established in advance, are objective, and are accessible to all.
Communality: Share! The process of science, including data, methods, and products, should be free and open for all to use. 
Disinterestedness: The quality of the science should be the only thing that matters, not the prestige of the person or place it was conducted. 
Organized Skepticism: Verify everything. Critical peer review matters, every study should be replicated, and criticism of one's evidence is expected and welcome. 

Roberts and his coauthors argued that psychology is too White. Which of Merton's norms does this result seem to violate, and why? 

Suggested answers:

a) This is a frequency claim--a statement about the frequency of the variable, "highlighting race" across 26,000 journal articles.

b) The frequency claim is about the proportion of journal editors who are White vs. a Person of Color (POC). The association claim is about a link between two variables: whether a journal has a White editor or not, and whether that journal published articles highlighting race or not.

c)  The frequency claim is about the proportion of article authors who are White vs. POC. The association claim is about a link between two variables: whether the article was written by a White author or not, and whether that article included POC as participants or not.

d) Open to discussion!

Here is a summary of this work.

What's the deal with predatory journals?

The published study claimed to have observed one parrot, one penguin, and one woodpecker. Photo Credit: Zdenek Machacek/Unsplash

Today's the first time that the National Audubon Society has been featured on this blog! The story is ornithological (sort of) but it's mainly a parable about predatory journals. The journalist for the Audubon Society tells the story of ornithologist Daniel Baldassarre, who published an article in the legitimate-sounding Scientific Journal of Research and Reviews (SJJR). The Audubon journalist explains that there are several odd features to this empirical article. 

The published article, “What’s the Deal with Birds?”, includes the following abstract: 

Many people wonder: what’s the deal with birds? This is a common query. Birds are pretty weird. I mean, they have feathers. WTF? Most other animals don’t have feathers. To investigate this issue, I looked at some birds. I looked at a woodpecker, a parrot, and a penguin. They were all pretty weird! In conclusion, we may never know the deal with birds, but further study is warranted.

The journalist explains:

...the sample size—a woodpecker, a parrot, and a penguin—... seems suspiciously small. The main figure, a graph plotted along an x-axis ranging from “weird beak” to “looks like a fish,” with a red line labeled “the deal,” is a textbook example of dadaist absurdity. The prose veers wildly between academic terminology and gormless observation. “This is the first study I am aware of to attempt to quantify the deal with birds,” Baldassarre writes. “Unfortunately, the results were ambiguous, although Bayesian approaches may prove useful in the future...When presented with weird behavior, birds exhibited a multimodal response including physical aggression and duetting, both of which were repeatable across highly variable contexts.” 

How did such a ridiculous article get published? Turns out that Dr. Baldasarre was trying to prove a point--some scientific journals are not legitimate--they are predatory, and will accept anything.

...the SJRR is a predatory journal, designed to bilk unwary academics out of money, and Baldassarre’s paper is a joke—quite literally—at their expense. 

[...] Baldasarre first had the idea to submit a joke paper in early February, when the latest in a long line of emails from one of these scam journals landed in his inbox. He slapped together the first iteration of “What’s the Deal With Birds”—a couple of paragraphs in the cursory format of a manuscript—only to see it rejected, perhaps as an obvious parody. Undaunted, he inserted some selections from an earlier legitimate paper to pad it out and resubmitted it to SJRR. 

[...]While SJRR initially demanded a $1,700 publication fee, Baldassarre was eventually able to bargain them down to nothing.

You can access the original paper, including its full-color figure, in its predatory-journal-glory here. (Oh no! It looks like SJRR might have taken it down, given all the negative publicity. Here is an independent link.) 

Questions

a) What is a predatory journal and how is it different from a legitimate journal? (Hint: Use the terms peer review and high quality science in your answer.)

b) If you're doing your own library research for a literature review, how can you know if an article you'd like to use is in a predatory journal or not? One way is to ask your librarian. If you are a student, go to your university's library home page, and find where you can access online help from a university librarian. It might be an online chat, a help desk, or an email consultation service. 

c) One potential way to check if a journal is predatory is to use Beall's List. Go there and try typing in the name of the journal (Scientific Journal of Research and Reviews) in this bird story, or try the name of the journal's publisher, Iris Publishers. Does Beall's List consider this journal or its publisher to be predatory? 

d) Visit the homepage of SJRR's publisher, Iris Publishers. Now visit the homepage of two legitimate publishers, Sage Journals and PLOS, both of which publish legitimate science journals. Just by looking at the websites, can you tell which publishers are legitimate?  

e) Predatory journals make money by charging authors (as much as $2000) to publish their articles. But importantly, just because a journal charges the authors for publication does not mean it is predatory. For example, the journal PlosONE is a legitimate journal that charges authors to publish their work. PlosONE is open access, meaning that anybody can access PlosONE articles for free--you don't have to have a subscription via a university library (i.e., the expenses of publication are transferred from libraries to researchers).  Visit the PlosONE webpage and find an example of a (presumably legitimate) psychology article there. 

f) Draw a Venn diagram or concept map depicting the relationships among the following concepts: predatory publishers, legitimate publishers, predatory journals, legitimate journals, authors pay to publish, open-access, peer reviewed. 

Testing a COVID drug: Design flaws and ethical lapses

Photo credit: Alissa Eckert, MS; Dan Higgins, MAMS/CDC

Here's a good example of a bad study. It's related to the COVID pandemic. In this story by NPR  journalist Vanessa Romo, you can read about a doctor in Texas who claims to be conducting research on the drug hydroxychloroquine, using COVID patients in a nursing home as participants. The doctor's research includes both design problems and ethical problems.

According to the journalist, 

...a doctor at a Texas nursing home started giving the anti-malaria drug hydroxychloroquine to dozens of elderly patients diagnosed with COVID-19 and tracking the outcomes in what he's calling an "observational study".

According to the doctor conducting this study, 

"It's actually going well. People are getting better," Armstrong told NPR, adding that after just a handful of days, some of the 39 patients on the medication are showing signs of improvement.

Armstrong admits it is difficult to quantify how much of his elderly patients' improvement is due to the malaria drug or how they would have fared without it. Nor can he explain why other patients are not responding to the tablet doses, though he notes many are only halfway through the five-day cycle.

Design questions

a) Why can't the doctor decide "how they would have fared without it"? Your answer should discuss the study's need for a comparison group. 

b) The doctor's study includes one group--all of the patients in the "study" are taking the drug. What kind of comparison group will be needed to conduct a true clinical trial?

c) If you're as far as Chapter 11, you'll recognize the design of this doctor's study. What is it?

Here's another quote from the article:

[The doctor] said he started by screening those he believed would benefit most and added more people each day. He monitored their blood oxygen saturation, temperatures and how well they were breathing. "The people who [I decided to put] on it were getting sicker but were not so sick that they had to go the hospital," Armstrong explained.

d) What design problems, if any, to you see with the doctor selecting only this type of patient for being in his "observational study?" And, what might have happened if he had included more severely ill patients for his study? (How might the results change?)

Ethical questions

According to federal guidelines and the ethical principles of the Belmont Report, when a person claims to be doing "research," they should obtain ethical approval from an IRB. Ethical approval would require such procedures as obtaining informed consent (respect for persons), minimizing harm (beneficence), and selecting participants fairly (justice). Read these next quotes from the story:

He acknowledged that some families were not aware their relatives were put on the drug, saying that "for the most part," he consulted with each nursing home resident prior to giving them on the tablets.

While the "overwhelming majority of them are awake and alert and can actually have a conversation," Armstrong said some suffer from middle stages of dementia. In some cases, he did not discuss prescribing the tablets with anyone at all before doing so. He said it is common for physicians to prescribe new medications to patients without explicit consent from the patient or family members. "It's not required," he said.

e) Review the sections of Chapter 4 on "respect for persons," "beneficence," and "justice," and apply details from these concepts to this example. 

f) Do you think study design flaws such as the ones you've noted in a), b) and c) should be considered ethical violations, too? Why or why not? 

Safety questions

A potential issue with this doctor's decision concerns the safety of the drug hydroxychloroquine: According to the journalist:

The Food and Drug Administration has not approved the drug for the treatment of COVID-19. 
and
[the drug] is known to have serious negative health impacts. That is why so many in the medical community worry about prescribing it without such proof.

g) The safety of the drug is relevant to the ethical principle of beneficence. Can you explain why?

Finally...

Another wider ethical issue (perhaps outside the scope of a research methods course) concerns how the doctor obtained the drug, which has been in short supply:

[The doctor] began making calls to track down a source for the medicine, which is in short supply.That's when his political connections proved useful.

Armstrong, who is a prominent GOP activist, called Republican Lt. Gov. Dan Patrick. He says Patrick reached out to Texas state Sen. Bryan Hughes, also a Republican, who knew someone on the board of the New Jersey-based company Amneal Pharmaceuticals. The company, which makes and distributes the drug, has donated more than a million tablets nationwide, including to the statesof Texas and Louisiana.Two days later, Armstrong had received more than enough medication to begin giving it to patients. 

h) Do you think it is reasonable for a doctor to use political connections to obtain medications that are in restricted supply? Why or why not? 

Suggested answers to selected questions

a) and b) and c) Because the doctor's "study" has only a group of patients who are taking the drug, he doesn't have a comparison group. In Chapter 11 this is called a one-group, pretest-posttest design, otherwise known as the Really Bad Experiment. The design is flawed. If patients improve on the drug, the doctor will not know whether their improvement was caused by the drug itself, or simply because their bodies were healing or because they were getting excellent nursing care, or even some other reason. 

The doctor's behavior in this COVID study reminds me of the Halstead radical mastectomy, which for decades was the dominant treatment for breast cancer in the U.S.. Surgeons at the time were so convinced that this treatment was the best one for breast cancer that they never bothered to compare it to another treatment--they never tested its efficacy. (When a randomized control study was finally conducted in the 1970s, it found that the surgery was no more effective than other, less invasive treatments.)

d) By selecting only relatively less sick patients, the doctor is stacking the deck in favor of finding evidence that the drug is effective. Many of the patients are likely to get better, so the doctor may conclude that the drug has worked. But of course, without a comparison group of equally healthy people who are not getting the drug, we cannot know if the patients would have recovered anyway.

e) f) g) The lack of informed consent--especially for vulnerable patients who have Alzheimer's--is a clear violation of the respect for persons principle. And the use of a risky treatment, one that is known to be dangerous to people with heart conditions, is likely a violation of the beneficence principle. I would argue that when a researcher conducts a flawed study like this, it also a violation of beneficence--the study puts participants at risk, while the benefit of the potential information gained from the research is negligible. We might also argue that the study violates justice: by studying only people at a nursing home, these patients bear the burden of being research participation when anyone can benefit from this research (assuming, of course, that the research were actually well-designed).

Many thanks to Dr. Stephen Chew for pointing me to this example!

Post script: Looking for more examples of bad studies, as described by science writers? Here are two:

Here's a critique of the many problems with another early published study on hydroxychloroquine.

Here's critique of studies supposedly investigating the death rate of COVID-19. 

Is "sugar high" a thing?

 

Sugar doesn't make kids hyperactive; at least not according to a null effect study and a meta-analysis. Photo: Alamy

Here's something "everybody says":  Kids will become hyperactive if they eat too much sugar. This folk belief is as much a part of North American cultural wisdom as learning styles and the catharsis hypothesis. But this column in the New York Times explains that, like learning styles and catharsis, the idea of a "sugar high" is also a myth. 

 

The physician who wrote the column traced the modern idea of sugar causing hyperactivity to 1975. But then the author describes a randomized clinical trial on this question. Here's the description: 

An extraordinarily rigorous study settled the question in 1994. Writing in the New England Journal of Medicine, a group of scientists tested normal preschoolers and children whose parents described them as being sensitive to sugar. Neither the parents, the children nor the research staff knew which of the children were getting sugary foods and which were getting a diet sweetened with aspartame and other artificial sweeteners. Urine was tested to verify compliance with the diets. Nine different measures of cognitive and behavioral performance were assessed, with measurements taken at five-second intervals.

The original study was, indeed, ambitious. The authors provided all food to the homes of the children in the study so they could completely control what sweeteners were being used. The study took place over nine weeks in a repeated-measures design--the diets were based on sucrose (sugar) one week, aspartame another week, and saccharine during another week. The researchers measured the children's cognitive performance and mood throughout the study.

The study concluded that sugar does not affect children’s behavior or cognitive function. An editorial that accompanied the study put a fine point on that conclusion, stating “there is no evidence that sugar alone can turn a child with normal attention into a hyperactive child.” 

Questions

a) What were the independent and dependent variables in the study described here? What are the levels of the IV?

b) The IV was manipulated as within subjects, using a repeated-measures design. Describe why counterbalancing would be important here, and how you could counterbalance in this study. Which of the four big validities does counterbalancing support?

c) What questions could you ask about the other three validities for this study? 

d) This study found a null effect (no impact of sucrose on children's behavior). There were only 48 children in the diet study, and this might be considered a low number--there may not have been enough kids in the study to detect a small effect of sucrose. However, the repeated measures design means this study can make a stronger case for the conclusion that sugar doesn't affect kids' behavior. Review the information in Chapter 11 to explain why the repeated measures design helps support the conclusion. 

The author of the piece mentions this, too:

One year later, an analysis that gathered data from all published studies on the subject reached the same conclusion.

e) This sentence is describing a meta-analysis. In your own words, describe what steps they would have taken to conduct this meta-analysis.

f) Why is a meta-analysis stronger evidence for a null effect of sugar on hyperactivity? 

g) Given this information, what would you say to a relative or friend who believes that sugar causes hyperactivity in their own child? 

 

Training tools for information literacy

Photo: fotogestoeber/Shutterstock

More than ever, our citizenry needs skills for information literacy. A recent study of almost 4000 U.S. teens, conducted by Stanford's History Education group, concluded that teenagers' ability to think critically about online information is "troubling." Their report included the findings that: 

• Fifty-two percent of students believed a grainy video claiming to show ballot stuffing in the 2016 Democratic primaries (the video was actually shot in Russia) constituted “strong evidence” of voter fraud in the U.S. [...]
• Two-thirds of students couldn’t tell the difference between news stories and ads (set off by the words “Sponsored Content”) on Slate’s homepage.
• Ninety-six percent of students did not consider why ties between a climate change website and the fossil fuel industry might lessen that website’s credibility. [...]

If you're inspired to build skills in your own students, consider assigning an activity designed to develop students' critical reading skills as a supplement to Chapter 2. The following tools have been developed by experts in library science, communication, and social cognition. All of these raise our awareness of the information we may encounter online: 

Games that Teach

1. The free online game Bad News  trains people to spread false ideas with the goal of teaching them how propaganda works. It also includes an (optional) embedded test of people's ability to detect real and fake Twitter accounts. Users earn badges for impersonating experts, exploiting emotion, and polarization. It's a legitimately fun game! What's more, two recent studies have indicated that this game improves people's reasoning about the news (Roozenbeek et al., 2019;  Basol et al., 2020).

2. The Fake News Game (still in development) plans to teach about fake news by challenging people to create their own fake news, building in elements intended to make it go viral. 

Online Curricula

3. Stanford's Civic Online Reasoning (COR) course presents several planned sets of lessons:

• • An three-lesson introduction to skeptical reading, looking for evidence, and fact-checking
  • An six-lesson overview of lateral reading, the strategy used by professional fact-checkers (and a skill associated with better information literacy)
  • A combination package of six lessons offering a little bit of everything (lateral reading, introduction to evidence, and "click restraint"). This might be the package to assign if you're only assigning one set of lessons.

4. Carl Bergstrom and Jevin West's site, Calling Bullshit, offers a syllabus for a full semester college course on misinformation. You can also assign some of their "tools and tricks" to give people practice in the following skills:

• • Spotting misleading axes and formatting in graphs
  • Deciding if a scientific paper is legitimate
  • Deciding if an online face is real or computer-generated
  • They also have case studies in misinformation, sorted by difficulty. 

5. Morton Ann Gernsbacher has developed and shared her free, online methods course focused primarily on scientific information literacy. Check it out for a full set of resources!

My own students have benefitted from working through various modules from these resources, in both methods and content courses. 

I'm grateful to David Dunning (University of Michigan), who shared some of these resources at his NITOP general session. I'm also grateful to UD librarians Meg Grotti and Alison Wessel for sharing resources with me. 

Can Probiotics Actually Treat Obesity?

 

Should you be suspicious of a study conducted by a company that produces probiotics products? Photo: Science History Images / Alamy Stock Photo

Guest post by Dr. Dina Gohar, University of Michigan

 

Hippocrates, considered to be the Father of Medicine, reportedly said, “all disease begins in the gut. ” It took doctors centuries to start studying the connection between our gut microbiota and our health, which is now well-supported by research. We can improve the health and diversity of our gut microbiota with active strategies, such as exercising regularly, eating a healthy diet, and getting enough sleep. We might also be able to improve our gut microbiota by simply consuming live “good” bacteria and microorganisms called probiotics. Probiotics are now sold as capsules, pills, and powders, added to fruit juices, granola bars, and candy, and even put into cosmetics. Fermented foods, such as cultured yogurt, sauerkraut, and kombucha, contain them too. A cloud of hype and hope surrounds probiotics, which have been touted as a treatment (and means of prevention) for everything from intestinal disorders, mood and anxiety disorders, belly fat and obesity-related disorders. 

A recent article in the New York Times suggested that a probiotic of heat-killed bacteria can treat metabolic disorders like obesity. This suggestion was based on a study in Nature Medicine.The  New York Times journalist summarized the study as follows:

Researchers tested the bacterium, Akkermansia muciniphila, in 32 men and women who met the criteria for metabolic syndrome by having at least three of five conditions: high fasting blood sugar, high blood pressure, high triglycerides, low HDL (the “good” cholesterol) or excessive waist circumference.

…In a three-month trial, volunteers were randomized to one of three groups: daily tablets containing live bacteria, pasteurized bacteria [A. muciniphila] or a placebo. Compared with the placebo group, those who took pasteurized A. muciniphila had significantly improved insulin sensitivity and total cholesterol, and decreases in several blood markers of inflammation and liver dysfunction. They also had decreased body weight, fat mass and waist circumference, though those differences were not statistically significant. The live bacteria were largely ineffective.

1. This study was an experiment. What is the independent variable (IV) and what are its levels? What are the dependent variables (DV)? What do you think of the outcome measures chosen for this study?
   
   
2. How was the independent variable manipulated:  between or within groups? Which of the four basic experiments is it probably an example of: Pretest/postttest design? Posttest-only? Repeated measures? or Concurrent measures?
   
   
3. Can the results of this study be used to support the causal claim that “Taking pasteurized bacteria can improve people’s insulin sensitivity and total cholesterol?” Why or why not? (Apply the three criteria for causality.)
4. Look at the sample size of this study; do you think it’s a problem? Why or why not? Which of the four big validities does the sample size best fit into? 
5. Do the results above support the claim that probiotics like A. muciniphila can treat obesity? Why or why not? (Is internal validity relevant here? What about construct validity?) Can you conclude that probiotics can help you lose weight? Why or why not?
6. The New York Times article critically failed to mention that the researchers who conducted this study have a patent on the probiotic treatment and even started a company based on it. What do you think about this potential conflict of interest? 
7. Do you think it matters that this study’s somewhat promising findings were only with dead bacteria, whereas probiotics are generally defined as being live bacteria? Why or why not? (Which of the four big validities is being asked about in this question, if any?)
   
   

You might agree that the probiotic study described by this journalist has a few serious problems. For an example of probiotic research done right, check out this NPR news story on a current clinical trial examining if a specific probiotic could eliminate specific GI problems in children on antibiotics (courtesy of Dr. Jess Hartnett’s statistics blog).

Of head horns and cell phones

Did the study collect the right data to conclude that "screen time found to cause 'head horns'?"  Photo Credit: Nature Scientific Reports

Maybe you heard the story last week about how cell phones are making horns grow on people's heads. Take a minute to watch this  short video of the morning crew from ABC news discussing it. The link is captioned, "Screen time found to cause head horns" In this video, you'll notice that the journalist announces, "New research shows that constantly looking down at your screen could give you so-called 'head horns'" .

And here's another video that recycles the same material along with extra commentary under the title, "Tech neck causing skull horns"

There's so much to unpack about this story. If you read beyond the first journalists' coverage, you'll find some great analyses of this study and the journalism around it. Turns out, the study itself had problems, and the journalistic coverage didn't help. 

Take a moment to read one of these two critiques of the "head horns" moment: 

Smartphones aren't making millenials grow horns. Here's how to spot a bad study. or Hold the phone: Are you really going to grow a horn from slouching over your phone? 

Use the information in the two critical stories to research the following questions.

First, there are issues related to the "journal to journalism" cycle (Chapter 1), 

a) The original journalist changed the name of the protuberances to "horns." Every journalist thereafter copied this nickname in their own reporting.  What were they originally called? Why do you think the journalist changed their name?

b) In Chapter 1, you learn that journalists don't always pick the most important studies to summarize. Why do you think journalists picked this particular story? Do you think the study is important? 

Second, there are problems with the study itself, which many journalists did not detect--at least at first.

c) One criticism with the study concerns its sample. What are the criticisms raised about this study's sample of 1200 x-rays?

d) Another criticism has to do with the researcher's potential conflicts of interest (CoIs). What were these, and why are CoIs a potential problem in scientific research and publication? 

e) Another criticism has to do with the proposed link between cell phone use and the "horns."  In order to establish a link between having these protuberances and screen time, the scientists would have had to measure both variables and compute the correlation. Did the researchers do this? 

f) Finally, many of the news headlines used causal verbs ("Screen time found to cause head horns"). Even the scientists overclaimed here when they were interviewed by the press. What kind of study would a researcher have to do in order to establish such a causal link? 

g) Are there any additional problems you discovered?

The news cycle around the head horns study illustrates the benefit of cross-reading and triangulating. If you come across a journalist's story that's just too weird to be true, read about the same topic on other sources, or on snopes.com. 

 

Which brand of car is the most reliable?

How might advertisers operationalize the concept of "reliability" in ways favorable to their own brand?  Credit: BNFWork/Shutterstock

You may have seen these ads on TV: Real people stand in a warehouse and are asked which cars are "most reliable." A Honda driver might predict, "Hondas are the most reliable." But then the cover is whisked off in a flourish: The most reliable car is actually the Chevy! In one recent ad (viewable here), the narrator reports: 

"Based on a recent nationwide survey, Chevy is more reliable than Toyota, Honda, and Ford."

Should statements like these convince you? The website Jalopnik publishes journalism about cars. In this column, a journalist walks readers through how they might think critically about  reliability claims in the Chevy commercials. For example, the journalist writes that the reliability claim was based on a survey.

...the survey was sent in 2018 to owners of 2015 model year cars, and the reporting of the “repairs in the past 12 months” relates to those 2015 cars in their third year of service. Chevy reiterated that “Independent statisticians reviewed the materials and concluded with 95% confidence that Chevy’s percentage of no parts repaired or replaced is better than Toyota, Honda, Ford or 23 other brands.”

The journalist adds:

Anyone who has studied research and statistics will tell you that it’s not difficult to form the conclusions you want to based on your use of what is called an “operational definition.” For this particular study, Ipsos is operationally defining reliability as the repairs a car has had within a 12 month period once the car has reached three years old.

a) How did Chevy operationalize "reliability" in their study? 

b) What are some alternative ways to operationalize a car brand's "reliability?"

c) Can you think of downsides to the operationalization that Chevy used? 

The journalist pointed out that, other than the operationalization of reliability, the survey's number of respondents might matter. To wit:

Setting aside the fact that self-reported surveys can be flawed, fewer than 49,000 respondents completed them out of almost 840,000. What Chevrolet did not provide is a breakdown of those respondents by brand, because if we were looking at the percentage of repairs made by various brand owners this could make a difference.

For example, if only 1,000 of those 49,000 respondents were Honda owners and 100 of them reported repairs within that 12 month period, that means that Honda would have a 10 percent repair rate. But if 15,000 of the respondents were Chevrolet owners and 1,200 of them reported repairs, that is an 8 percent repair rate. Yet this distribution was not provided so it’s difficult to ascertain exactly how those percentages compare within each sample size of brand owners.

d) What do you think of the journalist's reasoning, above? 

In another argument, the journalist compares the self-report survey used by Chevy with two other sources of reliability data, Consumer Reports and J.D. Power: 

...in a 2018 ranking of brand reliability, Consumer Reports ranked both Honda and Toyota well above Chevrolet, [which] occupied the lower part of the list along with several other domestic brands.

But like Chevrolet’s survey, this data is also self-reported based on the ownership experiences of Consumer Reports subscribers.

I also reached out to J.D. Power, as they are a respected reference when it comes to these kinds of “quality” rankings and asked them to comment on the methodology behind Chevrolet’s study. They declined. (Recall that Chevy has touted J.D. Power awards in this very same series of ads.)

e) Why might the journalist extol Consumer Reports or J.D. Power over Chevy's Ipsos survey? What might these organizations doing differently--if anything--in their measurement of reliability? 

f) Reflect on how the availability heuristic (the pop-up principle), cherry picking, bias blind spots, or the present-present bias (all from Chapter 2) might lead individuals to make faulty conclusions about which car brands are most reliable.

 

Thanks to Marianne Lloyd for sharing this idea!