Everyday Research Methods

When questionable survey methods reach the Supreme Court

According to the report, the Court may have been misled by the quality and true findings of survey on the habits of gun owners. 
Photo: B Christopher/Alamy Stock Photo

Polling research can play a critical role in shaping public opinion and public policy, so it's crucial that it be conducted well. Polling (survey) research is increasingly used in court cases, too. Because survey research can have an important impact, it's important to scrutinize a survey's research methods.

An example from the United States is a landmark gun-rights case (known as the Bruen case) from 2022, one of many pro-gun rulings in recent years.  In the Bruen case, the Court ruled that New York State could not outlaw concealed-carry handguns. In their written opinions, several of the justices cited polling research that supposedly found that  guns are "commonly" used for self-defense. 

The full story can be found here, and it was covered in the podcast, The Daily, on June 20, 2024 (available here).

According to reporting by New York Times journalists, the U.S. Supreme Court's decision in the Bruen case depended, in part, on the results of expert research: a survey of gun owners conducted by Georgetown business professor William English. English's survey data was used to argue that many gun owners use guns for self-defense. The Court accepted the research as expert testimony and allowed it to influence their reasoning.  However, when journalists started looking into the survey and the researcher who conducted it, they found instances of questionable scientific practice.

First, the researcher who conducted the survey has not made himself available to journalists to answer questions about his research and he also did not respond to requests that he testify in court about his work on the survey. This reticence to answer questions about one's research goes against scientific norms of transparency, or as Merton called it, communality. As the textbook says, "Scientists should transparently and freely share the results of their work with other scientists and the public", p. 15. From the journalist:

Government lawyers tried to talk with [Dr.English] in December, after his research surfaced in a Firearms Policy Coalition lawsuit seeking to overturn Washington State’s ban on assault weapons. But Dr. English did not respond to emails, phone calls or a certified letter, and a process server bearing a subpoena could not find him at the university or his home, where someone inside “looked at the door, then turned away,” according to court records.

Faced with efforts to compel his testimony, the pro-gun plaintiffs in [a different] Washington case chose to drop all references to his research.

Second, the Dr. English misled people about his impartiality as a scientist. Specifially, the journalists learned that the surveys he conducted were funded by pro-gun legal groups connected to the National Rifle Association. Failing to disclose one's funding is a potential conflict of interest and violates Merton's norm of disinterestedness, which says "scientists strive to discover the truth, whatever it is; they are not swayed by conviction, idealism, politics, or profit," (p. 15). From the story:

Dr. English seems at first glance to be an impartial researcher interested in data-driven insights. He has said his “scholarly arc” focuses on good public policy, and his lack of apparent ties to the gun lobby has lent credibility to his work.

But Dr. English’s interest in firearms is more than academic: He has received tens of thousands of dollars as a paid expert for gun rights advocates, and his survey work, which he says was part of a book project, originated as research for a National Rifle Association-backed lawsuit....

At least one of Dr. English's papers on the gun survey in question (which is published here) does not disclose this funding. 

Third, the journalists reported that Dr English's research methods probably led to misleading results and interpretations of the data. Specifically, the journalists reported:

... the wording of some questions could elicit answers overstating defensive gun use, and that he cherry-picked pro-gun responses.

An example comes from the self-defense question on the survey. In reporting his results, English asserted that 31% of his survey respondents said "yes" when asked, "Have you ever defended yourself or your property with a firearm, even if it was not fired or displayed?" But in that presentation, English left out the full text of the question. The Times report highlighted in blue the omitted portions:

Many policymakers recognize that a large number of people participate in shooting sports but question how often guns are used for self-defense. Have you ever defended yourself or your property with a firearm, even if it was not fired or displayed? Please do not include military service, police work, or work as a security guard.

In Chapter 6, you can read that this is a potential example of a leading question. Gun owners might be more motivated to answer "yes" to it, given its lead-in.

Here's another example of leading question wording (with a leading portion highlighted in blue), coupled with English's misleading presentation of results:

Some have argued that few gun owners actually want or use guns that are commonly classified as ‘assault weapons.’ Have you ever owned an AR-15 or similarly styled rifle? You can include any rifles of this style that have been modified or moved to be compliant with local law. Answering this will help us establish how popular these types of firearms are.

Elsewhere in his findings, Dr. English quotes about 30 gun owners attesting to the usefulness of large-capacity magazines for self-defense — but only two reported actually firing a gun, both at animals. He did not quote any of the hundreds of negative responses from gun owners, including: “There is no reason to have a magazine like that unless you’re fighting in a war.”

 

The journalist concludes that while English's research was only one piece of the full argument in Bruen and other pro-gun rulings, "...his credentials and findings carried weight. At least five briefs cited his research."

Reflection Questions:

a) What's the difference between writing a leading survey question and writing a misleading summary of one's research? 

b) What do you think might have happened in the court cases if Dr. English had been more up-front about who funded his research? 

c) What do you think might have happened in the court cases if briefs about Dr. English's presentation of his research had included the full texts of the questions he used?  

 

 

What's your love language?

Are there really only five love languages? Does everyone have only one primary love language? Do couples who match--have the same love language--experience better relationships? Photo: Olesia Bilkei/Shutterstock

It's nearly Valentine's Day, so it's a good time to feature this new story on so-called "love languages." According to a new review article by relationships researchers, it looks like we should move love languages into the category of "things that make research psychologists cringe" (along with the Myers Briggs Type Inventory, learning styles, and others).

Many journalists have covered an empirical review article recently published in the journal, Current Directions in Psychological Science. Here's an example of coverage from website Medical Express:

Even if you don't know your love language, you've probably heard of the concept. The theory's pervasiveness in pop culture has only increased in the 30-odd years since Baptist pastor Gary Chapman published his book "The Five Love Languages: The Secret to Love That Lasts." [...]

"We were very skeptical about the love languages idea, so we decided to review the existing studies on it," says Emily Impett, a professor in the UTM department of psychology who collaborated with UTM graduate student Gideon Park and York University Assistant Professor Amy Muise. "None of the 10 studies supported Chapman's claims."

Chapman's original book argues that there are five love languages: physical touch, words of affirmation, acts of service, quality time and gifts. He also argues that each person has a primary preference for one of these. Furthermore, he argues in his book that when two partners have the same love language (for example, both of them prefer physical touch) that their relationship will be better.

The journalist explains that Impett and her team addressed three weaknesses in Chapman's theory. The first element they investigated was whether people only have one primary love language. While Chapman's q"uiz" does identify one primary love language for each respondent, in fact, because of the way it is set up, Chapman's test forces people to compare each type to the others, so people must choose one. In contrast, Impett and her colleagues wrote that when people are provided a Likert scale and asked to rate each love language individually, most people report multiple preferences of showing and expressing love.

A second critique of Chapman's original thinking was that it was based on a restricted sample (mostly White, religious, middle-class Americans). In other cultural contexts, there may be different ways of expressing love. 

But let's focus on their third , and most important, claim:

Most importantly, Impett and her team found no scientific evidence for Chapman's central contention that people who choose partners that speak their love language, or learn to speak it, will have more successful relationships.

The researchers reviewed the literature and found

...no scientific evidence for Chapman's central contention that people who choose partners that speak their love language, or learn to speak it, will have more successful relationships. "There's no support for this matching effect," says Impett. "People are basically happier in relationships when they receive any of these expressions of love." 

This conclusion is a great example of the theory-data cycle, so let's walk through it. I've included Figure 1 for your reference.

a) The theory-data cycle starts with a theory. What is Chapman's theory about love languages? Remember that a theory is a set of general statements about how variables are related to each other, and Chapman's theory had several ideas. What are his main theoretical ideas?

The next step in the cycle is to select a particular research question. Let's work with the research question about matching love languages. The question might be, "do couples with matching love languages have better relationships?"

Then we design a study to test the research question. In one study that Impett and her team reviewed, researchers recruited a sample of couples and measured each person's love language. They divided the couples into two types: those who had the same love language and those that did not. They measured each couple's relationship satisfaction.

b) The next step in the theory-data cycle is to state a hypothesis for the study--what should the results of this specific study be if the theory is true? State specifically what the results should be--you can even sketch a graph of the predicted outcome.

c) The next step is to evaluate the data in light of the theory. What did the study actually find? Do these results support the theory, or not? That is, which arrow in the Figure above (the right one or the left one) will be be applying in this case?

In addition to providing data to counter the "love language" theory, Impett and her colleagues proposed an alternative metaphor: a "balanced diet." Specifically, they reported that most people appreciate many kinds of ways of showing love--not just one--and that we can all benefit from receiving some of each type, no matter what type it is. 

Challenge question:

d) Come up with a study that might test the "balanced diet" view of love expressions. What variables would you measure? How should the variables be related to each other if this theory is true?

 

 

 

What's your love language?

As it's nearly Valentine's Day, it's a good time to feature this new story on so-called "love languages." According to a new review by relationships researchers, it looks like we should lump this idea, along with the Myers Briggs Type Inventory and learning styles, into the category of "things that make research psychologists cringe."

Many journalists have covered an empirical review article recently published in the journal, Current Directions in Psychological Science. by psychologists. This post features an story for the website medical express.

Even if you don't know your love language, you've probably heard of the concept. The theory's pervasiveness in pop culture has only increased in the 30-odd years since Baptist pastor Gary Chapman published his book "The Five Love Languages: The Secret to Love That Lasts." [...]

"We were very skeptical about the love languages idea, so we decided to review the existing studies on it," says Emily Impett, a professor in the UTM department of psychology who collaborated with UTM graduate student Gideon Park and York University Assistant Professor Amy Muise. "None of the 10 studies supported Chapman's claims."

Chapman's book argues that there are five love languages: physical touch, words of affirmation, acts of service, quality time and gifts, and that each person has a primary preference for one of them. Furthermore, he argues in his book that when two partners have the same love language (for example, both of them prefer physical touch) that their relationship will be better.

The journalist explains that Impett and her team tested each of Chapman's ideas. One was that people only have one primary love language. In fact, Chapman's unscientific "quiz" forces people to choose one. When people are provided a likert scale for each type, they report multiple preferences of showing and expressing love. Second, Chapman's original thinking was based on a restricted sample (mostly White, religious, middle-class Americans), and in other cultural contexts, there may be different ways of expressing love. 

But let's focus on this claim:

Most importantly, Impett and her team found no scientific evidence for Chapman's central contention that people who choose partners that speak their love language, or learn to speak it, will have more successful relationships. "There's no support for this matching effect," says Impett. "People are basically happier in relationships when they receive any of these expressions of love." 

The researchers reviewed the literature and found

no scientific evidence for Chapman's central contention that people who choose partners that speak their love language, or learn to speak it, will have more successful relationships. "There's no support for this matching effect," says Impett. "People are basically happier in relationships when they receive any of these expressions of love." 

This conclusion is a great example of the theory-data cycle. Let's walk through it. I've included Figure 1 for your reference.

a) The theory-data cycle starts with a theory. What is Chapman's theory about love languages? Remember that a theory is a series of general statements about how variables are related to each other. 

b) A researcher can then design a study to test the theory. In one study that Impett and her team reviewed, researchers recruited a sample of couples and measured each person's love language, and also measured their relationship satisfaction. Then they divided the couples into two types: those who had the same love language and those that did not. 

c) The next step in the theory-data cycle is to state a hypothesis for the study--what should the results of this specific study be if the theory is true? Sketch a graph of the predicted outcome, or state specifically what the results should be. 

d) The next step is to evaluate the results in light of the theory. What did the study actually find? Do these results support the theory, or not? 

 

 

Increasing vaccination rates in evidence-based ways

One of the evidence-based techniques was to "say because," as these ads illustrate.
Images used with permission from the Government of Jersey

Jersey is a self-governing island in the English Channel off the northwest coast of France. Its population is around 108,000. After the COVID-19 vaccine became widely available, the island's government started campaigns to encourage people to get their shots.  Rather than making up communication strategies that just seemed right, they decided to design strategies based on social psychological evidence.  This article describes the studies they used and how they applied them. Let's take a look. 

Strategy: Say because

One strategy mentioned in the article involves the word "because."

The study:

Ellen Langer demonstrated the persuasive power of the word “because” in her famous studies in the 1970s. People waiting in line to use a photocopier were asked “Excuse me, I have five pages. May I use the copier?” 60 percent agreed. But when the request was accompanied with a reason — “May I use the Xerox machine because I’m in a rush?”—94 percent complied.

[...] even when the word because was followed with a completely meaningless reason — such as “May I use the Xerox machine, because I have to make copies?” just as many people said yes. The study wonderfully demonstrates the motivational influence of the word “because” and its continually reinforced association with the good rationales that typically follow it.

The application:

But just because people pay more attention to the word “because” doesn’t mean that what comes after is less important. Hence our two-step “because-then-reason” strategy. First, we prominently included the word “because” in our vaccination outreach campaigns. Then we followed the word “because” with a personalized reason meaningful to that particular messenger (and their audience).

The images in this post show examples of how the posters highlighted the word because. 

Questions about this strategy:

a)  Before basing an intervention on an empirical study, we'd better be sure that study was experimental (rather than correlational). Why?

b) What were the variables in Langer's study, and what were the levels?  Do you think that Langer's study was experimental or correlational? What are the clues? 

 

Strategy: Build ownership

The study:

A 2020 mega-study conducted by several dozen researchers led by Wharton School professor Katy Milkman demonstrated how a simple change to a text message increased uptake of flu shots. Reminder messages that included the words “your vaccine is reserved for you” boosted rates by as much as 11 percent compared to standard messages such as “protect yourself by getting a flu shot.”

The application:

Milkman’s study provided the inspiration for us to adopt a similar approach in the emails and texts we sent on behalf of the Government of Jersey to citizens whose turn it was to receive their Covid-19 vaccine. The words “your vaccine” were designed to give readers that sense of personal ownership.

c) What were the variables in Milkman's study? What were the levels of each variable?

d) Do you think Milkman's study was experimental or correlational? Explain your answer

e) Why do you think the journalist described this as a "mega-study"? Follow the link to test your hunch. 

f)  The journalist seems to mention only two levels of the independent variable in Milkman's study (the words “your vaccine is reserved for you” versus “protect yourself by getting a flu shot.”). But in the Milkman et al.'s actual study, there were many more levels. Check the link to the study to find out how many other conditions the researchers tested. 

 

Note to Instructors: Figure 1 of Milkman's study shows the effect sizes and CIs for all of the different interventions they tried. It's a really nice way to introduce these statistical concepts. 

 

Claim: "Drinking any amount of alcohol causes damage to the brain"

Photo credit: Dangubic/Deposit Photos

Here's a link to CNN's coverage of a study on alcohol use and gray matter in the brain. The headline reads, "Drinking any amount of alcohol causes damage to the brain, study finds." Let's consider how well the study supports the causal claim the journalist (or the editor) attached to it.  Here's one of CNN's introductory statements: 

In an observational study, which has not yet been peer-reviewed, researchers from the University of Oxford studied the relationship between the self-reported alcohol intake of some 25,000 people in the UK, and their brain scans.

a) The term, "observational study" in this context is synonymous with "correlational study." What are the key variables in the study? What does the quote above tell you about whether the variables were manipulated or measured? 

Here's the next paragraph in the CNN summary: 

The researchers noted that drinking had an effect on the brain's gray matter -- regions in the brain that make up "important bits where information is processed," according to lead author Anya Topiwala, a senior clinical researcher at Oxford.

"The more people drank, the less the volume of their gray matter," Topiwala said via email.

b) The CNN journalist erroneously wrote, "drinking had an effect on the brain's gray matter".  Let's walk through whether this correlational study here can support the journalist's causal claim. The first criterion is covariance, which is about the study's results. Does the study show covariance?

c) The second criterion is temporal precedence. The journalist doesn't provide information about whether the drinking behavior (variable A) was measured first in time, before brain scans measured people's gray matter (variable B). What are the two potential directions this relationship might go?

d) The third criterion is internal validity. There could be some third variable, C, which is associated with both A and B. See if you can come up with a potential C variable that could reasonably correlate with both amount of drinking and reduction in gray matter. 

You might have noticed that the opening to the CNN summary mentions that the study "has not yet been peer-reviewed". 

e) What happens during the peer review process? Do you think journalists should summarize, for the general public, studies that have not yet been peer-reviewed?

f) The empirical report has not yet been published, but you can read more about the study where it is posted on the preprint site, medarxiv. What are some pros and cons of putting one's research on "arxiv" sites like these? Consider issues such as openness of science, peer review, and public interest. 

 

Thanks again, Stephen Chew, for sharing the example!

Conspiracy reasoning vs. Scientific reasoning (the case of QAnon)

Conspiracy theories, unlike scientific theories, are unfalsifiable. Credit: buecax/Depositphotos

Early in the research methods course you learn about the theory-data cycle (Chapter 1). You also learn about the benefits of scientific reasoning over informal, everyday reasoning such as intuition, experience, or authority (Chapter 2). The QAnon conspiracy theory is a good example of several principles in these two chapters.

The quotes from this post come from this excellent story in The Atlantic, by journalist Adrienne LaFrance. 

In case you have never heard of the QAnon conspiracy theory, it started from a series of posts by an anonymous author named Q, who posted occasionally on anonymous "image boards" such as 4chan. Here is a quick recap by LaFrance:

In its broadest contours, the QAnon belief system looks something like this: Q is an intelligence or military insider with proof that corrupt world leaders are secretly torturing children all over the world; the malefactors are embedded in the deep state; Donald Trump is working tirelessly to thwart them. (“These people need to ALL be ELIMINATED,” Q wrote in one post.) The eventual destruction of the global cabal is imminent, Q prophesies, but can be accomplished only with the support of patriots who search for meaning in Q’s clues. To believe Q requires rejecting mainstream institutions, ignoring government officials, battling apostates, and despising the press. 

Part I. Falsifiability
Let's practice describing one way that conspiracy thinking is different from scientific thinking. One of the defining features of scientific thinking is that scientific claims are falsifiable. That is, scientists make claims/hypotheses that can be falsified  (disproven) by collecting and analyzing evidence. 

Most conspiracy theories a cannot be falsified, in part because they are stated in squishy, ambiguous terms, and in part because they are so multifaceted. As LaFrance writes,

In the face of inconvenient facts, [QAnon theory] has the ambiguity and adaptability to sustain a movement of this kind over time. For QAnon, every contradiction can be explained away; no form of argument can prevail against it.

Here is an example of some QAnon reasoning: 

At one point, Harger [a believer she interviewed] told me I should look into what happened to John F. Kennedy Jr.—who died in 1999, when his airplane crashed into the Atlantic Ocean off Martha’s Vineyard—suggesting that Hillary Clinton had had him assassinated. (Alternatively, a contingent of QAnon believers say that JFK Jr. faked his death and that he’s a behind-the-scenes Trump supporter, and possibly even Q himself. Some anticipate his dramatic public return so that he can serve as Trump’s running mate in 2020.) When I asked Harger whether there’s any evidence to support the assassination claim, he flipped my question around: “Is there any evidence not to?”

a) Why is Harger's theory (above) about JFK Jr.'s assassination unfalsifiable?

b) And, How does the theory's unfalsifiable nature make it more likely that Harger can believe in QAnon? 

c) Think like a scientist now. We can write a falsifiable statement (i.e., one that could be demonstrated by evidence to be wrong) related to JFK Jr. For example, to falsify the theory that "J.F.K. Jr. will become Trump's running mate in 2020" we could predict this statement:

"J.F.K. Jr.'s name will appear on the ballot with Trump in the 2020 Presidential election." (This is falsifiable because if someone else's name appears on the ballot, then this theory is wrong.)

Now see if you can come up with a falsifiable statement derived from the theory (above) that J.F.K. Jr.'s plane crash death was an assassination. Your statement can take the form, If J.F.K. Jr. was assassinated, then we should see __________. 

Part II. Accepting evidence
One feature of QAnon theory that makes it impossible to falsify is that its believers do not accept any evidence from legitimate sources. That is, whereas scientists use independently verifiable data to support or falsify theories, QAnon believers do not accept the legitimacy of such data. Here's LaFrance again:

Q frequently rails against legitimate sources of information as fake. Shock and Harger [the two believers] rely on information they encounter on Facebook rather than news outlets run by journalists. They don’t read the local paper or watch any of the major television networks. “You can’t watch the news,” Shock said. “Your news channel ain’t gonna tell us [expletive].” Harger says he likes One America News Network. Not so long ago, he used to watch CNN, and couldn’t get enough of Wolf Blitzer. “We were glued to that; we always have been,” he said. “Until this man, Trump, really opened our eyes to what’s happening. And Q. Q is telling us beforehand the stuff that’s going to happen.” I asked Harger and Shock for examples of predictions that had come true. They could not provide specifics and instead encouraged me to do the research myself. When I asked them how they explained the events Q had predicted that never happened, such as Clinton’s arrest, they said that deception is part of Q’s plan. Shock added, “I think there were more things that were predicted that did happen.” Her tone was gentle rather than indignant.

d) Why does QAnon's rejection of legitimate sources of information help its followers maintain their beliefs? 

As an aside, in the passage above, pay attention to the believer's statement that "deception is part of Q's plan." This feature virtually ensures that the theory remains unfalsifiable: Either believers will point to evidence that supports the theory, or, when faced with evidence that does not support the theory, they will claim that "deception is part of Q's plan." The theory is supported no matter what. 

Part III.  Confirmation bias.
In Chapter 2, you learn about quirks of human reasoning that make our intuition inferior to the scientific method. One of these quirks is the confirmation bias, the tendency of people to notice and remember evidence that confirms their beliefs, and to ignore or discount evidence that disconfirms them. 

e) How might confirmation bias be at play in the passage above, about events that Q predicted but that never happened? 

Confirmation bias is probably more prevalent when situations and theories are ambiguous. Ambiguity is clearly at play in QAnon, because Q usually makes general (rather than specific) threats. LaFrance writes,

—this is a very welcoming belief system, warm in its tolerance for contradiction—and are also what makes it possible for a practical man like Hayes [who has a youtube channel "explaining" Q's theories] to play the role that he does. QAnon is complex and confusing. People from all over the internet seek guidance from someone who seems levelheaded. 

f) Explain how confirmation bias, rather than scientific thinking (which strives to objectively evaluate evidence both for and against a theory), can explain the following reasoning about who Q himself is:

Many QAnon adherents see significance in Trump tweets containing words that begin with the letter Q. Recent world events have rewarded them amply. “I am a great friend and admirer of the Queen & the United Kingdom,” Trump began one tweet on March 29. The day before, he had tweeted this: “I am giving consideration to a QUARANTINE.” The Q crowd seized on both tweets, arguing that if you ignore most of the letters in the messages, you’ll find a confession from Trump: “I am … Q.”

Coda.
QAnon might seem like a harmless academic example of conspiracy theories. And, as LaFrance suggests, some people follow it just for fun (it might have even started out as a parody). However, as LaFrance argues, it has actually incited violence in the past. For example, in 2017, one believer walked into a Washington, D.C. pizza shop with a rifle because he had believed QAnon message boards that children were being held hostage in the shop's basement (the building doesn't have one). (He was wrong: "'The intel on this wasn’t 100 percent,' Welch told The New York Times after his arrest.")  LaFrance also reports, "In 2018, Reddit banned QAnon groups from its platform for inciting violence."  In the United States, several congressional candidates have promoted QAnon's theories. Unfalsifiable conspiracy thinking might give our leaders a (false) sense of confidence, but it certainly doesn't belong in government discourse.  

If QAnon supporters call for violence, revolt, or collective action and people do not think critically about what they say, this fake conspiracy theory will become a real threat to our democracy. 

Here's a great story of a person who used to believe in QAnon and has subsequently changed his mind. 

 

 

 

 

 

 

 

Racial bias in psychological science journals

A recent study found that in psychology, the authors, participants, topics, and editors of empirical journal articles are mostly White.
Photo credit: Africa Studio/Shutterstock

Is science fair?

You learned in Chapter 2 that empirical journals are the primary place where psychological scientists make their research public. Each empirical journal has an editorial board-- a group of scientists who have power to select the peer reviewers for each article and make the final decision about publication. 

Each step of the way, the stated goals of the publication process are to be fair, to focus on the quality of the science, and to publish the best science possible. But is the process fair?  In light of the renewed energy around anti-racism, a timely new study sheds light on racial diversity in the psychology publication process. The researchers asked, Is race an important variable in the scientific publication process?

The team coded all empirical journal articles that appeared in the most prestigious journals in psychology (over 26,000 articles). There were two journals each from social, cognitive, and developmental psychology. The authors measured:

• how often race was studied as a subject in these articles
• the race of the participants in the studies
• the proportion of the authors of scientific articles were White vs. Non-white
• the proportion of the editors of the journals themselves were White vs. Non-white

Here are some of the results of this published study of published studies, as summarized by the Association for Psychological Science's trade publication:

The results suggest that there is a hierarchy in psychology research that determines who and what gets published. “We are not saying anyone has bad intentions,” said Roberts [one of the study's authors]. “It’s a systemic problem across many fields, and psychological science is no exception.”

First, across the past five decades, articles in psychological journals that highlight race have been rare, and although developmental and social psychology journals have published a growing number of these articles, they have remained virtually nonexistent in cognitive psychology.

Second, most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism.

Third, many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color.

Let's parse these claims. 

Questions

a) What kind of a claim is it (frequency, association, or cause) to state that "across the past five decades, articles in psychological journals that highlight race have been rare"

[And what's "rare"? The published report says that "only 5% of publications highlighted race" (p. 4). This 5% figure is an average across cognitive journals (less than 1%) and social (5%) and developmental journals (8%)]

b) Let's take the second claim, "most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism"

The quoted text above provides two claims in one sentence. What is the frequency claim in the quoted text? What is the association claim in the quoted text? What are the two variables in the association claim?

c) Now for the third claim, "many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color."

As with b), the quoted test provides two claims in one sentence, a frequency claim and an association claim. What are these? 

d) In Chapter 1 of the 4th Edition (available Fall 2020), you will read about Merton's norms of science--a set of values that the community of scientists usually endorses and strives  to follow. Here they are, with short descriptions (for a full review, see Chapter 1 or this summary). 

Universalism: Anyone can do science as long as they follow the accepted criteria for science, which are established in advance, are objective, and are accessible to all.
Communality: Share! The process of science, including data, methods, and products, should be free and open for all to use. 
Disinterestedness: The quality of the science should be the only thing that matters, not the prestige of the person or place it was conducted. 
Organized Skepticism: Verify everything. Critical peer review matters, every study should be replicated, and criticism of one's evidence is expected and welcome. 

Roberts and his coauthors argued that psychology is too White. Which of Merton's norms does this result seem to violate, and why? 

Suggested answers:

a) This is a frequency claim--a statement about the frequency of the variable, "highlighting race" across 26,000 journal articles.

b) The frequency claim is about the proportion of journal editors who are White vs. a Person of Color (POC). The association claim is about a link between two variables: whether a journal has a White editor or not, and whether that journal published articles highlighting race or not.

c)  The frequency claim is about the proportion of article authors who are White vs. POC. The association claim is about a link between two variables: whether the article was written by a White author or not, and whether that article included POC as participants or not.

d) Open to discussion!

Here is a summary of this work.

Evidence-based tips for well-being during a pandemic

Dr. Santos provides evidence-based ways to enhance well-being during this pandemic. Credit: Yale University

How can people enhance their wellbeing during the Coronovirus pandemic? 

Dr. Laurie Santos (pictured) has been meeting with several news outlets, sharing evidence-based tips from her psychology of well-being class at Yale University. A version of the course is also offered on Coursera.

In the Twitter-embedded video here, produced by the World Economic Forum, Dr. Santos covers five tips for coping during the pandemic. In this space, I'm going to quote moments from the video, and you can practice classifying the claims and describing how the studies would have been conducted. The first three of these tips are also covered in this story from the World Economic Forum.

Tip 1: Keep social, even when socially distancing

Dr. Santos states in her segment that "Happy people spend more time with other people in general. They also prioritize time with their friends and family members"

a) What kind of claim is it to say that "happy people spend more time with other people in general"--frequency, association, or cause?  What are the variables in this claim? 

b) What kind of study was likely conducted to support this claim--correlational or experimental? 

c) Sketch a scatterplot with well-labeled axes that could depict this result.

Tip 2: Help others

To explain this tip, Dr. Santos mentions, "The research shows that doing random acts of kindness...can be incredibly powerful." For example, happy people are giving money to charity and volunteering in their communities. 

d) What kind of claim is it to say that "doing random acts of kindness...can make you happy"--frequency, association, or cause?  What are the variables in this claim? 

e) What kind of study would need to be conducted to support this claim--correlational or experimental? 

f) Can you imagine how you  might manipulate the variable, "doing random acts of kindness?"

Tip 3: Move--even if you can't leave your house

In this section, she says, "Exercise and moving your body is really correlated with improved well-being"

g) The verb, "correlated" gives this one away: It's an association claim. Dr. Santos specifies that they are "really correlated," which seems to suggest a strong effect size. What factors do we consider when we're trying to decide if a correlation (an effect size) is strong versus weak?

Tip 4: Savor everyday pleasures

"Research suggests that happy people tend to be more mindful. They tend to savor the present moment; they tend to be in the present moment"  Here Dr. Santos also strongly recommends taking deep breaths to calm the vagus nerve and connect to the present. 

h) How might you design an experiment to test the hypothesis that deep breathing can improve people's well-being during quarantine? Whom might you recruit for such a study? How would you operationalize each of the two variables? What results would you expect to find? 

Tip 5: Think about how you want to look back on this time

Finally, she shares one of my favorite tips for emotional regulation, psychological distancing: "Psychological distancing is the idea that if we think about ourselves and how we will have dealt with this crisis in the future...Think about how you want to tell your grandkids you lived through COVID-19, you don't want to be like, 'I panicked-bought and hoarded toilet paper..no, you want the story to be positive...'"

Experimental research has supported the causal claim that self-distancing--seeing yourself as though you are a fly on the wall, or even imagining looking back on yourself from the future--is associated with improved emotional regulation. It helps us take a cooler, less impulsive view of our emotions and process them instead of ruminating. For example, self-distancing helps people manage feelings of anger in constructive ways (here's a nice summary). 

i) Review the term translational research in your textbook (Ch 1) and explain how we might conduct translational research on psychological distancing during the pandemic. What is the basic research it is based on? What is the real-world context it's being translated to? And what kind of study could translate--that is, bridge the gap--between the two? 

 

 

 

 

 

 

Is "sugar high" a thing?

 

Sugar doesn't make kids hyperactive; at least not according to a null effect study and a meta-analysis. Photo: Alamy

Here's something "everybody says":  Kids will become hyperactive if they eat too much sugar. This folk belief is as much a part of North American cultural wisdom as learning styles and the catharsis hypothesis. But this column in the New York Times explains that, like learning styles and catharsis, the idea of a "sugar high" is also a myth. 

 

The physician who wrote the column traced the modern idea of sugar causing hyperactivity to 1975. But then the author describes a randomized clinical trial on this question. Here's the description: 

An extraordinarily rigorous study settled the question in 1994. Writing in the New England Journal of Medicine, a group of scientists tested normal preschoolers and children whose parents described them as being sensitive to sugar. Neither the parents, the children nor the research staff knew which of the children were getting sugary foods and which were getting a diet sweetened with aspartame and other artificial sweeteners. Urine was tested to verify compliance with the diets. Nine different measures of cognitive and behavioral performance were assessed, with measurements taken at five-second intervals.

The original study was, indeed, ambitious. The authors provided all food to the homes of the children in the study so they could completely control what sweeteners were being used. The study took place over nine weeks in a repeated-measures design--the diets were based on sucrose (sugar) one week, aspartame another week, and saccharine during another week. The researchers measured the children's cognitive performance and mood throughout the study.

The study concluded that sugar does not affect children’s behavior or cognitive function. An editorial that accompanied the study put a fine point on that conclusion, stating “there is no evidence that sugar alone can turn a child with normal attention into a hyperactive child.” 

Questions

a) What were the independent and dependent variables in the study described here? What are the levels of the IV?

b) The IV was manipulated as within subjects, using a repeated-measures design. Describe why counterbalancing would be important here, and how you could counterbalance in this study. Which of the four big validities does counterbalancing support?

c) What questions could you ask about the other three validities for this study? 

d) This study found a null effect (no impact of sucrose on children's behavior). There were only 48 children in the diet study, and this might be considered a low number--there may not have been enough kids in the study to detect a small effect of sucrose. However, the repeated measures design means this study can make a stronger case for the conclusion that sugar doesn't affect kids' behavior. Review the information in Chapter 11 to explain why the repeated measures design helps support the conclusion. 

The author of the piece mentions this, too:

One year later, an analysis that gathered data from all published studies on the subject reached the same conclusion.

e) This sentence is describing a meta-analysis. In your own words, describe what steps they would have taken to conduct this meta-analysis.

f) Why is a meta-analysis stronger evidence for a null effect of sugar on hyperactivity? 

g) Given this information, what would you say to a relative or friend who believes that sugar causes hyperactivity in their own child? 

 

Of head horns and cell phones

Did the study collect the right data to conclude that "screen time found to cause 'head horns'?"  Photo Credit: Nature Scientific Reports

Maybe you heard the story last week about how cell phones are making horns grow on people's heads. Take a minute to watch this  short video of the morning crew from ABC news discussing it. The link is captioned, "Screen time found to cause head horns" In this video, you'll notice that the journalist announces, "New research shows that constantly looking down at your screen could give you so-called 'head horns'" .

And here's another video that recycles the same material along with extra commentary under the title, "Tech neck causing skull horns"

There's so much to unpack about this story. If you read beyond the first journalists' coverage, you'll find some great analyses of this study and the journalism around it. Turns out, the study itself had problems, and the journalistic coverage didn't help. 

Take a moment to read one of these two critiques of the "head horns" moment: 

Smartphones aren't making millenials grow horns. Here's how to spot a bad study. or Hold the phone: Are you really going to grow a horn from slouching over your phone? 

Use the information in the two critical stories to research the following questions.

First, there are issues related to the "journal to journalism" cycle (Chapter 1), 

a) The original journalist changed the name of the protuberances to "horns." Every journalist thereafter copied this nickname in their own reporting.  What were they originally called? Why do you think the journalist changed their name?

b) In Chapter 1, you learn that journalists don't always pick the most important studies to summarize. Why do you think journalists picked this particular story? Do you think the study is important? 

Second, there are problems with the study itself, which many journalists did not detect--at least at first.

c) One criticism with the study concerns its sample. What are the criticisms raised about this study's sample of 1200 x-rays?

d) Another criticism has to do with the researcher's potential conflicts of interest (CoIs). What were these, and why are CoIs a potential problem in scientific research and publication? 

e) Another criticism has to do with the proposed link between cell phone use and the "horns."  In order to establish a link between having these protuberances and screen time, the scientists would have had to measure both variables and compute the correlation. Did the researchers do this? 

f) Finally, many of the news headlines used causal verbs ("Screen time found to cause head horns"). Even the scientists overclaimed here when they were interviewed by the press. What kind of study would a researcher have to do in order to establish such a causal link? 

g) Are there any additional problems you discovered?

The news cycle around the head horns study illustrates the benefit of cross-reading and triangulating. If you come across a journalist's story that's just too weird to be true, read about the same topic on other sources, or on snopes.com.