Everyday Research Methods

Interrogating Buzzfeed: Does Instagram really have "huge effects" on our psychology?

Do the studies really support cause and effect? And are those effects "huge"? Photo: Julia_Tim/Deposit Photos

This Buzzfeed headline is eyecatching and dramatic: Instagram Has Some Pretty Huge Effects On Our Psychology. Here's Everything We Know So Far.

I see two important issues to interrogate in this headline. First, it's a causal claim ("Instagram has some...effects on our psychology"). Second, it's making an argument about effect size ("pretty huge effects"). Let's see if they pan out. 

The article focuses on body image and mental health. Here's an example:

The research into how Instagram affects our psychology is relatively new, but there is a growing body of evidence to say that it has a profound effect on the way we perceive ourselves and the world around us.

Does the research support such a causal claim? Here's one study that the journalist summarized: 

Earlier this year, the UK's Royal Society for Public Health (RSPH) and the Young Health Movement published a report that looked at the impact of different social media platforms on mental health.

The report was based on a survey of over 1,400 British 14-to-24-year-olds that asked about the positive and negative implications of each social media platform for health and wellbeing.Instagram scored well on its ability for self-expression, but it was linked with some of the worst scores for sleep loss, body image concerns, and FOMO.

It's not completely clear from this summary what questions were included on this survey. However, a survey is unlikely to have manipulated exposure to Instagram, so it can't support the claim that "Instagram has implications for how people perceive themselves and the world around them."

What about the effect size of the associations? The journalist didn't report the data from the study, so we can't evaluate whether or not the effect was really "huge" or "profound." We'd have to search out the original paper for that. 

How about this other study, summarized later in the article: 

In a study published last year, Cohen and her co-authors found that it wasn’t so much social media use itself that impacted how people perceived their bodies but rather how they were using it.

Following body- and appearance-focused accounts on Instagram — as opposed to neutral accounts dedicated to things such as travel — was associated with idealizing thin bodies, obsessive surveillance of one’s own body, and striving for thinness.

a) Four variables are listed in the study above (Following body-....) What are they?

b) Were these variables measured or manipulated, do you think?  (Check your hunches here, by reading the abstract of the Cohen study.)

c) Why can't this study support a claim such as "following body-focused accounts impacts how people perceive their bodies"? Apply the three criteria for causation. 

d) Does the journalist provide any indication of the effect size of this association? If not, what data might have helped you decide if the link was "huge"?

Let's read a bit further in the journalist's article to see if we can find some experimental evidence. How about this one:

However, Cohen is hopeful that the positive body movements emerging on the platform will have far-reaching benefits for how women reflect on their own physiques.

She has recently submitted a paper for publication demonstrating that among 195 young women exposed to “body-positive” images, mood, body satisfaction, and body appreciation were all improved significantly.

The wording in the example above suggests that there was a manipulated variable, because it reads "among 195 young women exposed to 'body-positive' images'". When I read that, I picture that some people were exposed to these images, and some were not. 

e) Imagine you were going to conduct a conceptual replication of this experiment. What was the independent variable? What would its levels be? 

f) What would the dependent variable(s) be? (Hint: The journalist mentions three).

g Sketch a little bar graph of the results of your experiment--picking one of the DVs to graph. In your view, how far apart would the means and error bars have to be for you to be convinced that the effect is "huge"?

h) Would this experiment have the potential to support a causal claim? What confounds would you need to control for? 

Suggested answers to selected questions

c) Covariance: Yes--apparently this questionnaire study found a link between following more body-focused accounts and idealizing thin bodies more. 

Temporal precedence: Nope--this was a questionnaire study that measured both variables at the same time. We can't know if the IG accounts came first, or if people who idealize thin bodies more are more likely to want to follow body-focused accounts.

Internal validity: Also nope. In this correlational study we can't rule out every possible third variable. For example, maybe people with a history of eating disorders are both more likely to follow body-focused accounts and are more likely to idealize thin bodies.

d) No correlation was reported here--too bad! It would be nice if the journalist had given us some benchmarks indicating how much more likely people were to idealize thin bodies....perhaps a percentage change or a correlation coefficient, and a 95% CI for that change or coefficient. 

 

Thanks to UD student Emily Hogan for suggesting this article for the blog!

 

 

Don't believe it?

Macadamia nuts are in the same evolutionary family as peaches. (or are they?) Credit: gresey/Deposit photos

People's tendency to believe, share, and act upon disinformation (fake news) is a serious threat to civic engagement. Psychologists are studying what makes people more or less likely to believe false claims. Here are some excerpts from a brief article published in Buzzfeed, which summarized a few studies about people's willingness to believe what they read.

"What we find in our research really robustly is that we can push around people's feelings of truth using things like photographs and audio quality, and even the ease of pronouncing someone's name," said [researcher] Newman.

Here's an example of a study on how perceptions of "truthiness" can be influenced:

In 2012 Newman co-authored another paper with studies that found photographs can inflate the sense of "truthiness" that people get from reading a statement.

The sort of photographs that the researchers used for this research were "non-probative", meaning that they didn't actually contain any information about the claims being made.

One study ...found that if participants were given a trivia statement such as "Macadamia nuts are in the same evolutionary family as peaches", they were more likely to believe the statement was true if a photograph of macadamias simply appeared alongside the sentence [compared to no photo].

Bonus: Stephen Colbert (who coined the term "truthiness") also summarizes it in this clip. 

Questions about the first example:

a) What are the two variables in the macadamia nuts study? Name each variable, its apparent levels, and indicate if it is measured or manipulated.

b) Do you think the "photo appearance" variable was manipulated as between-groups or within-groups? Explain your reasoning.

c) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

d) Ask three questions about the statistical validity of the macadamia nut study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

 Here's a second example:

A study published in March found that people were less likely to believe an academic researcher on the radio if the audio quality of the recording was poor or glitchy.

The study involved 97 participants, who were instructed to listen to recordings of different pieces of science news and assess the quality of the research and the researcher who was reporting it.

Half of the audio recordings were edited to sound as though they were recorded on a bad phone line. The listeners who heard these recordings rated the credibility of the researcher and the studies they were discussing as significantly less favourable than recordings with higher audio quality.

Questions about the second example:

e) What are the two variables in the audio quality study? Name each variable, and indicate if it is measured or manipulated.

f) Do you think the "audio quality" variable was manipulated as between-groups or within-groups? Explain your reasoning.

g) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

h) Ask three questions about the statistical validity of the audio quality study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

i) Now consider the internal validity of the audio quality study: Why would random assignment be important in this study? 

Selected answers (use these answers as models to help answer the audio quality questions, if needed)

a) One variable is the appearance of a photo; its levels are "photo or no photo." It is manipulated.
A second variable is "level of belief in the trivia statement" or "rated truthiness." Its levels probably range from "low belief" to "high belief". It is measured. 

b) For a single fact, the presence of a photo would have to be manipulated as between groups--people would have to see the fact either with or without the photo. If they were exposed to both levels of the variable (a within-groups design), people might wonder why they are being asked to rate the same fact two times.  However, in the study itself, participants were exposed to multiple facts, some of which were accompanied by a photo and some of which were not....so that makes the variable within-subjects when you consider all of the facts together. 

c) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

d) Ask three questions about the statistical validity of the macadamia nut study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

For effect size, you could ask questions such as, "how far apart were the truthiness ratings?" "How big is the difference between the photo and no-photo group?" or "What is the d? (or other effect size)"
For precision you could ask, "what is the 95% CI of the effect size estimate?"
For replication you could ask, "has this effect been replicated in other studies? What do those other studies say?"

I would like to thank my student Emily Hogan for sharing this example and Buzzfeed story with me!

 

Meta-analyses: Strength training treats depression and anxiety

Strength training seems to improve both depression and anxiety, according to two meta-analyses. Photo credit: romanchazov27/Deposit Photos

Many of us start the new year resolved to improve our exercise habits. Exercise, of course, is great for our physical health, and we also know that aerobic exercise (i.e., "cardio") can improve mood. The benefits of aerobic exercise on depression and mood have been well-documented by several studies (here's an example of one meta-analysis). 

A recent article in the Washington Post describes the benefits of strength, or resistance, training for depression. The piece provides a good example of how a journalist covers meta-analysis for a popular audience.

The author himself suffers from depression, and starts his article by explaining his own workout strategy:

It’s strength training — the muscle-building, body-toning and sometimes grunt-inducing workout — that has helped transform my chronic mental illness into better mental health. In fact, once I began a regular regimen of resistance training, the positive shift in my mood, my confidence and my self-image was so pronounced that I switched careers.

Of course, while an anecdote about one person's experience can be compelling, it doesn't carry the same weight as a systematic observation. It's possible that the author was a biased observer of his own improvement. Therefore, we need experiments that randomly assign people to different exercise groups to test whether resistance training has a causal effect on depression and mood. 

It turns out that several researchers have, in fact, studied the effect of strength training, enough to collect and combine:  

[Dr. Jacob] Meyer was among the authors of a recent meta-analysis of the link between resistance training and depressive symptoms ... The study, published in the June 2018 issue of JAMA Psychiatry, analyzed 33 clinical trials for the effects of resistance exercise on depression. Results showed that resistance exercise “significantly reduced depressive symptoms” among research participants.

a) In your own words, what does a meta-analysis do?

b) In the meta-analysis described in the paragraph above, there were 33 studies, all of which were experiments. Given the context, what was the independent variable in each of these 33 studies? What was the dependent variable? 

c) Reread the description above, which states that strength training "significantly reduced depressive symptoms". This phrasing does not indicate the meta-analytic effect size (that is, the effect size they obtained by averaging the effect sizes of all 33 studies). To see that effect size, you'd have to turn to the original empirical article, here.  The article is open-access and the forest plot you will find there depicts  the mean, or overall, effect. What is the overall effect size? 

The meta-analysis above showed that strength training can help improve depression. What about a related mental health condition, anxiety? Turns out there's a separate meta-analysis on that:

Resistance exercise doesn’t affect depression alone; research shows that it can reduce symptoms of anxiety, too. For a 2017 study published in the journal Sports Medicine, Gordon and his colleagues conducted another meta-analysis of 16 studies involving 992 total participants; it concluded that resistance training “significantly improves anxiety symptoms among both healthy participants and participants with a physical or mental illness.”

d) How many studies were in this meta-analysis on strength training and anxiety? Estimate how many participants were in each study.

e) What would have been the independent variable in each of these studies? What would have been the dependent variable?

f) Once again, the journalist explains that resistance training "significantly improves" anxiety symptoms, but we don't hear about the effect size. The original empirical article  is behind a paywall. You might be able to access it from your own library. If you can't find it, I'll tell you that the mean effect size (d) of the studies was 0.31, 95% CI [0.17, 0.44]. What information do you need to evaluate this effect size? Is the effect strong or weak? 

As a bonus, the researchers are exploring why strength training might improve depression and anxiety. That is, they are exploring potential mediators of the relationship between strength training and mental health. Here is the first of two mediator explanations, as summarized by the journalist:

One notable finding was that participants showed an improvement in their mood regardless of whether they grew physically stronger from the exercises. ... “Perhaps it might be the sense of accomplishment and confidence that comes from exercising, rather than the achievement of actual strength gains, to explain why study subjects felt better,” Meyer says.

And the second:

Resistance training, like other exercise, induces the release of a protein called BDNF, or brain-derived neurotrophic factor, into the hippocampus region of the brain. Among other functions, the hippocampus is responsible for mood regulation, and in people who are depressed, it shrinks up to 25 percent of its normal volume. The release of BDNF triggers the growth of new brain cells in the hippocampus, restoring it to its full size and improving communication between cells.

“That’s the physiological theory behind why you feel so good after working out,” says Jason Sawyer, an assistant professor of wellness and exercise science at Rhode Island College in Providence.... 

g) Choose one of the mediator descriptions above and sketch a figure depicting the mediator argument. Use a box for each variable, and arrange the boxes in a triangle shape, using the figures at the end of Chapter 9 as a model. 

 

Selected answers

c) The overall (meta-analytic) effect size was d = 0.66 95% CI [0.48, 0.83]. Therefore, strength training groups scored about 2/3 of a standard deviation better than control groups, on average. To evaluate how strong this effect is, it might be helpful to compare the effect of strength training to some other treatments for depression, including cognitive-behavioral therapy or medication. (A 2013 meta-analysis found that CBT's effect on depression is at least d = 0.53, 95% CI [0.43, 0.62].)

d) There were 16 studies and a total of 992 participants, so that's about 165 participants per study. 

e) Each study would have used exercise condition as an IV and anxiety level as a DV. 

f) As you can see, the overall (meta-analytic) effect size of resistance training on anxiety was not as strong as the overall effect size of resistance training on depression (the two CIs do not overlap). This pattern suggests that strength training does more for depression than for anxiety. 

g) Here's how you might sketch one of these mediation proposals:

[strength training condition] -->  [sense of accomplishment] --> [improved mood]

Does "virtue signaling" change minds?

If a lot of us send virtue signals, it can communicate to others that norms are changing. Credit: SEEphotos7171/Shutterstock"

Among people fighting for racial justice, "virtue signaling" may sometimes be derided as empty action. "Virtue signals" are public messages of support for a social justice cause. They could take the form of a yard sign, a t-shirt, or a social media post. Critics sometimes argue that such messages are easy and cheap. It's easy for a company to put a message on their home page that supports Black Lives Matter. It's not hard to wear a t-shirt that supports a social justice cause. But are these expressions of support all talk and no action?

An article in Time magazine argues that virtue signaling can be more powerful than we think. The authors, psychological scientists Jamil Zaki and Mina Cikara, integrate several studies to make their argument. Here's part of their introduction: 

It’s critical to question people whose most radical act is temporarily changing their profile picture. It’s critical they follow their words with action; the time for talk has passed. But focusing solely on virtue signalers obscures their most important role. Like radio waves, the signals they send are received by someone, and those receivers matter, too.

Zaki and Cikara argue that virtue signaling is an important way of influencing others. If a lot of people change their profile pictures to support BLM, for example, more people will start to believe that this movement is popular and they may adopt it, too:

Opinions can work this way: when a particular viewpoint gets a lot of attention, people assume it’s popular, and shift towards it. We receive signals, and are changed by them.

What studies support their argument? Here's one that the authors summarize: 

As social norms shift, individuals shift with them: adopting popular opinions and behaviors, and dropping ones that fall out of style. Norms are especially powerful as they gain steam. In one set of studies, Gregg Sparkman and Greg Walton presented diners in a cafeteria [either] with evidence that 30% of the U.S. population was vegetarian, or that 30% of the population was newly vegetarian. People who learned about this second, “dynamic social norm” were twice as likely to order a meatless lunch themselves. They saw not just where the crowd was, but where it was going, and didn’t want to be left behind.

Here are some questions about the study summarized above:

a) Was this an experiment or a correlational study? How do you know? (If it's an experiment, what kind of study is it?)

b) What were the key variables in the study? Were they manipulated or measured? 

c) Sketch a small graph of the results being described. Would a bar graph or a scatterplot make more sense?

Here's another study Zaki and Cikara introduce:

Signaling and conformity are not inherently positive. In 2016, the U.S. elected a president whose campaign was jammed with racist talking points. Prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants). 

d) This summary doesn't really help you figure out how the study was designed. But here's a chance to think carefully: What kind of study would we need to conduct in order to test the claim that "prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants)."?  There are three variables you would need to include in your study. What are they? 

 

Suggested answers to selected questions

a) This was probably a posttest-only experiment, because it sounds like they manipulated the message about vegetarianism and measured what foods people took in the cafeteria. Here is a link to the original empirical article. 

d) The key dependent variable in this study is prejudice levels, so that's the first variable. Then, in order to establish that "prejudice increased", you'd need to measure prejudice both before and after the election, so that's another variable. Finally, in order to show that it increased specifically toward groups he targeted, you'd need to measure different types of groups (Muslims, immigrants, as well as other potential groups that Trump did not deride in his campaign). You can review the Crandall article here. Do their results really support what Zaki and Cikara are saying about it? (Pay attention to the difference between people's actual prejudice and the normativity of prejudice.) 

 

 

 

 

Racial bias in psychological science journals

A recent study found that in psychology, the authors, participants, topics, and editors of empirical journal articles are mostly White.
Photo credit: Africa Studio/Shutterstock

Is science fair?

You learned in Chapter 2 that empirical journals are the primary place where psychological scientists make their research public. Each empirical journal has an editorial board-- a group of scientists who have power to select the peer reviewers for each article and make the final decision about publication. 

Each step of the way, the stated goals of the publication process are to be fair, to focus on the quality of the science, and to publish the best science possible. But is the process fair?  In light of the renewed energy around anti-racism, a timely new study sheds light on racial diversity in the psychology publication process. The researchers asked, Is race an important variable in the scientific publication process?

The team coded all empirical journal articles that appeared in the most prestigious journals in psychology (over 26,000 articles). There were two journals each from social, cognitive, and developmental psychology. The authors measured:

• how often race was studied as a subject in these articles
• the race of the participants in the studies
• the proportion of the authors of scientific articles were White vs. Non-white
• the proportion of the editors of the journals themselves were White vs. Non-white

Here are some of the results of this published study of published studies, as summarized by the Association for Psychological Science's trade publication:

The results suggest that there is a hierarchy in psychology research that determines who and what gets published. “We are not saying anyone has bad intentions,” said Roberts [one of the study's authors]. “It’s a systemic problem across many fields, and psychological science is no exception.”

First, across the past five decades, articles in psychological journals that highlight race have been rare, and although developmental and social psychology journals have published a growing number of these articles, they have remained virtually nonexistent in cognitive psychology.

Second, most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism.

Third, many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color.

Let's parse these claims. 

Questions

a) What kind of a claim is it (frequency, association, or cause) to state that "across the past five decades, articles in psychological journals that highlight race have been rare"

[And what's "rare"? The published report says that "only 5% of publications highlighted race" (p. 4). This 5% figure is an average across cognitive journals (less than 1%) and social (5%) and developmental journals (8%)]

b) Let's take the second claim, "most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism"

The quoted text above provides two claims in one sentence. What is the frequency claim in the quoted text? What is the association claim in the quoted text? What are the two variables in the association claim?

c) Now for the third claim, "many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color."

As with b), the quoted test provides two claims in one sentence, a frequency claim and an association claim. What are these? 

d) In Chapter 1 of the 4th Edition (available Fall 2020), you will read about Merton's norms of science--a set of values that the community of scientists usually endorses and strives  to follow. Here they are, with short descriptions (for a full review, see Chapter 1 or this summary). 

Universalism: Anyone can do science as long as they follow the accepted criteria for science, which are established in advance, are objective, and are accessible to all.
Communality: Share! The process of science, including data, methods, and products, should be free and open for all to use. 
Disinterestedness: The quality of the science should be the only thing that matters, not the prestige of the person or place it was conducted. 
Organized Skepticism: Verify everything. Critical peer review matters, every study should be replicated, and criticism of one's evidence is expected and welcome. 

Roberts and his coauthors argued that psychology is too White. Which of Merton's norms does this result seem to violate, and why? 

Suggested answers:

a) This is a frequency claim--a statement about the frequency of the variable, "highlighting race" across 26,000 journal articles.

b) The frequency claim is about the proportion of journal editors who are White vs. a Person of Color (POC). The association claim is about a link between two variables: whether a journal has a White editor or not, and whether that journal published articles highlighting race or not.

c)  The frequency claim is about the proportion of article authors who are White vs. POC. The association claim is about a link between two variables: whether the article was written by a White author or not, and whether that article included POC as participants or not.

d) Open to discussion!

Here is a summary of this work.

Testing a COVID drug: Design flaws and ethical lapses

Photo credit: Alissa Eckert, MS; Dan Higgins, MAMS/CDC

Here's a good example of a bad study. It's related to the COVID pandemic. In this story by NPR  journalist Vanessa Romo, you can read about a doctor in Texas who claims to be conducting research on the drug hydroxychloroquine, using COVID patients in a nursing home as participants. The doctor's research includes both design problems and ethical problems.

According to the journalist, 

...a doctor at a Texas nursing home started giving the anti-malaria drug hydroxychloroquine to dozens of elderly patients diagnosed with COVID-19 and tracking the outcomes in what he's calling an "observational study".

According to the doctor conducting this study, 

"It's actually going well. People are getting better," Armstrong told NPR, adding that after just a handful of days, some of the 39 patients on the medication are showing signs of improvement.

Armstrong admits it is difficult to quantify how much of his elderly patients' improvement is due to the malaria drug or how they would have fared without it. Nor can he explain why other patients are not responding to the tablet doses, though he notes many are only halfway through the five-day cycle.

Design questions

a) Why can't the doctor decide "how they would have fared without it"? Your answer should discuss the study's need for a comparison group. 

b) The doctor's study includes one group--all of the patients in the "study" are taking the drug. What kind of comparison group will be needed to conduct a true clinical trial?

c) If you're as far as Chapter 11, you'll recognize the design of this doctor's study. What is it?

Here's another quote from the article:

[The doctor] said he started by screening those he believed would benefit most and added more people each day. He monitored their blood oxygen saturation, temperatures and how well they were breathing. "The people who [I decided to put] on it were getting sicker but were not so sick that they had to go the hospital," Armstrong explained.

d) What design problems, if any, to you see with the doctor selecting only this type of patient for being in his "observational study?" And, what might have happened if he had included more severely ill patients for his study? (How might the results change?)

Ethical questions

According to federal guidelines and the ethical principles of the Belmont Report, when a person claims to be doing "research," they should obtain ethical approval from an IRB. Ethical approval would require such procedures as obtaining informed consent (respect for persons), minimizing harm (beneficence), and selecting participants fairly (justice). Read these next quotes from the story:

He acknowledged that some families were not aware their relatives were put on the drug, saying that "for the most part," he consulted with each nursing home resident prior to giving them on the tablets.

While the "overwhelming majority of them are awake and alert and can actually have a conversation," Armstrong said some suffer from middle stages of dementia. In some cases, he did not discuss prescribing the tablets with anyone at all before doing so. He said it is common for physicians to prescribe new medications to patients without explicit consent from the patient or family members. "It's not required," he said.

e) Review the sections of Chapter 4 on "respect for persons," "beneficence," and "justice," and apply details from these concepts to this example. 

f) Do you think study design flaws such as the ones you've noted in a), b) and c) should be considered ethical violations, too? Why or why not? 

Safety questions

A potential issue with this doctor's decision concerns the safety of the drug hydroxychloroquine: According to the journalist:

The Food and Drug Administration has not approved the drug for the treatment of COVID-19. 
and
[the drug] is known to have serious negative health impacts. That is why so many in the medical community worry about prescribing it without such proof.

g) The safety of the drug is relevant to the ethical principle of beneficence. Can you explain why?

Finally...

Another wider ethical issue (perhaps outside the scope of a research methods course) concerns how the doctor obtained the drug, which has been in short supply:

[The doctor] began making calls to track down a source for the medicine, which is in short supply.That's when his political connections proved useful.

Armstrong, who is a prominent GOP activist, called Republican Lt. Gov. Dan Patrick. He says Patrick reached out to Texas state Sen. Bryan Hughes, also a Republican, who knew someone on the board of the New Jersey-based company Amneal Pharmaceuticals. The company, which makes and distributes the drug, has donated more than a million tablets nationwide, including to the statesof Texas and Louisiana.Two days later, Armstrong had received more than enough medication to begin giving it to patients. 

h) Do you think it is reasonable for a doctor to use political connections to obtain medications that are in restricted supply? Why or why not? 

Suggested answers to selected questions

a) and b) and c) Because the doctor's "study" has only a group of patients who are taking the drug, he doesn't have a comparison group. In Chapter 11 this is called a one-group, pretest-posttest design, otherwise known as the Really Bad Experiment. The design is flawed. If patients improve on the drug, the doctor will not know whether their improvement was caused by the drug itself, or simply because their bodies were healing or because they were getting excellent nursing care, or even some other reason. 

The doctor's behavior in this COVID study reminds me of the Halstead radical mastectomy, which for decades was the dominant treatment for breast cancer in the U.S.. Surgeons at the time were so convinced that this treatment was the best one for breast cancer that they never bothered to compare it to another treatment--they never tested its efficacy. (When a randomized control study was finally conducted in the 1970s, it found that the surgery was no more effective than other, less invasive treatments.)

d) By selecting only relatively less sick patients, the doctor is stacking the deck in favor of finding evidence that the drug is effective. Many of the patients are likely to get better, so the doctor may conclude that the drug has worked. But of course, without a comparison group of equally healthy people who are not getting the drug, we cannot know if the patients would have recovered anyway.

e) f) g) The lack of informed consent--especially for vulnerable patients who have Alzheimer's--is a clear violation of the respect for persons principle. And the use of a risky treatment, one that is known to be dangerous to people with heart conditions, is likely a violation of the beneficence principle. I would argue that when a researcher conducts a flawed study like this, it also a violation of beneficence--the study puts participants at risk, while the benefit of the potential information gained from the research is negligible. We might also argue that the study violates justice: by studying only people at a nursing home, these patients bear the burden of being research participation when anyone can benefit from this research (assuming, of course, that the research were actually well-designed).

Many thanks to Dr. Stephen Chew for pointing me to this example!

Post script: Looking for more examples of bad studies, as described by science writers? Here are two:

Here's a critique of the many problems with another early published study on hydroxychloroquine.

Here's critique of studies supposedly investigating the death rate of COVID-19. 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers. 

 

Dog people live longer

The study found an association between dog ownership and longevity. Which third variables were controlled for? Which third variables might still be responsible for the relationship? Photo: Shutterstock

I can't resist blogging about research on dogs. Here's a story on NPR that summarized several studies that have tested the correlation between longevity and dog ownership. The journalist introduces the topic this way: 

Dogs and humans that interact with one another get a jolt of oxytocin, the so-called "cuddle hormone." And, if you get to look at dogs and hug them every day, you just might live longer than people who don't have to clean animal hair off their clothes, according to a pair of studies out this month.

Here is a summary of one of the studies:

[researchers] looked over a 12-year period at dog owners in Sweden who have cardiovascular disease. They adjusted for the kinds of things we know affect cardiovascular health — age, demographics, socioeconomic status, marital status, number of children at home — and even after adjusting for all of that, they found a benefit of dog ownership [on longevity].

....they followed over 100,000 people and found a 21% reduction in deaths from any cause.

The study above used multiple regression. 

a) What was the dependent (criterion) variable in this study?  What were the predictor variables? 

b) Sketch a little regression table with results that would be consistent with what is mentioned above. Put the criterion variable at the top, and list the predictor variables. Which betas would be significant here? 

c) The researchers controlled for age, demographics, SES, marital status, and children--but regression doesn't control for all third variables. What else could be associated with both dog ownership and longevity? 

d) They mention a "21% reduction in deaths from any cause."  Which of the four big validities is this concerning? 

One of the experts who was interviewed for this story said:

In particular, I found it very convincing and striking that the benefits seems to be larger among individuals who live alone compared to multiperson households. That suggests the companionship of a dog is possibly very important to their heart health.

e) This is a moderator! Identify the variables in this relationship and complete this sentence: ____ moderates the relationship between ____ and _____.

 

In part of the article, the journalist presents quotes from an expert, who seems to equivocate on the causality issue:

"When you look at the big picture and look at all the evidence around dog ownership and cardiovascular health, it's pretty clear the signal is real and likely causal," he says.

In other words, he's convinced getting a dog improves your health — especially for those with heart disease.

The research is not definitive, though, Kazi adds. "They're not randomized trials, the gold standard for what we would do to evaluate a new drug," he says.

f) It is frustrating to read that the expert believes the "signal is real and likely causal" and then admits that "it's not definitive." You can set this expert straight. Why can't this correlational study support causation?  Apply covariance, temporal precedence, and internal validity. When you address internal validity, check out answer c) above!

 

 

Suggested answers

a) The criterion variable is longevity (being alive at the end of the study). The predictor variables are dog ownership, age, demographics, SES, marital status, and children living at home. 

b) The table might look like this:

DV: Longevity (or being alive after 12 years)

IV                            Beta (sig)
Dog ownership         .12*
Age                            -.11
SES                              .13*
Married                        .08*
Children at home        .07*

d) Current health and mobility might be associated with both dog ownership and longevity. After all, you  may need to have a certain level of health and mobility to own a dog, and healthier folks are more likely to be alive at the end of the study. 

e)  Living status (being alone or married) moderates the relationship between dog ownership and longevity.  

f) There is covariance--there is a link between owning a dog and living longer. 

There is temporal precedence-- the study took place over several years and dog ownership was measured first, and death rates later. 

There is not full internal validity. They controlled for age, demographics, SES, marital status, and children living at home, so they can rule those out. But they did not control for everything. One possibility is that current health and mobility might be associated with both--after all, you  may need to be at a certain level of health and mobility to own a dog, and those folks are more likely to be alive at the end of the study. 

 

 

 

Racial stereotypes behind how we explain school shootings

Participants and newspapers were more likely to blame video games when a school shooter was white than when he was black. Photo: Richard Levine / Alamy Stock Photo

What causes mass shootings? Of course, we have no experimental evidence on this question. Correlational evidence (both within the U.S. and evidence that compares the U.S. to other countries) is consistent with the argument that gun availability is a factor. 

However, this post isn't about what really causes school shootings. It's about what people think causes them. And it turns out that the race of a school shooter influences what people believe about the shooter. 

After some school shootings (a subset of mass shootings), people have sometimes looked for answers not in the availability of guns, but in video games. Now, there is actually no evidence linking video games to mass shootings. However, this ABC news story talks about when people are more likely to think that video games are responsible: 

There's little evidence showing a relationship between violent video games and school shootings, but video games are often blamed for overall violence in our communities. A recent study published in the Psychology of Popular Media Culture set out to find if we look for an external explanation for violence by whites more often than we do for African Americans.

"We try harder to make excuses for white perpetrators," said investigator Dr. James Ivory, a professor in the Department of Communications at Virginia Tech, in an interview with ABC News.

The first study gave people a mock news story about a school shooting. When asked what caused the shooting, people were more likely to blame video games if the shooter was white than if the shooter was black.

It turns out that in the original study (linked here;  if it's paywalled you may be able to access via your university library), participants read the mock news story and were randomly assigned to one of two photos of the perpetrator--a mugshot showing either a Black or a White man. 

a)  There were two variables in this study. Name each one. For each, was it manipulated or measured?

b) What kind of study is this, correlational or experimental?

c) Sketch a graph of the results of this study. What will you put on the y-axis? 

d) In the study above, what role does random assignment play? Why did the researchers randomly assign people to see either a Black or White perpetrator?  

 

Here is how ABC news described the second study:

The second study looked at thousands of news articles -- video games were mentioned more often in stories of white shooters [compared to black shooters]. 

e) What are the variables in the study above? Name each one; for each, was it manipulated or measured?

f) What kind of study is this, correlational or experimental?

g) Sketch a graph of the results of this study.

h) In the study above, the researchers could not randomly assign news article authors to write about a black vs. a white perpetrator. How does that change the impact of this study? 

i) Which of the two studies do you think provides the strongest evidence that race influences people's willingness to think video games are a cause of the shooting?

 

Suggested answers

a) One variable is race of the perpetrator (levels are Black or White), this was manipulated. The other variable is "likelihood to blame video games". This was measured.

b) This was an experiment because one variable was manipulated. 

d) Random assignment contributes to the study's internal validity. They can be sure that the same kinds of participants were in each group (reading about Black vs. White perpetrators). Therefore, people who play a lot of video games themselves were equally likely to be in the two groups. You might be interested to know that in the original study, the researchers also pretested the Black and White photos they used to be sure that the faces differed on race but not on perceived attractiveness or dangerousness. This pretesting also contributed to the experiment's internal validity.

e) The first variable was race of the perpetrator in the article (Black or White), which was measured. The second variable was whether or not the article mentioned video games (yes or no).

f) This was a correlational study because all variables were measured. 

h) Because this study used naturally-occurring crimes, the researchers may have an internal validity problem. It is possible that the Black and White perpetrated shootings differed not just on race but also on number of victims, age of victims, geography, or some other variable. This introduces an alternative explanation for the result--perhaps people blamed video games for White perpetrated shootings because they had more victims, had older victims, and so on.

i) You might have selected the first study as the strongest because it was an experiment and allows us to make a causal inference. Or you might have selected the second study because it was conducted on "real world" stimuli such as news stories--therefore it appears to have good ecological validity or external validity. 

Thanks to Marianne Lloyd for this idea!

Which Ice Cream Flavor Are You?

Many flavors, many personalities: How could we test whether an ice cream personality test is reliable and valid? Photo credit: Okanmetin/Getty Images

At the height of summer, we all start considering the same, timeless question: Can your personality can be accurately summarized by an ice cream flavor? This Buzzfeed poll asks you to answer a series of summer-activity-related questions and then delivers a verdict. Go take it yourself! I'll wait. 

The first time I took the poll, I answered the questions seriously and was told that I am characterized as "Coffee," along with this description: 

You’re a go-getter, a hard worker who doesn’t pause to reflect on what you have accomplished in the past because you are so focused on what you want to accomplish in the future. You seek to inspire the people around you and help to propel them to greater achievements. At the same time, you are approachable and kind rather than snotty and entitled. It’s basically the best of all worlds.

I sure felt good reading that personality description! The second time I took the quiz, I clicked on questions randomly haphazardly, just to see what other feedback there might be. This time I got "Chocolate", with this description:  

Mmm, you’ve got a truly rich and decadent side that you aren’t afraid to share. You make people feel good — really good. You’ve got a knack for excelling in social situations and know just how to be a crowd-pleaser. Keep doing your thing and living that luscious life.

Of course, nobody is going to mistake a Buzzfeed poll for a legitimate personality measure. But it does provide a fun way to consider issues raised in Chapters 5 and 6. Take the quiz yourself and then prepare these questions:

 

1.     What flavor ice cream did you get? What did the results say about you?

 

2.     Do you think this quiz is reliable? What kind of reliability is relevant for this measure? How would you empirically establish the quiz's reliability? Here's a hint: You'd start with a large sample of people and have them take this ice cream quiz, and then..... (what?)

3.  Now think about the measurement validity of this quiz. I'll narrow down the question for you here: Are people who score in the Coffee category actually more likely to be "hardworking," "inspiring," and "approachable," as the description suggests? How would you establish the criterion validity of this quiz? Here's a hint: You'd start with a large sample of people and have them take the ice cream quiz. You'd separate them into people who got the Coffee category and those who did not get Coffee. How should the Coffees be different from the non-coffees?

 

4.     What might we change about the quiz if we wanted to study ice cream preferences in a scientific way? 

 

5.  Read this description on the Barnum effect. Reflect on how the Barnum effect might be at play for people who take this Buzzfeed quiz. 

 

Many thanks for this idea to Josh Maxwell of the University of New Mexico!

 

Suggested answers to selected questions.

2. Test-retest reliability is the most relevant. To establish it, you'd have a large sample of people take the Buzzfeed quiz at one time point and record everyone's category (e.g., Coffee, Chocolate, etc.) Then you'd track down the same group of people again some time later (maybe 4 weeks later) and have them take the same quiz again. People who were scored Coffee at Time 1 should also score Coffee at Time 2, and people who were scored as Chocolate at Time 1 should also score as Chocolate at Time 2. (You'd compute the correlation between these two category assignments using Cohen's Kappa.)

3. To establish this criterion validity, you'd recruit a large sample of people and have them take the quiz, recording those who got Coffee and those who did not. Then you'd need to measure variables such as "hardworking" "inspiring" and "approachable." I'm not sure how to measure "inspiring" or "hardworking," but you could probably measure "approachable" using the Big Five trait of Agreeableness (here's one measure of that). If the Buzzfeed poll is valid, then a simple bar graph (and an independent groups t-test) should show that people who are assigned Coffee should score higher on agreeableness than people who were assigned some other flavor.