Everyday Research Methods

Evidence-based tips for procrastination

What do the studies suggest about why people procrastinate? Photo credit: Timothy Hodgkinson/Shutterstock

The BBC magazine, Science Focus, collected a set of studies on procrastination that might help students at the beginning of a new school year.  The journalist opens with a couple of simple frequency claims. 

[If you procrastinate,] you’re not alone: an estimated 20 per cent of adults (and above 50 per cent of students) regularly procrastinate.

Then the article addresses three topics: 1. Is procrastination a sign of poor time management? 2. Is procrastination unhealthy? 3. How can we stop? Let's take each topic and apply some research methods content as we go.

1. Is procrastination a sign of poor time management?

The journalist quoted psychologist Fuschia Sirois, who explained that contrary to what many people believe, procrastinators aren't more likely to be bad at time management. Instead, she explains, it's about emotion regulation:

“At its core, procrastination is about not being able to manage your moods and emotions. Although many think impulsivity and self-control are the problems – and they do play a factor – underneath is a poor emotional response.”

As Sirois explains, every person faces stressful situations, demanding tasks that trigger brain activity that involves a brain region known amygdala. And it’s the amygdala that processes emotions and signals threats, capable of prompting a ‘fight or flight’ response linked to procrastination.

“Interestingly, people who say they are chronic procrastinators tend to have larger grey matter volume in the amygdala,” says Sirois.

“This means they will also be more sensitive to the potential negative consequences of their actions, leading to more negative emotions and procrastination.”

a) What were the variables in the amygdala study? Was this study correlational or experimental? 

b) Sketch a scatterplot (or bar graph) of the results Dr. Sirois describes here.

 

2. Is procrastination "bad for your health"?

The journalist implies causality with this headline, and with lines like these: 

...procrastination can cause a lot more problems than missed deadlines. Over decades Sirois has examined the impact of chronic procrastinating on a person’s health, her findings worrying at best – and downright terrifying at worst.

Let's see if the evidence can support these causal claims. He quotes Dr. Sirois again, who stated, 

“People who chronically procrastinate – people who make it a habit – have higher levels of stress and a greater number of acute health problems. They are more likely to have headaches or insomnia or digestive issues. And they’re more susceptible to the flu and colds.”

Even more alarming, Sirois has found that procrastination is a factor that can lead to hypertension and cardiovascular disease, with chronic procrastinators more likely to put off healthy behaviour such as exercise.

c) Is Dr. Sirois describing correlational or experimental evidence here? (Explain your answer by specifying the variables she mentions).

d) Why can't the evidence described by Dr. Sirois support the journalist's assertions about "the impact of chronic procrastinating"? Specify the temporal precedence problem and the internal validity problems here (when you discuss internal validity, specify a particular third, or "C" variable that might correlate with both A and B). 

3. How can we stop procrastinating?

In this section, the journalist describes some intervention studies targeting procrastination.

For example, one compelling Psychological Science paper described how downsizing larger metrics of time (think 48 hours instead of 2 days, or 10,950 days instead of 30 years) can make events seem more immediate, prompting people to engage in upcoming tasks.

You can follow the underlined link in the quoted text (above) to see that this paper described several experiments (for example, the abstract states, "we manipulated time metric".

e) Given the journalist's description (and the empirical study's abstract), name the independent variable in this experiment and name its levels.

f) Name the dependent variable in this experiment.

g) Sketch a bar graph of the study's result, as described by the journalist. 

h). Because this study was an experiment, it is more likely to support causation.
Think about something you have to do in the near future. How can you apply the results from the Lewis and Oyserman study in Psychological Science to that task, to prevent yourself from procrastinating on it? 

 

Selected Answers

a) One variable was "saying you are a chronic procrastinator or not" and the other was "volume of grey matter in the amygdala". Both variables must have been measured, so this was a correlational study.  

b) One axis should have "grey matter volume in amygdala" and the other should have "level of chronic procrastination"; the scatterplot should have a positive slope, but you don't have information on how strong the relationship was, so you can't really know how spread out the dots should be. 

c) This is likely correlational evidence. They are measuring (not manipulating) people's procrastination habits. They are measuring (one cannot manipulate) people's acute health problems, susceptibility to the flu, hypertension, and cardiovascular disease. 

d) We can take acute health problems as an example. Even if there is an association between chronic procrastination (A) and acute health problems (B), we still don't know if the procrastination led to the health problems (A to B), or if the health problems lead to more procrastination (B to A). In addition, there could be some outside variable such as life stressors or poverty (C) that is associated with both procrastination (A) and health problems (B). 

e) The independent variable is apparently "time metric" and the levels were hours vs. days, or perhaps days versus years. 

f) The dependent variable seems to be "how immediate events seem to be"

g) The x-axis should have hours and days, and the y-axis should indicate "how immediate events seem to be". The bar for "hours" should be higher than the bar for "days". 
This study seems to suggest that if you have homework due in two days, you might be more likely to get started on it if you say to yourself, "My homework is due in 48 hours." 

 

Estimating how many U.S. adults identify as nonbinary

 

Photo credit: nito103/Deposit Photos

Press outlets have been covering a report by the Williams Institute (based at UCLA), a research organization collecting data on demographic information on sexual orientation and gender identity. The quotes in this post come from this summary from PBS.

The new study has estimated the number of LGBTQ adults in the U.S. who identify as nonbinary. (Nonbinary is a term that is often used to describe people whose gender identities are more than the man/woman category; some nonbinary people use they/them pronouns.)  It's important to estimate how many U.S. adults identify as nonbinary (as well as other sexual- and gender-minority identities) because anti-discrimination laws potentially affect them. PBS reports that:

Nonbinary people are part of a “growing subgroup” of the LGBTQ population, Wilson said, adding there’s been greater visibility for nonbinary people. More and more celebrities, like singers Demi Lovato and Sam Smith, and actors Indya Moore and Amandla Stenberg, have talked about identifying themselves as nonbinary or in related terms.

Here's a portion of the PBS summary: 

According to the study, which focused on people ages 18 to 60, nonbinary people make up about 11 percent of the adult LGBTQ population in the country. The study builds on the work of two prior surveys — one on trans adults, [TransPop] the other on cisgender LGBTQ adults [Generations]— to provide more comprehensive data on nonbinary people, a historically underresearched subgroup.

When the researchers extrapolated out form this subsample, they concluded that

There are about 1.2 million LGBTQ adults in the U.S. who are nonbinary.

The data in this report constitute a frequency claim, because they are estimating the level of one variable (that is, nonbinary identity).

First let's talk about the precision of these estimates. The report from the Williams Institute (here) includes 95% CIs for the estimates. For example, the two estimates from the PBS story were presented this way:

Number of nonbinary LGBTQ adults in the U.S., including transgender and cisgender people N [95% CI]	1,219,000 [1,000,000 - 1,481,000]
% [95% CI] of LGBTQ adult population	11.1 [9.3, 13.2]

a) In your own words, describe what the first CI means. Use this definition and template to help you: The 95% CI is meant to capture the true value in the population 95% of the time. The true value in the population might be as low as ____ or as high as ____. 

b) Sometimes the 95% CI misses the true value in the population. What percentage of the time will the CI miss?

Next lets' talk about external validity (generalizability) and sampling. As you consider these estimates keep in mind that there is a distinction between estimating the number of all U.S. adults who identify as non-binary and the number of U.S. LGBTQ adults who identify as non-binary. These values may be different. 

c) Given what you've read so far, what seems to be the population of interest of this study? 

To know if the sample is generalizable, you need to know how they obtained the sample. The PBS summary doesn't say anything about the sampling method used in this study, and neither does the Williams Institute report. To locate information about the sampling, I had to go to the methods sections of the two studies on which the nonbinary study is based: The TransPop study (which studied a sample of transgender U.S. adults) and the Generations study (which studied a sample of LGBTQ U.S. adults). 

In these methods sections, they say that these two studies (TransPop and Generations) worked with the Gallup organization to recruit nationally representative samples of transgender and LGBTQ adults. Gallup performs a Daily Tracking Survey in which they call a national probability sample of 1000 adults, almost every day of the year. These adults are sampled using random-digit dialing, including both landline and cell phone numbers. Gallup asks a variety of questions about well-being, political attitudes, and economic situations. At the end of each  telephone survey, the interviewer asks, “I have one final question we are asking only for statistical purposes. Do you, personally, identify as lesbian, gay, bisexual, or transgender?” Gallup retained the records for people who said yes to this question over one year, and the Generations and TransPop studies contacted these folks later. At this second contact, researchers checked to be sure each person was eligible for the Generations or TransPop study, and if they were, the researchers sent them a survey by mail or email. To be eligible, people had to identify as sexual minority or transgender, be within certain age ranges, and be able to complete surveys in English. 

After people were recruited into the TransPop and Generations studies in this way, they were asked multiple survey questions, including the target question about whether they identified as nonbinary. 

d) In your own words, explain why the procedure used above is likely to lead to a probability (in other words, a generalizable, or externally-valid) sample of LGBTQ adults in the United States. 

e) Reading the description above, which type of random was used here--random sampling or random assignment? 

f) To reiterate, the study concluded that about 11% of LGBTQ people in the United States identify as nonbinary. What method could we use to estimate the percentage of all people in the United States identify as nonbinary? 

g) What do you predict might happen to the estimate of nonbinary people in the U.S. if they redid this study 20 years in the future? 

 

Suggested Answers

a) The 95% CI is meant to capture the true value in the population 95% of the time. The true value in the population might be as low as 1 million or as high as 1.48 million. Importantly, our CI may have missed the true population mean. We don't know if it missed it or not, but we know the probability that our CI missed it--see question b)!

b) the 95% CI will miss the population mean in 5% of cases.  

More information: If we replicated the study, the new estimate and 95% CI will be a little bit different from the first one. But we'd still expect that the replication estimate of the number of nonbinary adults will be between 1 million and 1.48 million. We'd be surprised if our new estimate was a lot lower or a lot higher than this range. 

c) The population of interest is LGBTQ adults in the United States (who have access to a cell phone or landline)

d) The Gallup sample is generalizable because their Daily Tracking Survey goes out to a random sample U.S. adults. Any LGBTQ folks that are called in that survey are therefore also a random sample, and random samples are generalizable.  

e) They used random sampling. (As a reminder, random assignment is used in experiments, to assign participants to different groups.)  

f) One method could be to ask directly of a random sample of US adults. For example, Gallup could potentially add a nonbinary identity question to their Daily Tracking Survey for one year.

Another method was the one used in the present study. Gallup learned that 3.5% of their Daily Tracking Survey sample answered yes to the LGBTQ question, and then, of that subsample, a certain percentage (about 11%) identified as nonbinary. After adjusting weights, they got a percentage, (maybe around .05%) which they multiplied by the entire U.S. adult population, to produce the estimate that about 1,219,000 adults (or between 1 and 1.48 million) identify as nonbinary. To see some of the statistics I quoted here, visit this page)

g) Answers will vary

Interrogating Buzzfeed: Does Instagram really have "huge effects" on our psychology?

Do the studies really support cause and effect? And are those effects "huge"? Photo: Julia_Tim/Deposit Photos

This Buzzfeed headline is eyecatching and dramatic: Instagram Has Some Pretty Huge Effects On Our Psychology. Here's Everything We Know So Far.

I see two important issues to interrogate in this headline. First, it's a causal claim ("Instagram has some...effects on our psychology"). Second, it's making an argument about effect size ("pretty huge effects"). Let's see if they pan out. 

The article focuses on body image and mental health. Here's an example:

The research into how Instagram affects our psychology is relatively new, but there is a growing body of evidence to say that it has a profound effect on the way we perceive ourselves and the world around us.

Does the research support such a causal claim? Here's one study that the journalist summarized: 

Earlier this year, the UK's Royal Society for Public Health (RSPH) and the Young Health Movement published a report that looked at the impact of different social media platforms on mental health.

The report was based on a survey of over 1,400 British 14-to-24-year-olds that asked about the positive and negative implications of each social media platform for health and wellbeing.Instagram scored well on its ability for self-expression, but it was linked with some of the worst scores for sleep loss, body image concerns, and FOMO.

It's not completely clear from this summary what questions were included on this survey. However, a survey is unlikely to have manipulated exposure to Instagram, so it can't support the claim that "Instagram has implications for how people perceive themselves and the world around them."

What about the effect size of the associations? The journalist didn't report the data from the study, so we can't evaluate whether or not the effect was really "huge" or "profound." We'd have to search out the original paper for that. 

How about this other study, summarized later in the article: 

In a study published last year, Cohen and her co-authors found that it wasn’t so much social media use itself that impacted how people perceived their bodies but rather how they were using it.

Following body- and appearance-focused accounts on Instagram — as opposed to neutral accounts dedicated to things such as travel — was associated with idealizing thin bodies, obsessive surveillance of one’s own body, and striving for thinness.

a) Four variables are listed in the study above (Following body-....) What are they?

b) Were these variables measured or manipulated, do you think?  (Check your hunches here, by reading the abstract of the Cohen study.)

c) Why can't this study support a claim such as "following body-focused accounts impacts how people perceive their bodies"? Apply the three criteria for causation. 

d) Does the journalist provide any indication of the effect size of this association? If not, what data might have helped you decide if the link was "huge"?

Let's read a bit further in the journalist's article to see if we can find some experimental evidence. How about this one:

However, Cohen is hopeful that the positive body movements emerging on the platform will have far-reaching benefits for how women reflect on their own physiques.

She has recently submitted a paper for publication demonstrating that among 195 young women exposed to “body-positive” images, mood, body satisfaction, and body appreciation were all improved significantly.

The wording in the example above suggests that there was a manipulated variable, because it reads "among 195 young women exposed to 'body-positive' images'". When I read that, I picture that some people were exposed to these images, and some were not. 

e) Imagine you were going to conduct a conceptual replication of this experiment. What was the independent variable? What would its levels be? 

f) What would the dependent variable(s) be? (Hint: The journalist mentions three).

g Sketch a little bar graph of the results of your experiment--picking one of the DVs to graph. In your view, how far apart would the means and error bars have to be for you to be convinced that the effect is "huge"?

h) Would this experiment have the potential to support a causal claim? What confounds would you need to control for? 

Suggested answers to selected questions

c) Covariance: Yes--apparently this questionnaire study found a link between following more body-focused accounts and idealizing thin bodies more. 

Temporal precedence: Nope--this was a questionnaire study that measured both variables at the same time. We can't know if the IG accounts came first, or if people who idealize thin bodies more are more likely to want to follow body-focused accounts.

Internal validity: Also nope. In this correlational study we can't rule out every possible third variable. For example, maybe people with a history of eating disorders are both more likely to follow body-focused accounts and are more likely to idealize thin bodies.

d) No correlation was reported here--too bad! It would be nice if the journalist had given us some benchmarks indicating how much more likely people were to idealize thin bodies....perhaps a percentage change or a correlation coefficient, and a 95% CI for that change or coefficient. 

 

Thanks to UD student Emily Hogan for suggesting this article for the blog!

 

 

Don't believe it?

Macadamia nuts are in the same evolutionary family as peaches. (or are they?) Credit: gresey/Deposit photos

People's tendency to believe, share, and act upon disinformation (fake news) is a serious threat to civic engagement. Psychologists are studying what makes people more or less likely to believe false claims. Here are some excerpts from a brief article published in Buzzfeed, which summarized a few studies about people's willingness to believe what they read.

"What we find in our research really robustly is that we can push around people's feelings of truth using things like photographs and audio quality, and even the ease of pronouncing someone's name," said [researcher] Newman.

Here's an example of a study on how perceptions of "truthiness" can be influenced:

In 2012 Newman co-authored another paper with studies that found photographs can inflate the sense of "truthiness" that people get from reading a statement.

The sort of photographs that the researchers used for this research were "non-probative", meaning that they didn't actually contain any information about the claims being made.

One study ...found that if participants were given a trivia statement such as "Macadamia nuts are in the same evolutionary family as peaches", they were more likely to believe the statement was true if a photograph of macadamias simply appeared alongside the sentence [compared to no photo].

Bonus: Stephen Colbert (who coined the term "truthiness") also summarizes it in this clip. 

Questions about the first example:

a) What are the two variables in the macadamia nuts study? Name each variable, its apparent levels, and indicate if it is measured or manipulated.

b) Do you think the "photo appearance" variable was manipulated as between-groups or within-groups? Explain your reasoning.

c) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

d) Ask three questions about the statistical validity of the macadamia nut study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

 Here's a second example:

A study published in March found that people were less likely to believe an academic researcher on the radio if the audio quality of the recording was poor or glitchy.

The study involved 97 participants, who were instructed to listen to recordings of different pieces of science news and assess the quality of the research and the researcher who was reporting it.

Half of the audio recordings were edited to sound as though they were recorded on a bad phone line. The listeners who heard these recordings rated the credibility of the researcher and the studies they were discussing as significantly less favourable than recordings with higher audio quality.

Questions about the second example:

e) What are the two variables in the audio quality study? Name each variable, and indicate if it is measured or manipulated.

f) Do you think the "audio quality" variable was manipulated as between-groups or within-groups? Explain your reasoning.

g) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

h) Ask three questions about the statistical validity of the audio quality study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

i) Now consider the internal validity of the audio quality study: Why would random assignment be important in this study? 

Selected answers (use these answers as models to help answer the audio quality questions, if needed)

a) One variable is the appearance of a photo; its levels are "photo or no photo." It is manipulated.
A second variable is "level of belief in the trivia statement" or "rated truthiness." Its levels probably range from "low belief" to "high belief". It is measured. 

b) For a single fact, the presence of a photo would have to be manipulated as between groups--people would have to see the fact either with or without the photo. If they were exposed to both levels of the variable (a within-groups design), people might wonder why they are being asked to rate the same fact two times.  However, in the study itself, participants were exposed to multiple facts, some of which were accompanied by a photo and some of which were not....so that makes the variable within-subjects when you consider all of the facts together. 

c) Sketch a small bar graph depicting the results of this study, as described by the journalist. 

d) Ask three questions about the statistical validity of the macadamia nut study--your three questions should address effect size, precision, and replication. (Note--the answers to your questions might be present in the original empirical article.)

For effect size, you could ask questions such as, "how far apart were the truthiness ratings?" "How big is the difference between the photo and no-photo group?" or "What is the d? (or other effect size)"
For precision you could ask, "what is the 95% CI of the effect size estimate?"
For replication you could ask, "has this effect been replicated in other studies? What do those other studies say?"

I would like to thank my student Emily Hogan for sharing this example and Buzzfeed story with me!

 

Meta-analyses: Strength training treats depression and anxiety

Strength training seems to improve both depression and anxiety, according to two meta-analyses. Photo credit: romanchazov27/Deposit Photos

Many of us start the new year resolved to improve our exercise habits. Exercise, of course, is great for our physical health, and we also know that aerobic exercise (i.e., "cardio") can improve mood. The benefits of aerobic exercise on depression and mood have been well-documented by several studies (here's an example of one meta-analysis). 

A recent article in the Washington Post describes the benefits of strength, or resistance, training for depression. The piece provides a good example of how a journalist covers meta-analysis for a popular audience.

The author himself suffers from depression, and starts his article by explaining his own workout strategy:

It’s strength training — the muscle-building, body-toning and sometimes grunt-inducing workout — that has helped transform my chronic mental illness into better mental health. In fact, once I began a regular regimen of resistance training, the positive shift in my mood, my confidence and my self-image was so pronounced that I switched careers.

Of course, while an anecdote about one person's experience can be compelling, it doesn't carry the same weight as a systematic observation. It's possible that the author was a biased observer of his own improvement. Therefore, we need experiments that randomly assign people to different exercise groups to test whether resistance training has a causal effect on depression and mood. 

It turns out that several researchers have, in fact, studied the effect of strength training, enough to collect and combine:  

[Dr. Jacob] Meyer was among the authors of a recent meta-analysis of the link between resistance training and depressive symptoms ... The study, published in the June 2018 issue of JAMA Psychiatry, analyzed 33 clinical trials for the effects of resistance exercise on depression. Results showed that resistance exercise “significantly reduced depressive symptoms” among research participants.

a) In your own words, what does a meta-analysis do?

b) In the meta-analysis described in the paragraph above, there were 33 studies, all of which were experiments. Given the context, what was the independent variable in each of these 33 studies? What was the dependent variable? 

c) Reread the description above, which states that strength training "significantly reduced depressive symptoms". This phrasing does not indicate the meta-analytic effect size (that is, the effect size they obtained by averaging the effect sizes of all 33 studies). To see that effect size, you'd have to turn to the original empirical article, here.  The article is open-access and the forest plot you will find there depicts  the mean, or overall, effect. What is the overall effect size? 

The meta-analysis above showed that strength training can help improve depression. What about a related mental health condition, anxiety? Turns out there's a separate meta-analysis on that:

Resistance exercise doesn’t affect depression alone; research shows that it can reduce symptoms of anxiety, too. For a 2017 study published in the journal Sports Medicine, Gordon and his colleagues conducted another meta-analysis of 16 studies involving 992 total participants; it concluded that resistance training “significantly improves anxiety symptoms among both healthy participants and participants with a physical or mental illness.”

d) How many studies were in this meta-analysis on strength training and anxiety? Estimate how many participants were in each study.

e) What would have been the independent variable in each of these studies? What would have been the dependent variable?

f) Once again, the journalist explains that resistance training "significantly improves" anxiety symptoms, but we don't hear about the effect size. The original empirical article  is behind a paywall. You might be able to access it from your own library. If you can't find it, I'll tell you that the mean effect size (d) of the studies was 0.31, 95% CI [0.17, 0.44]. What information do you need to evaluate this effect size? Is the effect strong or weak? 

As a bonus, the researchers are exploring why strength training might improve depression and anxiety. That is, they are exploring potential mediators of the relationship between strength training and mental health. Here is the first of two mediator explanations, as summarized by the journalist:

One notable finding was that participants showed an improvement in their mood regardless of whether they grew physically stronger from the exercises. ... “Perhaps it might be the sense of accomplishment and confidence that comes from exercising, rather than the achievement of actual strength gains, to explain why study subjects felt better,” Meyer says.

And the second:

Resistance training, like other exercise, induces the release of a protein called BDNF, or brain-derived neurotrophic factor, into the hippocampus region of the brain. Among other functions, the hippocampus is responsible for mood regulation, and in people who are depressed, it shrinks up to 25 percent of its normal volume. The release of BDNF triggers the growth of new brain cells in the hippocampus, restoring it to its full size and improving communication between cells.

“That’s the physiological theory behind why you feel so good after working out,” says Jason Sawyer, an assistant professor of wellness and exercise science at Rhode Island College in Providence.... 

g) Choose one of the mediator descriptions above and sketch a figure depicting the mediator argument. Use a box for each variable, and arrange the boxes in a triangle shape, using the figures at the end of Chapter 9 as a model. 

 

Selected answers

c) The overall (meta-analytic) effect size was d = 0.66 95% CI [0.48, 0.83]. Therefore, strength training groups scored about 2/3 of a standard deviation better than control groups, on average. To evaluate how strong this effect is, it might be helpful to compare the effect of strength training to some other treatments for depression, including cognitive-behavioral therapy or medication. (A 2013 meta-analysis found that CBT's effect on depression is at least d = 0.53, 95% CI [0.43, 0.62].)

d) There were 16 studies and a total of 992 participants, so that's about 165 participants per study. 

e) Each study would have used exercise condition as an IV and anxiety level as a DV. 

f) As you can see, the overall (meta-analytic) effect size of resistance training on anxiety was not as strong as the overall effect size of resistance training on depression (the two CIs do not overlap). This pattern suggests that strength training does more for depression than for anxiety. 

g) Here's how you might sketch one of these mediation proposals:

[strength training condition] -->  [sense of accomplishment] --> [improved mood]

Does "virtue signaling" change minds?

If a lot of us send virtue signals, it can communicate to others that norms are changing. Credit: SEEphotos7171/Shutterstock"

Among people fighting for racial justice, "virtue signaling" may sometimes be derided as empty action. "Virtue signals" are public messages of support for a social justice cause. They could take the form of a yard sign, a t-shirt, or a social media post. Critics sometimes argue that such messages are easy and cheap. It's easy for a company to put a message on their home page that supports Black Lives Matter. It's not hard to wear a t-shirt that supports a social justice cause. But are these expressions of support all talk and no action?

An article in Time magazine argues that virtue signaling can be more powerful than we think. The authors, psychological scientists Jamil Zaki and Mina Cikara, integrate several studies to make their argument. Here's part of their introduction: 

It’s critical to question people whose most radical act is temporarily changing their profile picture. It’s critical they follow their words with action; the time for talk has passed. But focusing solely on virtue signalers obscures their most important role. Like radio waves, the signals they send are received by someone, and those receivers matter, too.

Zaki and Cikara argue that virtue signaling is an important way of influencing others. If a lot of people change their profile pictures to support BLM, for example, more people will start to believe that this movement is popular and they may adopt it, too:

Opinions can work this way: when a particular viewpoint gets a lot of attention, people assume it’s popular, and shift towards it. We receive signals, and are changed by them.

What studies support their argument? Here's one that the authors summarize: 

As social norms shift, individuals shift with them: adopting popular opinions and behaviors, and dropping ones that fall out of style. Norms are especially powerful as they gain steam. In one set of studies, Gregg Sparkman and Greg Walton presented diners in a cafeteria [either] with evidence that 30% of the U.S. population was vegetarian, or that 30% of the population was newly vegetarian. People who learned about this second, “dynamic social norm” were twice as likely to order a meatless lunch themselves. They saw not just where the crowd was, but where it was going, and didn’t want to be left behind.

Here are some questions about the study summarized above:

a) Was this an experiment or a correlational study? How do you know? (If it's an experiment, what kind of study is it?)

b) What were the key variables in the study? Were they manipulated or measured? 

c) Sketch a small graph of the results being described. Would a bar graph or a scatterplot make more sense?

Here's another study Zaki and Cikara introduce:

Signaling and conformity are not inherently positive. In 2016, the U.S. elected a president whose campaign was jammed with racist talking points. Prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants). 

d) This summary doesn't really help you figure out how the study was designed. But here's a chance to think carefully: What kind of study would we need to conduct in order to test the claim that "prejudice subsequently increased, specifically toward groups he targeted (e.g., Muslims, immigrants)."?  There are three variables you would need to include in your study. What are they? 

 

Suggested answers to selected questions

a) This was probably a posttest-only experiment, because it sounds like they manipulated the message about vegetarianism and measured what foods people took in the cafeteria. Here is a link to the original empirical article. 

d) The key dependent variable in this study is prejudice levels, so that's the first variable. Then, in order to establish that "prejudice increased", you'd need to measure prejudice both before and after the election, so that's another variable. Finally, in order to show that it increased specifically toward groups he targeted, you'd need to measure different types of groups (Muslims, immigrants, as well as other potential groups that Trump did not deride in his campaign). You can review the Crandall article here. Do their results really support what Zaki and Cikara are saying about it? (Pay attention to the difference between people's actual prejudice and the normativity of prejudice.) 

 

 

 

 

Racial bias in psychological science journals

A recent study found that in psychology, the authors, participants, topics, and editors of empirical journal articles are mostly White.
Photo credit: Africa Studio/Shutterstock

Is science fair?

You learned in Chapter 2 that empirical journals are the primary place where psychological scientists make their research public. Each empirical journal has an editorial board-- a group of scientists who have power to select the peer reviewers for each article and make the final decision about publication. 

Each step of the way, the stated goals of the publication process are to be fair, to focus on the quality of the science, and to publish the best science possible. But is the process fair?  In light of the renewed energy around anti-racism, a timely new study sheds light on racial diversity in the psychology publication process. The researchers asked, Is race an important variable in the scientific publication process?

The team coded all empirical journal articles that appeared in the most prestigious journals in psychology (over 26,000 articles). There were two journals each from social, cognitive, and developmental psychology. The authors measured:

• how often race was studied as a subject in these articles
• the race of the participants in the studies
• the proportion of the authors of scientific articles were White vs. Non-white
• the proportion of the editors of the journals themselves were White vs. Non-white

Here are some of the results of this published study of published studies, as summarized by the Association for Psychological Science's trade publication:

The results suggest that there is a hierarchy in psychology research that determines who and what gets published. “We are not saying anyone has bad intentions,” said Roberts [one of the study's authors]. “It’s a systemic problem across many fields, and psychological science is no exception.”

First, across the past five decades, articles in psychological journals that highlight race have been rare, and although developmental and social psychology journals have published a growing number of these articles, they have remained virtually nonexistent in cognitive psychology.

Second, most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism.

Third, many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color.

Let's parse these claims. 

Questions

a) What kind of a claim is it (frequency, association, or cause) to state that "across the past five decades, articles in psychological journals that highlight race have been rare"

[And what's "rare"? The published report says that "only 5% of publications highlighted race" (p. 4). This 5% figure is an average across cognitive journals (less than 1%) and social (5%) and developmental journals (8%)]

b) Let's take the second claim, "most journals have been edited by White editors, under whom there has been a notable dearth of published articles highlighting race and racism"

The quoted text above provides two claims in one sentence. What is the frequency claim in the quoted text? What is the association claim in the quoted text? What are the two variables in the association claim?

c) Now for the third claim, "many of the publications that highlight race have been written by White authors, who employed significantly fewer participants of color."

As with b), the quoted test provides two claims in one sentence, a frequency claim and an association claim. What are these? 

d) In Chapter 1 of the 4th Edition (available Fall 2020), you will read about Merton's norms of science--a set of values that the community of scientists usually endorses and strives  to follow. Here they are, with short descriptions (for a full review, see Chapter 1 or this summary). 

Universalism: Anyone can do science as long as they follow the accepted criteria for science, which are established in advance, are objective, and are accessible to all.
Communality: Share! The process of science, including data, methods, and products, should be free and open for all to use. 
Disinterestedness: The quality of the science should be the only thing that matters, not the prestige of the person or place it was conducted. 
Organized Skepticism: Verify everything. Critical peer review matters, every study should be replicated, and criticism of one's evidence is expected and welcome. 

Roberts and his coauthors argued that psychology is too White. Which of Merton's norms does this result seem to violate, and why? 

Suggested answers:

a) This is a frequency claim--a statement about the frequency of the variable, "highlighting race" across 26,000 journal articles.

b) The frequency claim is about the proportion of journal editors who are White vs. a Person of Color (POC). The association claim is about a link between two variables: whether a journal has a White editor or not, and whether that journal published articles highlighting race or not.

c)  The frequency claim is about the proportion of article authors who are White vs. POC. The association claim is about a link between two variables: whether the article was written by a White author or not, and whether that article included POC as participants or not.

d) Open to discussion!

Here is a summary of this work.

Testing a COVID drug: Design flaws and ethical lapses

Photo credit: Alissa Eckert, MS; Dan Higgins, MAMS/CDC

Here's a good example of a bad study. It's related to the COVID pandemic. In this story by NPR  journalist Vanessa Romo, you can read about a doctor in Texas who claims to be conducting research on the drug hydroxychloroquine, using COVID patients in a nursing home as participants. The doctor's research includes both design problems and ethical problems.

According to the journalist, 

...a doctor at a Texas nursing home started giving the anti-malaria drug hydroxychloroquine to dozens of elderly patients diagnosed with COVID-19 and tracking the outcomes in what he's calling an "observational study".

According to the doctor conducting this study, 

"It's actually going well. People are getting better," Armstrong told NPR, adding that after just a handful of days, some of the 39 patients on the medication are showing signs of improvement.

Armstrong admits it is difficult to quantify how much of his elderly patients' improvement is due to the malaria drug or how they would have fared without it. Nor can he explain why other patients are not responding to the tablet doses, though he notes many are only halfway through the five-day cycle.

Design questions

a) Why can't the doctor decide "how they would have fared without it"? Your answer should discuss the study's need for a comparison group. 

b) The doctor's study includes one group--all of the patients in the "study" are taking the drug. What kind of comparison group will be needed to conduct a true clinical trial?

c) If you're as far as Chapter 11, you'll recognize the design of this doctor's study. What is it?

Here's another quote from the article:

[The doctor] said he started by screening those he believed would benefit most and added more people each day. He monitored their blood oxygen saturation, temperatures and how well they were breathing. "The people who [I decided to put] on it were getting sicker but were not so sick that they had to go the hospital," Armstrong explained.

d) What design problems, if any, to you see with the doctor selecting only this type of patient for being in his "observational study?" And, what might have happened if he had included more severely ill patients for his study? (How might the results change?)

Ethical questions

According to federal guidelines and the ethical principles of the Belmont Report, when a person claims to be doing "research," they should obtain ethical approval from an IRB. Ethical approval would require such procedures as obtaining informed consent (respect for persons), minimizing harm (beneficence), and selecting participants fairly (justice). Read these next quotes from the story:

He acknowledged that some families were not aware their relatives were put on the drug, saying that "for the most part," he consulted with each nursing home resident prior to giving them on the tablets.

While the "overwhelming majority of them are awake and alert and can actually have a conversation," Armstrong said some suffer from middle stages of dementia. In some cases, he did not discuss prescribing the tablets with anyone at all before doing so. He said it is common for physicians to prescribe new medications to patients without explicit consent from the patient or family members. "It's not required," he said.

e) Review the sections of Chapter 4 on "respect for persons," "beneficence," and "justice," and apply details from these concepts to this example. 

f) Do you think study design flaws such as the ones you've noted in a), b) and c) should be considered ethical violations, too? Why or why not? 

Safety questions

A potential issue with this doctor's decision concerns the safety of the drug hydroxychloroquine: According to the journalist:

The Food and Drug Administration has not approved the drug for the treatment of COVID-19. 
and
[the drug] is known to have serious negative health impacts. That is why so many in the medical community worry about prescribing it without such proof.

g) The safety of the drug is relevant to the ethical principle of beneficence. Can you explain why?

Finally...

Another wider ethical issue (perhaps outside the scope of a research methods course) concerns how the doctor obtained the drug, which has been in short supply:

[The doctor] began making calls to track down a source for the medicine, which is in short supply.That's when his political connections proved useful.

Armstrong, who is a prominent GOP activist, called Republican Lt. Gov. Dan Patrick. He says Patrick reached out to Texas state Sen. Bryan Hughes, also a Republican, who knew someone on the board of the New Jersey-based company Amneal Pharmaceuticals. The company, which makes and distributes the drug, has donated more than a million tablets nationwide, including to the statesof Texas and Louisiana.Two days later, Armstrong had received more than enough medication to begin giving it to patients. 

h) Do you think it is reasonable for a doctor to use political connections to obtain medications that are in restricted supply? Why or why not? 

Suggested answers to selected questions

a) and b) and c) Because the doctor's "study" has only a group of patients who are taking the drug, he doesn't have a comparison group. In Chapter 11 this is called a one-group, pretest-posttest design, otherwise known as the Really Bad Experiment. The design is flawed. If patients improve on the drug, the doctor will not know whether their improvement was caused by the drug itself, or simply because their bodies were healing or because they were getting excellent nursing care, or even some other reason. 

The doctor's behavior in this COVID study reminds me of the Halstead radical mastectomy, which for decades was the dominant treatment for breast cancer in the U.S.. Surgeons at the time were so convinced that this treatment was the best one for breast cancer that they never bothered to compare it to another treatment--they never tested its efficacy. (When a randomized control study was finally conducted in the 1970s, it found that the surgery was no more effective than other, less invasive treatments.)

d) By selecting only relatively less sick patients, the doctor is stacking the deck in favor of finding evidence that the drug is effective. Many of the patients are likely to get better, so the doctor may conclude that the drug has worked. But of course, without a comparison group of equally healthy people who are not getting the drug, we cannot know if the patients would have recovered anyway.

e) f) g) The lack of informed consent--especially for vulnerable patients who have Alzheimer's--is a clear violation of the respect for persons principle. And the use of a risky treatment, one that is known to be dangerous to people with heart conditions, is likely a violation of the beneficence principle. I would argue that when a researcher conducts a flawed study like this, it also a violation of beneficence--the study puts participants at risk, while the benefit of the potential information gained from the research is negligible. We might also argue that the study violates justice: by studying only people at a nursing home, these patients bear the burden of being research participation when anyone can benefit from this research (assuming, of course, that the research were actually well-designed).

Many thanks to Dr. Stephen Chew for pointing me to this example!

Post script: Looking for more examples of bad studies, as described by science writers? Here are two:

Here's a critique of the many problems with another early published study on hydroxychloroquine.

Here's critique of studies supposedly investigating the death rate of COVID-19. 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers. 

 

Dog people live longer

The study found an association between dog ownership and longevity. Which third variables were controlled for? Which third variables might still be responsible for the relationship? Photo: Shutterstock

I can't resist blogging about research on dogs. Here's a story on NPR that summarized several studies that have tested the correlation between longevity and dog ownership. The journalist introduces the topic this way: 

Dogs and humans that interact with one another get a jolt of oxytocin, the so-called "cuddle hormone." And, if you get to look at dogs and hug them every day, you just might live longer than people who don't have to clean animal hair off their clothes, according to a pair of studies out this month.

Here is a summary of one of the studies:

[researchers] looked over a 12-year period at dog owners in Sweden who have cardiovascular disease. They adjusted for the kinds of things we know affect cardiovascular health — age, demographics, socioeconomic status, marital status, number of children at home — and even after adjusting for all of that, they found a benefit of dog ownership [on longevity].

....they followed over 100,000 people and found a 21% reduction in deaths from any cause.

The study above used multiple regression. 

a) What was the dependent (criterion) variable in this study?  What were the predictor variables? 

b) Sketch a little regression table with results that would be consistent with what is mentioned above. Put the criterion variable at the top, and list the predictor variables. Which betas would be significant here? 

c) The researchers controlled for age, demographics, SES, marital status, and children--but regression doesn't control for all third variables. What else could be associated with both dog ownership and longevity? 

d) They mention a "21% reduction in deaths from any cause."  Which of the four big validities is this concerning? 

One of the experts who was interviewed for this story said:

In particular, I found it very convincing and striking that the benefits seems to be larger among individuals who live alone compared to multiperson households. That suggests the companionship of a dog is possibly very important to their heart health.

e) This is a moderator! Identify the variables in this relationship and complete this sentence: ____ moderates the relationship between ____ and _____.

 

In part of the article, the journalist presents quotes from an expert, who seems to equivocate on the causality issue:

"When you look at the big picture and look at all the evidence around dog ownership and cardiovascular health, it's pretty clear the signal is real and likely causal," he says.

In other words, he's convinced getting a dog improves your health — especially for those with heart disease.

The research is not definitive, though, Kazi adds. "They're not randomized trials, the gold standard for what we would do to evaluate a new drug," he says.

f) It is frustrating to read that the expert believes the "signal is real and likely causal" and then admits that "it's not definitive." You can set this expert straight. Why can't this correlational study support causation?  Apply covariance, temporal precedence, and internal validity. When you address internal validity, check out answer c) above!

 

 

Suggested answers

a) The criterion variable is longevity (being alive at the end of the study). The predictor variables are dog ownership, age, demographics, SES, marital status, and children living at home. 

b) The table might look like this:

DV: Longevity (or being alive after 12 years)

IV                            Beta (sig)
Dog ownership         .12*
Age                            -.11
SES                              .13*
Married                        .08*
Children at home        .07*

d) Current health and mobility might be associated with both dog ownership and longevity. After all, you  may need to have a certain level of health and mobility to own a dog, and healthier folks are more likely to be alive at the end of the study. 

e)  Living status (being alone or married) moderates the relationship between dog ownership and longevity.  

f) There is covariance--there is a link between owning a dog and living longer. 

There is temporal precedence-- the study took place over several years and dog ownership was measured first, and death rates later. 

There is not full internal validity. They controlled for age, demographics, SES, marital status, and children living at home, so they can rule those out. But they did not control for everything. One possibility is that current health and mobility might be associated with both--after all, you  may need to be at a certain level of health and mobility to own a dog, and those folks are more likely to be alive at the end of the study.