Everyday Research Methods

Therapy dogs in the emergency room

The study focused on therapy dogs' effect on perceptions of pain in emergency room patients. Photo: monkeybusiness/Deposit Photos

Adults and children alike can benefit from petting a therapy dog (a friendly pet who makes rounds with a trainer, perhaps to a clinic, pediatric unit, or even on campus during finals week). In this study, covered by journalists at NPR news, researchers from the University of Saskatchewan tested the efficacy of therapy dogs for people who visited an emergency room (ER) complaining of pain. 

 

Here is how the journalist described the study: 

In a controlled trial, researchers asked more than 200 patients to report their pain levels. One group of patients received a 10-minute visit from a therapy dog and the other group did not. After the dog visit, researchers asked patients in both groups to report their pain levels again.

Those who spent 10 minutes with the dog reported less pain [over time], the study found.

Based on this simple description, you can assume that this study was an experiment. 

 

a) What was the independent variable (IV)? What were its levels?

b) Was the IV manipulated as independent groups or within groups? 

c) What was the dependent variable (DV)? 

d) Which of the designs in Chapter 10 does this represent: Posttest only? Pretest-posttest? Repeated measures? or Concurrent measures? 

e) Sketch a small graph of the study's outcome. You don't know what the exact values are, but you should be able to estimate what should be higher and lower. Use either a bar graph or a line graph, following examples in Chapter 10. 

 

The journalist states, "A new study shows that a 10-minute visit from a therapy dog can help relieve patients' pain in the emergency room."  

f) What makes this a causal claim?

b) Assuming that people were randomly assigned to the two groups. Can the study support this causal claim? Name and apply the three causal criteria. 

 

You can read the real study for free: The empirical article is open access in PLOS. 

Dog people live longer

The study found an association between dog ownership and longevity. Which third variables were controlled for? Which third variables might still be responsible for the relationship? Photo: Shutterstock

I can't resist blogging about research on dogs. Here's a story on NPR that summarized several studies that have tested the correlation between longevity and dog ownership. The journalist introduces the topic this way: 

Dogs and humans that interact with one another get a jolt of oxytocin, the so-called "cuddle hormone." And, if you get to look at dogs and hug them every day, you just might live longer than people who don't have to clean animal hair off their clothes, according to a pair of studies out this month.

Here is a summary of one of the studies:

[researchers] looked over a 12-year period at dog owners in Sweden who have cardiovascular disease. They adjusted for the kinds of things we know affect cardiovascular health — age, demographics, socioeconomic status, marital status, number of children at home — and even after adjusting for all of that, they found a benefit of dog ownership [on longevity].

....they followed over 100,000 people and found a 21% reduction in deaths from any cause.

The study above used multiple regression. 

a) What was the dependent (criterion) variable in this study?  What were the predictor variables? 

b) Sketch a little regression table with results that would be consistent with what is mentioned above. Put the criterion variable at the top, and list the predictor variables. Which betas would be significant here? 

c) The researchers controlled for age, demographics, SES, marital status, and children--but regression doesn't control for all third variables. What else could be associated with both dog ownership and longevity? 

d) They mention a "21% reduction in deaths from any cause."  Which of the four big validities is this concerning? 

One of the experts who was interviewed for this story said:

In particular, I found it very convincing and striking that the benefits seems to be larger among individuals who live alone compared to multiperson households. That suggests the companionship of a dog is possibly very important to their heart health.

e) This is a moderator! Identify the variables in this relationship and complete this sentence: ____ moderates the relationship between ____ and _____.

 

In part of the article, the journalist presents quotes from an expert, who seems to equivocate on the causality issue:

"When you look at the big picture and look at all the evidence around dog ownership and cardiovascular health, it's pretty clear the signal is real and likely causal," he says.

In other words, he's convinced getting a dog improves your health — especially for those with heart disease.

The research is not definitive, though, Kazi adds. "They're not randomized trials, the gold standard for what we would do to evaluate a new drug," he says.

f) It is frustrating to read that the expert believes the "signal is real and likely causal" and then admits that "it's not definitive." You can set this expert straight. Why can't this correlational study support causation?  Apply covariance, temporal precedence, and internal validity. When you address internal validity, check out answer c) above!

 

 

Suggested answers

a) The criterion variable is longevity (being alive at the end of the study). The predictor variables are dog ownership, age, demographics, SES, marital status, and children living at home. 

b) The table might look like this:

DV: Longevity (or being alive after 12 years)

IV                            Beta (sig)
Dog ownership         .12*
Age                            -.11
SES                              .13*
Married                        .08*
Children at home        .07*

d) Current health and mobility might be associated with both dog ownership and longevity. After all, you  may need to have a certain level of health and mobility to own a dog, and healthier folks are more likely to be alive at the end of the study. 

e)  Living status (being alone or married) moderates the relationship between dog ownership and longevity.  

f) There is covariance--there is a link between owning a dog and living longer. 

There is temporal precedence-- the study took place over several years and dog ownership was measured first, and death rates later. 

There is not full internal validity. They controlled for age, demographics, SES, marital status, and children living at home, so they can rule those out. But they did not control for everything. One possibility is that current health and mobility might be associated with both--after all, you  may need to be at a certain level of health and mobility to own a dog, and those folks are more likely to be alive at the end of the study. 

 

 

 

Did the show "Thirteen Reasons Why" cause an increase in suicides?

The study used a quasi-experimental design.  Credit: Beth Dubber/Netflix/Kobal/Shutterstock

Update 7/19/2019: Netflix has decided to remove the scene depicting the main character's actual suicide, perhaps in response to data from this and other studies. Here's one story on the change. 

The Netflix series "Thirteen reasons why" tells the story of a teenage girl who commits suicide, leaving behind audio recordings about people who contributed to her decision. When the series was released in April, 2017, mental health and school professionals became concerned about the possibility that young viewers of the show would experience increased risk of suicide themselves. The phenomenon of "suicide contagion" occurs when a publicized suicide leads to other suicide attempts. Young people may be particularly susceptible to its effects. Suicide is the third leading cause of death among American teenagers (ages 12 to 19, according to the CDC). 

This Mashable journalist covers an empirical journal article published in JAMA Psychiatry in 2019.  The Mashable story reports: 

A new study, conducted by different authors, came to a similar conclusion: In the three months after 13 Reasons Why launched, the youth suicide rate for 10- to 19-year-olds rose unexpectedly by 13 percent. Based on historical trends dating back to 1999, 94 more American adolescents and teens died by suicide during the three-month period than otherwise expected. Similarly, the study published last month looked at suicide deaths in the nine months following the show's launch and found an additional 195 fatalities. 

Students and instructors will find it useful to consult the publicly-available JAMA Psychiatry article that contains graphs of this study. Specifically, the solid orange lines in Figure 2 show the suicide rates for 24 months of time: 15 months before the show debuted, and about 9 months after. 

a) This study used a quasi-experimental design. Which of the designs from Chapter 13 was it? (Non-equivalent control group posttest only? Non-equivalent control group pretest-posttest? Interrupted time series design? Non-equivalent control group posttest-only design?)

b) In the quote above, the journalist reported that an 13 percent increase in suicide equals an additional 94 adolescent teen deaths. This figure captures the effect size of the study's result. What do you think of this effect size? Would you call it small or large? 

The empirical journal article also documents the publicity the show received via social media. Here's how the journalist describes the evidence: 

Niederkrotenthaler and his co-authors looked at potential exposure to the show between April 1 to June 30, 2017, when the first season of 13 Reasons Why had just debuted and was most popular on Twitter and Instagram. Using social media data as a proxy for understanding peak interest in the show, the researchers suggest the show's portrayal of suicide created a 'contagion effect' The phenomenon can happen when media coverage of suicide leads to suicide clusters. 

Students and instructors can consult the empirical article's Figure 1 to see the results of the social media analysis. 

b) Why did the authors of the JAMA article include the social media piece? How does this evidence help them make their argument? 

The researchers cannot establish for certain that viewing the show caused the increase in suicide. Here's the Mashable journalist: 

While the researchers couldn't determine whether the children who took their own lives watched the series, an accompanying editorial written by researchers at Harvard and Stanford argues the study provides "strong evidence" that the Netflix show may have led to that increase. 

"One cannot draw definitive causal conclusions from such data, but that no similar increase was seen in suicide rates other than for the age group to which the media portrayal pertained provides some further compelling evidence that the excess suicides may indeed have been owing to the series," the editorial's authors wrote. 

c) The quote above adds an additional detail about the study. The researchers not only tracked the suicide rates of people aged 10-19; they also tracked suicide rates of people in the 20-29 and 30+ age group. The researchers found an increase in suicide after April 2017 only for the teenage group. How does this pattern of results support the argument that the show led to increased teen suicide? 

If you want to talk to someone or are experiencing suicidal thoughts, text the Crisis Text Line at 741-741 or call the National Suicide Prevention Lifeline at 1-800-273-8255. Here is a list of international resources. 

Can puppy pictures help your relationship?

Does pairing this cute puppy with a photo of your partner help 
improve your relationship?
Photo credit: Matanya/Wikimedia Commons

Professor James McNulty, a social psychologist who studies relationships, was asked by the Department of Defense to develop an intervention to help couples cope with the separation of a deployment. How can couples stay close when their partner is far away for months at a time? 

McNulty and his team of students tested an intervention in which couples viewed photos of their partner placed next to cute images of puppies, kittens, or baby bunnies.  Would just seeing your spouse next to cuteness make you feel closer to your spouse? Here's how a journalist from National Geographic described the study: 

For six weeks, one group of partners was shown a stream of images every three days that repeatedly paired their partner's image with that of a positive image, like a puppy. The control group saw pictures of their partners paired with neutral images, such as a shirt button.

Every two weeks [beginning at the start of the study], the couples' attitude toward each other were measured by the positive and negative words used when reviewing a series of faces, of which their partner was included. When couples were evaluated at the end of the study, those who had been shown positive images had more positive associations with their partners and improved marriage quality.

There were 144 couples in the study. You can read the empirical journal article here, in the journal Psychological Science. 

Questions.

a) McNulty and his team conducted an experiment. What are the independent and dependent variables? Was the independent variable manipulated as within groups or independent groups? 

b) Which type of experiment does it appear to be, based on the description: Posttest-only? Pretest-posttest? Concurrent measures? Repeated measures? 

c) Think about the design of this study and consider the 12 internal validity threats presented in Table 11.1. Which threats can you probably rule out, given the design of this study? Which threats do you think might still be in question? What questions might you ask about the study to assess these remaining threats to internal validity? 

 

Suggested answers:

a) The independent variable was which type of treatment they were in: Pairing their partner's face with cute animals or with neutral objects. This was an independent groups IV. The dependent variables included the positive and negative emotions people felt, and their marriage quality. That is, there were multiple dependent variables in the study.

b) This appears to be an independent groups design; specifically, a prettest-posttest design.

c) The researchers randomly assigned couples to the two groups, so we can rule out selection effects. It's not a repeated measures design, so we don't need to worry about order effects. Because they used a control group and an experimental group, we can rule out threats such as maturation, testing, regression, instrumentation, and history, because these threats would have applied equally to both experimental and control groups and thus evened out. We might ask about attrition (selection-attrition) in the two groups in case the unhappiest couples dropped out of the experimental group more often (however, if attrition was similar in the two groups then attrition is not an internal validity threat). We might wonder if some couples guessed the hypothesis of the study and changed their answers (experimental demand). Since both groups received a treatment (either the cute photos or the neutral photos), we can probably rule out a placebo effect. The dependent measures appear to be self-report, so observer bias is unlikely here. 

Do kids do better in private schools?

What evidence would it take to convince us that it's the schools, not other factors, that are responsible for the outcomes of private school students? Photo: Image Source / Alamy Stock Photo

A large study has compared the outcomes of children who've attended private schools to those who've attended public schools.  A journalist summarized the report in the Washington Post. The study provides a nice example of how multivariate regression can be used to test third variable hypotheses. 

When we look simply at the educational acheivement of students in private schools vs. public schools, private school students have higher achievement scores. However, all such studies are correlational, because the two variables--Type of School and Level of Achievement--are measured. 

Therefore, such studies how covariance, because the results depict a relationship. The study may even show temporal precedence, because attending school presumably precedes the measure of achievement. However, such studies are weak on internal validity. We can think of several alternative explanations for why children in private schools are scoring higher. 

One major alternative explanation is socioeconomic status. Children from wealthier families are more likely to afford private schools. And in general, children from wealthier families tend to score higher on achievement tests. 

The Washington Post journalist quoted one of study's authors, Robert Pianta, who summed up the study's results this way:

“You only need to control for family income and there’s no advantage,” Pianta said in an interview. “So when you first look, without controlling for anything, the kids who go to private schools are far and away outperforming the public school kids. And as soon as you control for family income and parents’ education level, that difference is eliminated completely.”

Questions

a) Draw little diagrams similar to those in Figure 8.15 (in the 3rd ed.) to depict the arguments being made in this study. What would A be? What about B? In the quote from Pianta, above, what would the C variable(s) be? 

b) The researchers used type of school (private vs. public), which is a categorical variable. But in some analyses, the researchers also used "number of years in private school" as an alternative version of this variable. Is "number of years in private school" categorical, ordinal, interval, or ratio data? 

c) Sketch a mock-up regression table with the criterion variable  at the top and predictors below (Use Table 9.1 as a model). Which variable do you think the researchers selected as the criterion (dependent) variable in their analyses? Which variable(s) would have been the predictors? 

d) Now that you know what the results were, think about how the beta associated with "number of years in private school"  would change when parental SES is added and removed from the regression analyses. 

Original journal article seems to be open-access. Give it a try!

Suggested answers

a) A and B would be Type of School and Level of Acheivement. It doesn't really  matter which one is A and which one is called B. C would be Family Income and/or Parental Education.

b) Ratio data (zero is meaningful in this scale because you could attend zero years of private school)

c) The criterion variable would be Achievement, and the predictors would be Number of Years of Private School, Family Income, and Parental Education.

d)When the Number of Years of Private School is on the table (in the analysis) by itself, its beta is likely to be positve and significant (more years of private school goes with higher achievement). When Family Income and Parental Education are added to the table, the beta for Number of Years of Private School should drop to zero.  This pattern of results is consistent with the argument that Family Income and Parental Education are the alternative explanation for the original relationship .

Vacations and job stress

To what extent does the evidence support a causal influence of vacations on happiness and stress? Photo: Syda Productions/Shutterstock

Here are some quasi-experimental and correlational studies on vacations, just in time for the end of summer. The APS website describes a few studies consistent with the argument that vacations can be good for your mental health. Here's one study by researchers Sabine Sonnentag and Jana Kühnel:

The researchers surveyed 131 teachers before and after a two week break from school.

First, they had the teachers complete a measure of exhaustion—how emotionally drained and burned out they felt the day before heading out for vacation. The teachers then completed weekly surveys on how engaged they were with their work, relaxed, and stressed they felt four weeks after returning from vacation.

As predicted, the results indicated that vacationing had a beneficial effect. Not only did the teachers report feeling less tired and emotionally burned out, they also reported feeling more engaged and positive about their work.

a) This is a quasi-experiment. What is the study's "independent" variable? What is/are its dependent variable(s)?

b) Would you call the design a non-equivalent groups posttest only? non-equivalent groups pretest/posttest? Interrupted time series? Or non-equivalent groups interrupted time series? 

c) Consider the 12 internal validity threats in Table 11.1. Which threats can this study rule out? Which threats might still apply?

d) Sketch a graph of the results of the study, incorporating this (more negative) message: 

But, these benefits were fairly short-lived, particularly for those teachers who came back to especially difficult students and heavy workloads. Within four weeks, the vacation’s positive benefits had faded and teachers were back to their initial levels of stress and emotional exhaustion.

The article also suggests that when it comes to spending money, money spent on vacations is associated with more happiness than money spent on material goods. 

...psychological scientists Amit Kumar and Thomas Gilovich of Cornell University and Matthew Killingsworth of University of California, San Francisco tracked moment-to-moment data from 2,266 adults as part of a large-scale experience-sampling project. Participants received notifications from the researchers on their iPhones at random times throughout the day.

Comparing data from individual participants across different times, Gilovich and colleagues found that people were happier at times when they were thinking about a future experiential purchase, like a ski trip, than they were at times when they weren’t thinking about a purchase at all. There was no relative increase or decrease in happiness when people were thinking about a future material purchase.

e) The above study is a correlational one, with a twist. The researchers computed a correlation for each individual person, using "experience" as the unit of analysis. Given the results described above, what might a bar graph depict for a typical person in this study? (what would be on each axis, and what would the results pattern depict?)

Measuring charter schools

Are charter schools effective?  Philadelphia Inquirer columnist Joel Naroff wrote about the data we might use to decide whether charter schools are doing well. He describes the pitfalls of using achievement test scores to measure school effectivness. His article uses the language of multiple regression and internal validity to describe how difficult this process is.

Specifically, Naroff mentions the following about charter schools, discussing

....what researchers call "selection bias." Parents have to choose to send their children to a charter school, meaning that charters start with students who have high parental involvement.

a) To translate his comment into our own research methods terms:  Imagine a study that compared achievement test scores in a city's charter schools with the city's traditional public schools. Sketch a simple bar graph showing that charter schools score higher in achievement than traditional schools. Does this result mean that charter schools cause better school performance?  What are some internal validity issues here?  Apply the term  "selection bias" or "selection effect," as the author is doing above.

This article also states that

Unfortunately, we don't adjust test scores for differences in factors such as intelligence, income, parental involvement, or school facilities. Instead, we use misleading, unadjusted test-score comparisons. That means some schools that test well might not be doing all they can, while those with failing scores might be making outstanding progress. We are largely clueless about what is happening.

In this paragraph, the author seems to be advocating for using multiple regression to evaluate the effectiveness of charter schools. 

b) Why would multiple regression help us rule out the selection bias problem the author mentioned earlier?

c) Sketch a possible table of multiple regression results, as the author suggests above: Assume that achievement test score is the dependent variable of interest. What would be your predictor variables? What results would show that charter schools really are more effective than traditional schools, even controlling for these selection effect variables?  What results would show that charter schools are not more effective than traditional schools?

d) Do you agree that "we are largely clueless" about what is going on? Do you think the regression approach he mentions is a good idea? Are there any downsides to the regression approach?

 

Suggested answers to question c:

Below is a possible table of (fabricated) results that would show that charter schools really are more effective than traditional schools, even controlling for the selection effect variables the author mentioned.

DV: Achievement test scores

Predictor                                                                                    Beta

Traditional vs. Chater (0 = traditional, 1 = charter)       0.15**
Entering achievement test score                                         0.59*
IQ of child                                                                                0.20**
Child's family income                                                            0.11**
Child's parental involvement                                               0.18*
School's level of facilities                                                       0.08

Note: Data are fabricated for teaching purposes.

* represents a statistically significant effect. This table suggests that charter schools are associated with better achievement test scores, even when controlling for earlier scores, IQ of child, income, parental involvement, and school facilities.

 

Now, below is a possible table of (fabricated) results that would show that charter schools are not more effective than traditional schools, even controlling for these selection effect variables

DV: Achievement test scores

Predictor                                                                                    Beta

Traditional vs. Chater (0 = traditional, 1 = charter)       0.03
Entering achievement test score                                        0.57*
IQ of child                                                                               0.24**
Child's family income                                                           0.13**
Child's parental involvement                                               0.16*
School's level of facilities                                                       0.06

Note: Data are fabricated for teaching purposes.

* represents a statistically significant effect. This table suggests that charter schools are not associated with better achievement test scores, when controlling for earlier scores, IQ of child, income, parental involvement, and school facilities.

Does using marijuana lower your IQ?

A study tested long-term IQ changes of people who smoked marijuana as teenagers. The study is described here.

...participants were tested for IQ twice in the study: once at age 13, and once at age 38.  ...researchers found that those who used marijuana regularly before the age of 18 showed a drop in IQ score. The same drop in IQ wasn't found for those who began to smoke up regularly after 18.

a) According to the description above, what kind of a study is this?  Experimental? Correlational? Quasi-experimental?  Give the most precise answer you can. 

b) Make a graph of the results described in this sentence from the journalist's report: 

Essentially, the more participants smoked before age 18, the heavier the IQ loss recorded by the researchers.

c) Finally, the journalist reports that:

...while the study's findings seem to give some weight to the "stoner" stereotype of longtime heavy cannabis users, other factors, such as depression, could also explain both marijuana use and a lower IQ in participants.

What kind of validity is this sentence addressing? What kind of study would be able to rule out the idea that "other factors, such as depression" are responsible for the lower IQ and marijuana use? (There a couple of study designs that could address this issue.)

 

Should we stop calling people "consumers?"

Time magazine presents a story this month about a line of research on materialism. The full story is accessible here.

One of the studies conducted by the research team, led by psychologist Galen Bodenhausen, was on the effect of calling people "consumers" versus "individuals."

In one of the study’s tests, participants were presented with the following hypothetical scenario: There is a water shortage, and four people (including the participants) must share a well. Sometimes in the experiment, participants were called “consumers,” and other times they were referred to as “individuals.” And apparently, words can be powerful.

The “consumers” rated themselves as less trusting of others to conserve water, less personally responsible and less in partnership with the others in dealing with the crisis. The consumer status, the authors concluded, “did not unite; it divided.”

a) In this experiment, what was the independent variable ? What was the dependent variable?

b) Because this is an experiment, you may be able to make a causal claim from this research. What would the causal claim be?

c) What internal validity concerns might you raise about this study? If you were the authors, how would you have ensured internal validity in this experiment?

Curly or straight? A journalist experiments on her hair...

In this ABC news video, a reporter shows what happened when she presented herself with her naturally curly hair or her straightened hair, in three situations.

1. Consider the first experiment--with the panel of men.

a. What were the independent and dependent variables in the reporter's study?

b. How did she manipulate the independent variable--as within groups or between groups?

c. Interrogate the internal validity of this study. What confounds do you notice and how do they affect her ability to conclude that "men like women with straight hair better"? How might you redesign her study to make better conclusions?

d. Interrogate the statistical validity of her report---what questions are worth asking?

e. What about external validity? Ask questions about two aspects of this study's external validity.

 

Journalist Taryn Winter Brill demonstrated what happened when she straightened her natually curly hair. Photo by Suzanne Kreiter/The Boston Globe via Getty Images

 

2. Now consider the second experiment--with the group of children.

a. What were the independent and dependent variables in this second study?

b. How did she manipulate the independent variable this time--as within groups or between groups?

c. Interrogate the internal validity of this study. What problems do you notice and how do they affect her ability to conclude that "kids prefer women with straight hair"? What should she change in a future study?

3. Finally, consider the last experiment--the job interview.

a. What were the independent and dependent variables in this third study?

b. How did she manipulate the independent variable this time--as within groups or between groups?

c. Interrogate the internal validity of this study. What problems do you notice and how do they affect her ability to conclude that "employers are more likely to hire women with curly hair"?