Everyday Research Methods

Art Makes You Smart

Here's a great example of a field experiment. In a piece in the New York Times, a group of researchers write about a study they conducted, in which students were randomly assigned to visit museums.

In the opening paragraphs, you'll see a comment about the ability of resesarch to support causal claims:

For many education advocates, the arts are a panacea: They supposedly increase test scores, generate social responsibility and turn around failing schools. Most of the supporting evidence, though, does little more than establish correlations between exposure to the arts and certain outcomes. Research that demonstrates a causal relationship has been virtually nonexistent.

According to the Times story, the Crystal Bridges Museum of American Art is an art museum that opened in 2011. Part of its endowment includes funds for schools to take free trips to tour the museum. Because there were more requests from schools than the museum could accommodate, the museum (guided by these researchers) decided to conduct a lottery to decide which students would visit. They used the lottery "winners" as the experimental group and the other kids as the control group. As the authors say:

Through a large-scale, random-assignment study of school tours to the museum, we were able to determine that strong causal relationships do in fact exist between arts education and a range of desirable outcomes.

Here's part of the description of the study that the researchers conducted:

Over the course of the following year, nearly 11,000 students and almost 500 teachers participated in our study, roughly half of whom had been selected by lottery to visit the museum. Applicant groups who won the lottery constituted our treatment group, while those who did not win an immediate tour served as our control group.       

Several weeks after the students in the treatment group visited the museum....[we] asked them to write an essay in response to a work of art that was unfamiliar to them. These essays were then coded using a critical-thinking-skills assessment program developed by researchers....       

Further, we directly measured whether students are more likely to return to Crystal Bridges as a result of going on a school tour. Students who participated in the study were given a coupon that gave them and their families free entry to a special exhibit at the museum. The coupons were coded so that we could determine the group to which students belonged. Students in the treatment group were 18 percent more likely to attend the exhibit than students in the control group.

 

Questions

a) What is the independent variable in this design?  Is it independent groups or within groups? Is this experiment a posttest only design? Prettest/posttest? Repeated measures? Or concurrent groups?

b) There are several dependent variables listed. What are they?

c) Sketch a graph of the outcome of the study, selecting one of the dependent variables.

d) The headline says, "Art makes you smart," which is a causal claim. Does the design of the study support this causal claim? (If not this one, can it support the claim that "Visiting a museum makes kids' critical thinking scores improve?")  Apply the three causal criteria of covariance, temporal precedence, and internal validity to this example.

e) Write a question about this study's construct validity; internal validity, external validity, and statistical validity

Commuting and political engagement

An NPR story carried the headline, "Study: Commuting Adversely Affects Political Engagement." The verb in the headline, "affects" makes this a causal claim. Is there evidence in the story that commuting causes changes in political engagement?

In an interview, researcher Joshua Johnson stated the relationship in causal terms, too:

We found that when people spent more time commuting [or spending] extra hours on their commute, that made them less likely to be engaged in politics.

a) Think for a moment: What are the variables in this research? Would they most likely be measured or manipulated?

b) Sketch a scatterplot that would show the relationshiop between commuting time and political engagement.

c) Can the researchers really support a causal claim from these data, at least as decribed here?

 

Shankar Vedantam, the NPR host who described the study, explained:

There's something about commuting in particular that seems to affect engagement, and the researchers are drawing here on this paper on earlier work by the behavioral economist Daniel Kahneman. He's found that commuting ranks among the most unpleasant parts of people's day. There's something uniquely stressful about commuting, and so when you get home after a hellacious day, you really have nothing to give to other people in terms of civic engagement, in terms of getting involved in your neighborhood politics.

Here's a reminder to my readers--just because you can offer a good explanation for a correlation, that doesn't change the fact that a correlational study cannot definitively support causation!

But back to the story. The correlational pattern gets more complex--it involves a moderator! I'll quote  the transcript of the NPR report:

INSKEEP: Are all people affected the same way by this stress?

VEDANTAM: So this is a really important point, Steve, because it turns out that even though there's a general connection between political engagement and commuting, the effect is not experienced evenly by everyone. Commuting disproportionately seems to cause the poor to disengage from politics. And as we go up the income ladder, the effects that commuting have on political engagement actually decrease....

Commuting is stressful for everyone, but the poor find it harder to buffer themselves against the effects of the stress. When you're well off, you come home from a terrible day, you can go out for dinner. You can buy yourself a treat. When you're poor, you have less access to those kinds of safety nets.

d) In this example, we can say that social class moderates the relationship between commuting and political engagement. Sketch a moderator table that would capture this pattern.

 

Suggested Answers:

a) Think for a moment: What are the variables in this research? Would they most likely be measured or manipulated?

The two variables are time spent commuting and degree of political engagement. These are both measured variables, because it would be practically and ethically impossible to assign people to have a long commute.

b) Sketch a scatterplot that would show the relationshiop between commuting time and political engagement.

Your scatterplot should have Time spent commuting on one axis, and Degree of political engagement on the other. The dots should be spread in a negatively sloping pattern, because the more time people spend commuting, the less politically engaged people are.

c) Can the researchers really support a causal claim from these data, at least as decribed here?

No. There is covariance here: time spent commuting goes with less political engagement. Temporal precedence seems ambiguous: Which variable came first? Did commuting come first, followed by less engagement? Or do less politically engaged people prefer to work farther from the center of things? Finally, internal validity isn't clear--there may be third variables that explain this relationship. Perhaps younger people tend to commute further, and they are also less politically engaged. Perhaps less educated people commute further and are also less engaged. We would want to see Johnson's original research report--he probably statistically controlled for several such variables. His report would tell us which of these possible third variables were statistically controlled for, to help rule out third variable explanations.

d) In this example, we can say that social class moderates the relationship between commuting and political engagement. Sketch a moderator table that would capture this moderation pattern.

Here's one possible pattern (These data are fabricated, but I made them to fit the pattern described):

Social class level        Relationship between commuting time and engagement

Lower SES                    -0.21*

Middle SES                 0.03

 

Note: Initially, to make the moderator easier to understand, I oversimplified the quote.  But in fact, the moderation pattern was a little more complex:

Commuting disproportionately seems to cause the poor to disengage from politics. And as we go up the income ladder, the effects that commuting have on political engagement actually decrease until we get to the very wealthy, where the longer your commute, the more likely you are to be politically engaged.

To represent the full pattern, you might show this (again, I fabricated the data to illustrate):

Social class level        Relationship between commuting time and engagement

Lower SES                    -0.21*

Middle SES                 0.03

Highest SES                0.17*