Everyday Research Methods

Question wording matters

Collecting accurate data in a poll is difficult business. Many of us focus on the sampling strategies of polling organizations, and rightly so: External validity depends on whether we include "cell phone only" numbers in our samples, how we account for different rates of responding across certain groups, and so on. However, this post reminds us that question wording also matters. Construct validity is just as important for polls.

In this report, Pew Research Center describes how people's opinions change depending on how a question is posed. The poll asked Americans whether we should increase the size of the House of Representatives. People in the poll were randomly assigned to hear either the original question, or the same question with additional context. Notice how people's support changes when additional context is added: 

“The Electoral College, Congress, and representation,” Pew Research Center, Washington, D.C. (April 26, 2018). https://pewrsr.ch/2rLd0OI, accessed on May 16, 2018

 

a) Even though this is a polling example, Pew conducted an experiment. What are its independent and dependent variables? 

b) What causal claim could this polling result support? 

c) In your opinion, which polling question--the original one or the one with added context--provides the most accurate measure of people's opinion? 

d) Find another poll question on the Pew Research web site. Speculate how some additional information might change people's responses. 

 

 

 

 

 

 

 

Below is another presentation of the same results. Notice how the two  different question wordings seem to affect people differently, depending on their political affiliation.

“The Electoral College, Congress, and representation,” Pew Research Center, Washington, D.C. (April 26, 2018). https://pewrsr.ch/2rLd0OI, accessed on May 16, 2018

 

e) Why might Democrats be more influenced by the contextual information provided? 

f) This could be viewed as a 2 x 2, IV x PV factorial design, in which the IV is "question wording" (original or added context) and the PV is "political affiliation" (Democrat or Republican). If you've studied Chapter 12, perhaps you can identify whether there are main effects and interactions in this pattern of data. 

The same Pew report contains additional examples of how question wording alters responses to a question about the Senate, and shows how different subgroups respond to questions about the Electoral College. 

Claim: Standing up at your desk could make you smarter

I have a standing desk in my office. Am I getting smarter as a result? An editorial summarizes some of the evidence.

I'm standing at my desk as I compose this post....could that make my writing go better? Yes, according to an editorial entitled, "Standing up at your desk could make you smarter." The editorial leads with a strong causal claim and then describes three studies, each with a different design. Here's one of the studies:  

A study published last week...showed that sedentary behavior is associated with reduced thickness of the medial temporal lobe, which contains the hippocampus, a brain region that is critical to learning and memory.

The researchers asked a group of 35 healthy people, ages 45 to 70, about their activity levels and the average number of hours each day spent sitting and then scanned their brains with M.R.I. They found that the thickness of their medial temporal lobe was inversely correlated with how sedentary they were; the subjects who reported sitting for longer periods had the thinnest medial temporal lobes.

a) What were the two variables in this study? Were they manipulated or measured? Was this a correlational or experimental study? 

b) The author writes that the study "showed that sedentary behavior is associated with reduced thickness of the medial temporal lobe." Did he use the correct verb? Why or why not?

Here's a second study described in the editorial:

Intriguingly, you don’t even have to move much to enhance cognition; just standing will do the trick. For example, two groups of subjects were asked to complete a test while either sitting or standing [randomly assigned]. The test — called Stroop — measures selective attention. Participants are presented with conflicting stimuli, like the word “green” printed in blue ink, and asked to name the color. Subjects thinking on their feet beat those who sat by a 32-millisecond margin.

c) What are the two variables in this study? Were they manipulated or measured? Was this a correlational or experimental study? 

d) Does this study support the author's claim that "you don't have to move much to enhance cognition; just standing will do the trick"? Why or why not?

e) Bonus: What kind of experiment was being described here? (Posttest only, prettest/posttest, repeated measures, or concurrent measures?) Comment, as well, on the effect size. 

It’s also yet another good argument for getting rid of sitting desks in favor of standing desks for most people. For example, one study assigned a group of 34 high school freshmen to a standing desk for 27 weeks. The researchers found significant improvement in executive function and working memory by the end of the study. 

f) What are the variables in this study? Were they manipulated or measured? 

g) Do you think this study can support a causal claim about standing desks improving executive function and working memory? 

The author added the following statement to the third study on high school freshmen:

True, there was no control group of students using a seated desk, but it’s unlikely that this change was a result of brain maturation, given the short study period.

h) What threat to internal validity has the author identified in this statement?

i)  What do you think of his evaluation of this threat? 

j) Of the three studies presented, which provides the strongest evidence for the claim that "standing up at your desk could make you smarter"? What do you think? On the basis of this evidence, should I keep standing here?