Collecting accurate data in a poll is difficult business. Many of us focus on the sampling strategies of polling organizations, and rightly so: External validity depends on whether we include "cell phone only" numbers in our samples, how we account for different rates of responding across certain groups, and so on. However, this post reminds us that question wording also matters. Construct validity is just as important for polls.
In this report, Pew Research Center describes how people's opinions change depending on how a question is posed. The poll asked Americans whether we should increase the size of the House of Representatives. People in the poll were randomly assigned to hear either the original question, or the same question with additional context. Notice how people's support changes when additional context is added:
a) Even though this is a polling example, Pew conducted an experiment. What are its independent and dependent variables?
b) What causal claim could this polling result support?
c) In your opinion, which polling question--the original one or the one with added context--provides the most accurate measure of people's opinion?
d) Find another poll question on the Pew Research web site. Speculate how some additional information might change people's responses.
Below is another presentation of the same results. Notice how the two different question wordings seem to affect people differently, depending on their political affiliation.
e) Why might Democrats be more influenced by the contextual information provided?
f) This could be viewed as a 2 x 2, IV x PV factorial design, in which the IV is "question wording" (original or added context) and the PV is "political affiliation" (Democrat or Republican). If you've studied Chapter 12, perhaps you can identify whether there are main effects and interactions in this pattern of data.
The same Pew report contains additional examples of how question wording alters responses to a question about the Senate, and shows how different subgroups respond to questions about the Electoral College.