Here's a post for producers of research (especially students who are designing their own surveys).
I'm a fan of the Pew Research Center's polling website, in part because they are transparent about their methods, and in part because they share educational content about how to do good surveys. For example, they have videos on random sampling and question wording that I've blogged about here before; both are worth watching.
A new example of Pew's educational content is this recent piece, called "How we keep our online surveys from running too long".
This article is referring to panel research, in which people are randomly sampled from a national address database, provided tablets and internet access if they need it, and agree to complete surveys every so often, often for several years. Pew's panel is the American Trends Panel (ATP). You can read more about panel research--and its advantages over telephone-only surveys--in Chapter 7 .
A concern for panel research is the length of a survey. You can't make your survey too long, because people will stop responding. Pew strives to keep their ATP surveys at 15 minutes or less. Here's the overview of how they do it:
In a nutshell, we classify survey questions based on their format. Each question format has a “point” value reflecting how long it usually takes people to answer. Formats that people tend to answer quickly (e.g., a single item in a larger battery) have a lower point value than questions that require more time (e.g., an open-ended question where respondents are asked to write in their own answers). A 15-minute ATP survey is budgeted at 85 points, so before the survey begins, researchers sum up all of the question point values to make sure the total is 85 or less.
There are different point values for different types of questions. You can see examples of each format in their story (here). But here's the overall approach:
| Question format | Assigned points |
| Stand-alone question | 1 |
| Batter item (multiple, similar questions) | 0.67 |
| Open-ended question | 5 or 8, depending on the requested length |
| Check-all that apply question | 2 |
| Vignette | 1 point per 50 words |
| Thermometer rating | 1.5 |
Being a data-driven company, Pew has stored information on how long each type of question takes--that's the basis for this point system. They report:
When administering an online survey, researchers can see how long it takes each respondent to answer the questions on each screen. ... more challenging open-ended questions can take people minutes to answer, while an ordinary stand-alone question takes only about 10 seconds or so.
So...if you are a student developing your own survey for a research project, you can probably use these estimates, too!
