Everyday Research Methods

Producer focus: Estimating how long your survey will take

How long will your survey take? Photo: MarutStudio/Shutterstock

Here's a post for producers of research (especially students who are designing their own surveys). 

I'm a fan of the Pew Research Center's polling website, in part because they are transparent about their methods, and in part because they share educational content about how to do good surveys. For example, they have videos on random sampling and question wording that I've blogged about here before; both are worth watching.

A new example of Pew's educational content is this recent piece, called "How we keep our online surveys from running too long".

This article is referring to panel research, in which people are randomly sampled from a national address database, provided tablets and internet access if they need it, and agree to complete surveys every so often, often for several years. Pew's panel is the American Trends Panel (ATP). You can read more about panel research--and its advantages over telephone-only surveys--in Chapter 7 .

A concern for panel research is the length of a survey. You can't make your survey too long, because people will stop responding. Pew strives to keep their ATP surveys at 15 minutes or less. Here's the overview of how they do it:

In a nutshell, we classify survey questions based on their format. Each question format has a “point” value reflecting how long it usually takes people to answer. Formats that people tend to answer quickly (e.g., a single item in a larger battery) have a lower point value than questions that require more time (e.g., an open-ended question where respondents are asked to write in their own answers). A 15-minute ATP survey is budgeted at 85 points, so before the survey begins, researchers sum up all of the question point values to make sure the total is 85 or less.

There are different point values for different types of questions. You can see examples of each format in their story (here). But here's the overall approach:

Question format	Assigned points
Stand-alone question	1
Batter item (multiple, similar questions)	0.67
Open-ended question	5 or 8, depending on the requested length
Check-all that apply question	2
Vignette	1 point per 50 words
Thermometer rating	1.5

Being a data-driven company, Pew has stored information on how long each type of question takes--that's the basis for this point system. They report:

When administering an online survey, researchers can see how long it takes each respondent to answer the questions on each screen. ... more challenging open-ended questions can take people minutes to answer, while an ordinary stand-alone question takes only about 10 seconds or so.

So...if you are a student developing your own survey for a research project, you can probably use these estimates, too!  

 

Really? "Sleep this way to add 5 years to your life"

A friendly reminder that when all the variables are measured, causal claims aren't supportable. Photo credit: Sleeping: MicrostockAsia/Deposit Photos

CNN's headline was "sleep this way to add 5 years to your life." This headline should immediately kick in your "correlation is not causation" spidey-sense.

a) What makes this headline causal?

b) What are the two variables in the headline? 

c) Why might you suspect this headline is based on a correlational study (rather than an experiment)?

Now that these preliminaries are out of the way, let's find out if the study behind the headline  can support causation.

Here's the splashy intro:

Want to live longer? Then prioritize sleep in your life: Following five good sleep habits added nearly five years to a man's life expectancy and almost 2.5 years to a woman's life, a new study found.

The article then provides advice such as getting at least 7 hours of sleep, and so on. The journalist links that advice to the following research summary:

"Recent studies have shown irregularity in sleep timing and duration have been linked to metabolic abnormalities ...," he said. "Encouraging maintenance of regular sleep schedules with consistent sleep durations may be an important part of lifestyle recommendations for the prevention of heart disease."

d) In the example above--What are the two variables in the study? What makes the study correlational? 

e) Apply the three causal criteria to the example in d). The study can support covariance (why?). What about temporal precedence? What about internal validity? What specific third variables might be responsible for the link between sleep timing and metabolic abnormalities?

 

Later in the piece, CNN shares data from another study, this one using multiple regression:

The preliminary study, presented Thursday at an annual meeting of the American College of Cardiology, analyzed data from over 172,000 people who answered sleep questionnaires between 2013 and 2018 as part of the National Health Interview Survey. The annual survey is done by the CDC and the National Center for Health Statistics.

 

Each of the five healthy sleep habits — falling asleep easily, staying asleep, getting seven to eight hours of zzz's, waking up rested and foregoing sleep meds — was assigned a number. People were scored on how many of the five habits they had.

 

About four years later, researchers compared those scores with National Death Index records to see if their sleep behaviors contributed to an early death from certain diseases or any cause.

 

The team then factored out other potential causes for a higher risk of dying, such as alcohol consumption, lower socioeconomic status and existing medical conditions.

 

f) Make a table of the variables in this study:

Variable name	How was this variable operationalized?	What are the variable's possible levels?	Is this manipulated or measured?
			(add rows as needed)

 

g) The study directly above meets both covariance and temporal precedence....can you explain why? 

h) Imagine that a critic says, about the study directly above, "Hmmm....it could be that healthy sleep habits predicted early death because of alcohol consumption--maybe people who drink more alcohol are more likely to have unhealthy sleep, and they are also more likely to die early." What do you say? 

 

Finally, if you're looking for an example of a moderator, we've got you:

 

Men who followed all five of the healthy sleep habits had a life expectancy that was 4.7 years greater than people who had none or only one of the five elements of low-risk sleep, the study found.

 

The impact of healthy sleep habits was much lower for women: Those who followed all five sleep habits gained 2.4 years compared with those who did none or only one.

 

i) Put this into a moderator sentence: _____ moderates the relationship between _____ and _____ such that ____.

 

Thanks Stephen Chew for sharing yet another great example from the "correlation is not causation" files.

 

Selected answers

a) The headline is causal because it takes the form of advice: If you do X, it will cause Y (here, If you sleep better, it will cause increased life span)

b) Type of sleep and lifespan are the two variables.

c) You might suspect this headline is based on a correlational study (rather than an experiment) because lifespan is definitely a measured variable (you can't manipulate it), and sleep seems likely to be measured as well. Of course, it's possible to randomly assign volunteers to one of two sleep patterns---you'll want to read the article closely to see!

d) One variable is irregularity in sleep timing and duration; the other is metabolic abnormalities. 

e) The study can support covariance because the results showed a correlation between sleep irregularity and metabolic abnormalities. The study may not have temporal precedence--it's not clear if the variables were measured at the same time, or if sleep irregularity was actually measured first. What about internal validity? It's possible that some pre-existing medical condition (such as depression or addiction) goes with both sleep irregularities and with metabolic abnormalities.  Other third variables might work, too--make sure you explain how the variable you are thinking of links to both sleep irregularities and with metabolic abnormalities.

f)

Variable name	How was this variable operationalized?	What are the variable's possible levels?	Is this manipulated or measured?
Healthiness of sleep habits	Survey; summed five habits (falling asleep easily, staying asleep, etc.)	0 to 5	Measured
Early death	National Death Index records	Alive or dead	Measured
Alcohol consumption	Doesn't say how this was operationalized	Not clear--lo to hi	Measured
SES	Doesn't say how this was operationalized	Not clear--lo to hi	Measured

 

g) It meets covariance because the results show an association between healthiness of sleep habits and early death. It meets temporal precedence because the sleep habits were measured first, and then people were followed for four years, when early death was measured. 

h) This argument doesn't work as a third variable critique because the researchers controlled for alcohol consumption in their multiple regression (The CNN article said, "The team then factored out other potential causes for a higher risk of dying, such as alcohol consumption, lower socioeconomic status and existing medical conditions").

 

i)  Gender moderates the relationship between sleep quality and early death such that the relationship is present for both, but is stronger for men than for women.