Everyday Research Methods

Observational research: What are parents doing?

Several press sources are covering a research article published in the journal Pediatrics, which observed parenting behaviors in naturalistic settings.

Here's how the story was covered by ABCnews.

The researchers visited a sample of fast food restaurants in the Boston area, found tables in which parents were sitting with kids, and made notes of the parents' and children's behavior. One of the major findings was the degree to which the parents were engaged by their mobile devices:

Parents in 40 of the 55 families observed were absorbed in their mobile devices, according to the study.

a) What kind of claim is the journalist apparently making above? What is/are the variable(s) in the claim?

b) How would you ask about the external validity of the study?  

The researchers observed both parents and children: 

...  almost a third of the parents used their devices continuously throughout their meal....Some children appeared unaffected and ate their meals in silence. Other children were more provocative, with one set of siblings singing “Jingle bells, Batman smells” to get their dad’s attention.

 c) How would you ask about the construct validity of these measures of parental and child behavior? 

 There is an inconsistency in the journalist's story. Although the headline of the study makes an association claim, "Parents on smartphones ignore their kids, study finds," the body of the story reports that:

The degree to which the device was used, however, did not necessarily directly relate to the way in which the child reacted, according to the study.

Indeed, if you read the original journal article in Pediatrics, you'll see that the real study did not report much evidence of associations between parental behavior and child behavior.

 

Suggested answers

a) This line about "40 out of 55 families" is  a frequency claim--it seems to simply be documenting "what parents do"  when eating in restaurants with children. 

b) You'd want to ask "how did they choose these 55 families to observe? Did they use a random sampling/probability sampling technique?"  If they sampled families based on convenience, it would be difficult to infer that the same proportion of all American families show these cell phone habits.

You might also ask about the degree to which the setting generalizes. Does parental behavior in restaurants generalize to their behavior at home, or to other settings where they are with the kids?

c) To ask about construct validity, you'd want to ask about how well the coders were trained. Did the coders have good inter-rater reliability? Did they follow a codebook? What behaviors did they decide to code and how well did they code them? 

 

Thanks to my colleague Agnes Ly for this story idea!

 

Stop using cash, reduce crime?

The Washington Post covered a criminal justice story recently that tracked what happened to crime rates after communities changed the way welfare benefits are distributed. 

Previously, people received checks for public assistance. After they cashed these checks, they could have become targets of crime. Since the 1990's, however, public assistance benefits have gradually changed, and are now distributed via electronic debit cards (EBT cards). 

Did the shift to EBT cards cause a change in crime rate? Let's check it out. 

According to a team of researchers at the University of Missouri, the overall crime rate dropped 10 percent after the introduction of EBT cards for welfare. 

The graphic [in the story] looks at crime trends in Missouri before and after the switch from cash to debit cards, for all crimes and broken down by individual crimes. In most crime categories the change before and after the switch is striking - upward trends in assault, burglary, car theft and robbery are completely reversed.

a) What kind of quasi-experimental study is this? (Hint--looking at the graphs in the story will help you answer this question!)
Options: 
non-equivalent control groups pretest-posttest design
non-equivalent control groups posttest only design
interrupted time series design
non-equivalent control groups interrupted time series design

 

The journalist explains the following aspect of the results:

To put these results in perspective, the overall 10 percent decrease in crime corresponded to 47 fewer crimes per 100,000 people per county per month as a direct result of switching welfare benefits from cash to credit. This finding is fairly astonishing.... 

b) which big validity is the author addressing in the quote above?  

Here's another detail presented in the story. Rates of assault, burglary, car theft, larceny, and robbery all dropped after EBT cards, but rates of rape did not change. According to the authors, this result fits the theory that the availability of cash leads to less crime, because rapes are not typically about getting cash. 

c) Can the authors support the causal claim implied in the article's title: "To fight crime in your community, stop using cash"?

 

 

Suggested answers

a) this is an interrupted time-series design--it measured crime rates repeatedly up until some key event (the introduction of EBT cards), and repeatedly afterwards. 

b) By discussing the magnitude of the 10% increase in practical terms (47 fewer crimes per 100,000 people), the author is discussing one aspect of statistical validity (effect size). 

c) Let's apply the three causal criteria to this claim. There is definitely covariance--periods of time with EBT cards go with a lower crime rate, and periods of time with cash-based benefits go with a higher crime rate.

There is also temporal precedence. EBT cards came first, and crime rates were recorded afterwards.  

What about internal validity? We could work through the 12 major threats to internal validity here. The design and results of this study allow us to rule out selection, regression, maturation, and a variety of other internal validity threats. However, one possible culprit is a history threat--what else might have happened in the 1990's --at the same time that EBT cards were introduced--that might have also explained this drop in all crimes except for rape? Perhaps a change in policing strategy, political structure, or other crime policy occurred. By reading the scientists' original article, you might be able to check out how well the researchers addressed such history threats.  

 

Thanks to Carrie Smith for this study suggestion!