There's a great example of a quasi-experiment in the news. The research question concerned the skills of surgeons. As the story points out, surgeons are similar to musicians, in that their manual dexterity skills develop with constant use and practice. The study asked, Would surgeons lose their skills at surgery after a few days off?
This summary of the research was reported by NPR's Shankar Vedantam. The research was conducted by Lorens Helmchen and Jason Hockenberry of George Mason University. They analyzed hospital records of about 56,000 surgery patients. As Vedantam explains:
...the researchers compared the outcomes of patients in two different groups. In the first group, the patient's surgeon had performed surgery on other patients the previous day. In the second group, the patient's surgeon had not performed surgery the previous day.
The previous day's lack of surgery might have been a weekend day, a vacation day, or even an office visit day with no surgeries.
Now there are three possible outcomes. One, that the surgeons are so skillful that the break makes no difference whatsoever. The second possibility -which is what I would have picked - is that the surgeons are actually going to be better when they come back from vacation because they're going to be refreshed.
...unfortunately, [the data supported] the third hypothesis.... The outcomes were worst when the doctors had not practiced surgery the previous day.
a) Which of the four quasi-experimental designs from Ch 13 does this study appear to follow? The non-equivalent control group post-test only design? The non-equivalent control group prettest-posttest design? The interrupted time-series design ? Or the non-equivalent control group interrupted time-series design?
b) Sketch a graph of this outcome.
The story on NPR has some additional information about the study:
VEDANTAM: I want to be very clear, the difference is very small. In fact, it was so small that an individual doctor or even an individual hospital probably would not notice the difference. And I asked Helmchen to describe the size of the effect that he found.
HELMCHEN: If you take 100,000 heart bypass surgery patients, of those about 2700 die before they just charged from the hospital. Our study suggests that every additional day that the surgeon was away from the operating room increases that number by an additional 70 patients.
(NPR host INSKEEP's response): So if they had a day off, there's a tiny difference. If they had a couple days off, there's a little bigger difference. If they had a two-week vacation or a month-long sabbatical, there's a big difference.
c) What kind of validity is being discussed in the segment above? Does this conversation change how you drew your sketch in part b?
Finally, the NPR story discussed a number of explanations for these findings. The first two explanations appear to be mediators. That is, they are reasons why vacation days might lead to worse outcomes for patients. Here they are:
It could ...be that the surgeons are doing fine during surgery, but they're missing potential complications. Helmchen and Hockenberry find that when surgeons come back from a break hospital costs go down. So it could be surgeons are ordering fewer tests or not thinking about very rare risks. There's another possibility, which is it might have to do with the team surrounding the surgeon and on the first day back, the team is still sort of getting its act together or getting its edge together and they're not quite as good as they were when they've had several days of practice.
d) Sketch one or both of these mediator explanations, following the models in Figure 9.13 (p. 260).
The other explanation that the story mentioned for this finding is that there was a third variable problem, or an internal validity problem:
There's a final explanation, and this is completely innocuous, which is the hospitals are lining up the sickest patients when the surgeons come back first, but because they're so sick they're more likely to die.
e) What is the third variable here? Can you sketch it according to the models for third variable problem in Figure 9.13?
f) You knew it was coming: The headline of this story reads, "Study: Time away can hurt surgeons' job performance." The verb, "Hurt" is a causal one. Can the study really support NPR's causal-claim headline?
Suggested answers:
a) I think that the best option here is to call this quasi-experiment a non-equivalent control groups, posttest only design. Patients are non-randomly assigned to surgeons who have either just had a vacation day, or who have not.
b) Given the design, a simple bar graph might suffice here. The x-axis would have "surgeons who just had a vacation day" and "surgeons who did surgery the day before." The y-axis would have "Quality of patient outcomes." The bar for the "surgeons who just had a vacation day" would be lower in height.
c) This section is talking about the effect size of the result, which is part of statistical validity. This means that the two bars you drew in the graph should be different, but not greatly.
By the way, this might be an example of a small effect size that, nonetheless, has large practical implications. The researcher himself mentioned that the vacation day effect adds 70 deaths to the usual number of 2700 deaths out of 100,000 heart bypass patients. That's a small effect, but a large number of lives.
d) You could sketch the first mediator explanation like this:
Surgeon vacation day ---> surgeon not thinking about rare risks and not ordering tests --> worse patient outcomes
You could sketch the second mediator explanation like this:
Surgeon vacation day --> surgical team re-learning to work together --> worse patient outcomes
e) You could sketch this internal validity, or third-variable problem, like so:
Sicker patients ---> worse outcomes
and Sicker patients ---> surgeons returning after a break
f) Of course, we can't determine causation from a quasi-experiment.
There is covariance (surgeons returning after a day off of surgery had worse patient outcomes). And there is temporal precedence (the day off came before the patient outcome).
But as noted in question e), patients were not randomly assigned to surgeon days. Therefore, we can't rule out third variables such as patient health in this association. The study does not meet the internal validity criterion. NPR should have used the wimpier association headline: "After a surgeon's day off, surgery patients' outcomes are worse."
