Here's a good example of a bad study. It's related to the COVID pandemic. In this story by NPR journalist Vanessa Romo, you can read about a doctor in Texas who claims to be conducting research on the drug hydroxychloroquine, using COVID patients in a nursing home as participants. The doctor's research includes both design problems and ethical problems.
According to the journalist,
...a doctor at a Texas nursing home started giving the anti-malaria drug hydroxychloroquine to dozens of elderly patients diagnosed with COVID-19 and tracking the outcomes in what he's calling an "observational study".
According to the doctor conducting this study,
"It's actually going well. People are getting better," Armstrong told NPR, adding that after just a handful of days, some of the 39 patients on the medication are showing signs of improvement.
Armstrong admits it is difficult to quantify how much of his elderly patients' improvement is due to the malaria drug or how they would have fared without it. Nor can he explain why other patients are not responding to the tablet doses, though he notes many are only halfway through the five-day cycle.
Design questions
a) Why can't the doctor decide "how they would have fared without it"? Your answer should discuss the study's need for a comparison group.
b) The doctor's study includes one group--all of the patients in the "study" are taking the drug. What kind of comparison group will be needed to conduct a true clinical trial?
c) If you're as far as Chapter 11, you'll recognize the design of this doctor's study. What is it?
Here's another quote from the article:
[The doctor] said he started by screening those he believed would benefit most and added more people each day. He monitored their blood oxygen saturation, temperatures and how well they were breathing. "The people who [I decided to put] on it were getting sicker but were not so sick that they had to go the hospital," Armstrong explained.
d) What design problems, if any, to you see with the doctor selecting only this type of patient for being in his "observational study?" And, what might have happened if he had included more severely ill patients for his study? (How might the results change?)
Ethical questions
According to federal guidelines and the ethical principles of the Belmont Report, when a person claims to be doing "research," they should obtain ethical approval from an IRB. Ethical approval would require such procedures as obtaining informed consent (respect for persons), minimizing harm (beneficence), and selecting participants fairly (justice). Read these next quotes from the story:
He acknowledged that some families were not aware their relatives were put on the drug, saying that "for the most part," he consulted with each nursing home resident prior to giving them on the tablets.
While the "overwhelming majority of them are awake and alert and can actually have a conversation," Armstrong said some suffer from middle stages of dementia. In some cases, he did not discuss prescribing the tablets with anyone at all before doing so. He said it is common for physicians to prescribe new medications to patients without explicit consent from the patient or family members. "It's not required," he said.
e) Review the sections of Chapter 4 on "respect for persons," "beneficence," and "justice," and apply details from these concepts to this example.
f) Do you think study design flaws such as the ones you've noted in a), b) and c) should be considered ethical violations, too? Why or why not?
Safety questions
A potential issue with this doctor's decision concerns the safety of the drug hydroxychloroquine: According to the journalist:
The Food and Drug Administration has not approved the drug for the treatment of COVID-19.
and
[the drug] is known to have serious negative health impacts. That is why so many in the medical community worry about prescribing it without such proof.
g) The safety of the drug is relevant to the ethical principle of beneficence. Can you explain why?
Finally...
Another wider ethical issue (perhaps outside the scope of a research methods course) concerns how the doctor obtained the drug, which has been in short supply:
[The doctor] began making calls to track down a source for the medicine, which is in short supply.That's when his political connections proved useful.
Armstrong, who is a prominent GOP activist, called Republican Lt. Gov. Dan Patrick. He says Patrick reached out to Texas state Sen. Bryan Hughes, also a Republican, who knew someone on the board of the New Jersey-based company Amneal Pharmaceuticals. The company, which makes and distributes the drug, has donated more than a million tablets nationwide, including to the statesof Texas and Louisiana.Two days later, Armstrong had received more than enough medication to begin giving it to patients.
h) Do you think it is reasonable for a doctor to use political connections to obtain medications that are in restricted supply? Why or why not?
Suggested answers to selected questions
a) and b) and c) Because the doctor's "study" has only a group of patients who are taking the drug, he doesn't have a comparison group. In Chapter 11 this is called a one-group, pretest-posttest design, otherwise known as the Really Bad Experiment. The design is flawed. If patients improve on the drug, the doctor will not know whether their improvement was caused by the drug itself, or simply because their bodies were healing or because they were getting excellent nursing care, or even some other reason.
The doctor's behavior in this COVID study reminds me of the Halstead radical mastectomy, which for decades was the dominant treatment for breast cancer in the U.S.. Surgeons at the time were so convinced that this treatment was the best one for breast cancer that they never bothered to compare it to another treatment--they never tested its efficacy. (When a randomized control study was finally conducted in the 1970s, it found that the surgery was no more effective than other, less invasive treatments.)
d) By selecting only relatively less sick patients, the doctor is stacking the deck in favor of finding evidence that the drug is effective. Many of the patients are likely to get better, so the doctor may conclude that the drug has worked. But of course, without a comparison group of equally healthy people who are not getting the drug, we cannot know if the patients would have recovered anyway.
e) f) g) The lack of informed consent--especially for vulnerable patients who have Alzheimer's--is a clear violation of the respect for persons principle. And the use of a risky treatment, one that is known to be dangerous to people with heart conditions, is likely a violation of the beneficence principle. I would argue that when a researcher conducts a flawed study like this, it also a violation of beneficence--the study puts participants at risk, while the benefit of the potential information gained from the research is negligible. We might also argue that the study violates justice: by studying only people at a nursing home, these patients bear the burden of being research participation when anyone can benefit from this research (assuming, of course, that the research were actually well-designed).
Many thanks to Dr. Stephen Chew for pointing me to this example!
Post script: Looking for more examples of bad studies, as described by science writers? Here are two:
Here's a critique of the many problems with another early published study on hydroxychloroquine.
Here's critique of studies supposedly investigating the death rate of COVID-19.
