Everyday Research Methods

Respect for "persons" in animal research

On the Farm Sanctuary, chicken pig, and cow participants can only participate in research if they choose to. Credit: Xalanx/Deposit Photos

Here's a second story on animal ethics for the month of February. In this journalist's report, you can read about the Farm Sanctuary in Watkins Glen, NY, USA. The sanctuary has a "new, in-house research team" that is doing a special kind of research on its animal residents. One thing that makes the research special is its view of the autonomy of its animal participants. Each animal only participates in a study if it chooses to. 

According to the journalist, it's been rare to do research on farm animals: 

Chickens can anticipate the future, goats appear to solicit help from humans, and pigs may pick up on one another’s emotions.

But scientists still know far less about the minds of chickens or cows than they do about those of apes or dogs, said Christian Nawroth, a scientist studying behavior and cognition at the Research Institute for Farm Animal Biology in Germany. “I’m still baffled how little we know about farm animals, given the amount or the numbers that we keep,” he said.

The farm hopes its data will help change people's minds, convincing them that animals have emotions and personalities. Indeed, farm animals are sentient, as argued by the Cambridge Declaration on Consciousness.  At the Farm Sanctuary, research guidelines for their animal work explicitly forbid human dominance and control over them: 

Among other stipulations, the guidelines prohibit invasive procedures — forbidding even blood draws unless they are medically necessary — and state that the studies must benefit the animals. And participation? It’s voluntary.

“Residents must be recognized as persons,” the guidelines state, “and always be provided with choice and control over their participation in an experimental study.”

As an example, the researchers at Farm Sanctuary were studying whether chickens enjoy learning. They created an experimental setting which invited chickens to learn to knock a lid off of a bowl. No chicken was forced to participate in the study; instead, the researchers slowly introduced the setting to the flock, and enticed individual chickens to enter the arena over time with snacks. The journalist reports:

So far, the researchers have tested eight chickens, half of whom were in the control group, and it is too early to draw firm conclusions about chickenkind. (The original group of recruits dwindled after one bird died, another failed to meet the study criteria, and three others dropped out — in one case, to spend time in the nest box instead. “I think she really just was highly motivated to sit on some eggs,” Ms. Prasad-Shreckengast said.)

As this passage describes, waiting for full consent makes the research process slower, but is consistent with the ethical values of these researchers. Another issue with the focus on autonomy concerns generalizability:

 

Personality remains a tricky issue. By limiting their study to chickens who, in essence, raised their wings to volunteer, they may have enrolled an unusually bold group of birds, potentially skewing their results. So the researchers are now administering personality assessments and may try to repeat the study with more birds.

 

In the last part of the article, you can read about the sanctuary's studies on pigs and cows, as well. 

 

Questions:

a) In Chapter 4, you can read about the Three Rs: Researchers are expected to reduce the number of animals used, refine procedures to reduce pain and suffering, and replace animals with computer or cell models. Which of the Three Rs are addressed by the research at Farm Sanctuary? 

 

b)  In what ways do the Farm Sanctuary studies move beyond the minimal considerations required by the Three Rs?

 

c) Are animals in the Farm Sanctuary studies treated the same (ethically-speaking) as human research participants?  Why or why not? You might structure a response around the three core ethical principles of Respect for Persons, Beneficence, and Justice. 

 

Animal ethics: What do we owe lab animals?

Researchers are expected to reduce the number of animals used, refine procedures to reduce pain and suffering, and replace animals with computer or cell models. What about repayment of animal subjects? Photo credit: Aleksei Gorodenkov/Alamy Stock Photo

As you read in Chapter 4, the use of animals in research studies is well-regulated around the world. In the United States, many animals are protected by the Animal Welfare Act, and any federally-funded research facilities must follow the review policies of IACUCs. In Canada, researchers follow guidance of the CCAC (Canadian Council on Animal Care).

In both the U.S. and Canada, animal research is guided the "Three Rs", Replacement, Reduction, and Refinement. You're introduced to these terms in Chapter 4, but here's how journalist Brandon Keim defined them: 

Researchers are encouraged to replace animals when alternatives are available, reduce the number of animals used and refine their use so as to minimize the infliction of pain and suffering.

In India, a government body enforces a "Fourth R," Rehabilitation. This means providing care and rehabilitation to animals after they have finished being in experiments.  Now this article, published in January 2023 in the New York Times discusses a similar idea in the U.S, including how people are expressing a form of gratitude to animal participants. What do humans owe lab animals? 

The journalist tells the story of a person who, after receiving the COVID-19 vaccine, came to appreciate that lab-reared monkeys had played a role in the vaccine's development: 

“I thought, I’m afraid of a stupid needle,” [Ms. Strohacker] said. “And these animals have to deal with this all the time.” She reflected on how her newfound freedom, and quite possibly her health, came at the expense of animals suffering or dying to develop the vaccines.

Merely being grateful for those animals seemed insufficient; Ms. Strohacker wanted to give something tangible in return. A little online research returned the National Anti-Vivisection Society’s sanctuary fund, which supports the care of retired lab animals. She made a small donation. “To give thanks was the very least I could do,” Ms. Strohacker said.

The Times reports that the COVID-19 vaccines probably helped save 20 million lives in their first year. Some ethicists would argue that animal research may be considered ethical when it does have a clear social benefit; however, even then, the animals must be treated in ways that minimize harm. The Times reports on the view that a fourth R should be "Repayment." Here's an example: 

Inklings of the idea of repayment can already be found in the research community, most visibly in laboratories that make arrangements for animals — primarily monkeys and other nonhuman primates — to be retired to sanctuaries. In the case of dogs and companion species, including rats, they are sometimes adopted as pets.

When adoptions are not possible, another option is a memorial. Some research labs provide plaques or public displays of appreciation for animals that have served in their lab.

Another option, especially if an animal was euthanized in the service of research, is to pay it forward: 

If someone receives a cornea or a heart from a pig — or funding to study those procedures — then they might pay for the care of another pig at a farmed animal sanctuary, Dr. Sharp proposed: “You’re going to have animals who stand in for the whole.”

“You’re contributing to the group, even if you’re not contributing to the individual,” Dr. Walker said. “That can be really relevant to the animal case.” For example, research on captive axolotls, a critically-endangered species of salamander, has yielded insights into breast cancer, spina bifida and tissue regeneration; in return, people might support efforts to help wild axolotls now struggling to survive in polluted canals in Mexico City.

Other researchers in the story described how they provide better-than-required food and housing for the rats in their research, providing large, naturalistic enclosures and playing with the rats every day. 

Questions:

a) It costs money to treat animals well during research and to repay them (or others of their species) after they participate.  Who should pay for these expenses? 

b) Two cases of alleged animal mistreatment in research have been in the news lately. One is the case of a beagle-breeding facility that was ordered to be closed down. In the other case, Elon Musk's company, Neuralink, is being accused of causing undue distress, pain, and harm to monkeys in allegedly "shoddy" experiment.  If you choose, you could research these stories and apply concepts such as the Three Rs to what you find. Specifically, how did these cases violate the Three Rs? 

c) Can you think of additional ways, besides those mentioned, to repay the animals that have been involved in psychological and medical research? 

Can a cuddly object or blanket reduce adult anxiety?

Does holding a comfort object actually reduce anxiety in adults? That's an empirical question!  Photo: NDanko/Shutterstock

Apparently, about half of American adults still have a childhood teddy bear, and 40% still sleep with it (we can look into the methodology of this survey in a later post!). Do such comfort objects provide anything beyond sentimental feelings?  What about the products marketed to help calm people or help them feel less lonely--products like weighted blankets, or the robotic, stuffed PARO seal, which was created to engage older adults with dementia? 

Time magazine journalist Jamie Friedlander Serrrano covered a variety of studies on these types of comfort objects. The story described several studies that have experimentally tested the effects of weighted blankets, teddy bears, and the PARO seal. One study featured by the journalist was published in the empirical journal, PLOS One. This study focused on a robotic cushion that was plush and soft, and which also mimicked breathing by inflating and deflating every few seconds.

The journalist explained that the researchers wanted to see if holding this object would help students feel less anxious before a stressful oral math exam. 

Students in the test group held the object—a plush, baby-blue cushion the size of a throw pillow that automatically inflates and deflates, mimicking inhaling and exhaling—for eight minutes before their exam. One control group did a guided breathing meditation instead, while another control group didn’t do anything special. Haynes and her team found that clutching the stuffed breathing cushion reduced anxiety as much as doing the guided meditation.

“This indicated that the cushion could be similarly effective as a breathing meditation for anxiety,” says [scientist] Haynes, who is now completing her postdoctoral fellowship at Saarland University in Germany. “We didn’t give the students in the experiment any guidance about using the cushion. We didn’t tell them to follow it with their breathing or anything—it was just purely the act of holding it as it slowly breathed that eased their anxiety. I think we thought it would help with anxiety, but we were pleasantly surprised that it was as effective as the breathing meditation.”

Lywood—who is currently working on commercializing the breathing cushion through her company Sooothe—believes the findings highlight our innate need for touch, even if the source isn’t human or alive, for that matter. “We take touch for granted,” she says. But because many people have been deprived of it during the pandemic, she points out, “we are sort of rediscovering how valuable it is.”

Questions

1. Using the table below, classify the two main variables mentioned in this study (hint: One variable was manipulated and one was measured)

Variable name (conceptual variable)	How was this variable operationalized?	What are this variable's levels?	Was this a manipulated or measured variable?	Was this an IV or a DV?

2. Was the independent variable manipulated as independent groups or within-groups? 

3. Which of the four basic experimental designs was used: posttest only? pretest-posttest? repeated measures? concurrent measures?  Explain your answer. 

4. Sketch a little bar graph of the results of the anxiety result.

5. The journalist doesn't specify whether people were randomly assigned to the three conditions, and when you check the PLOS One article, it also doesn't mention that people were randomly assigned. I don't think we should assume that researchers did NOT take this step--sometimes they just forget to mention it. In any case, why would you need to randomly assign people to the three conditions? That is, what kinds of alternative explanations does random assignment help rule out?

6. One of the authors of the study also created, and hopes to market, the "breathing" cushion. Is this an ethical problem for the study? Is this a scientific problem for the study? 

Tuskegee Syphilis Study: An anniversary

Four months after Heller's story broke, the Tuskegee study was terminated with this letter.
Credit: U.S. National Archives

In this photo from the 1950s, a man participating in the study had his blood drawn.
Credit: U.S. National Archives

The year 2022 marks the 5oth anniversary of a major news event--journalist Jean Heller's story about the Tuskegee syphilis study. In Chapter 4 you can read about the study, in which poor Black men in Alabama were recruited and purposefully not treated for syphilis. The news story first appeared in the Associated Press (the AP). To honor its 50-year anniversary, the AP re-ran their original news report from July, 1972. 

Here is the lede (the opening passage) from Heller's original story:

For 40 years the U.S. Public Health Service has conducted a study in which human guinea pigs, denied proper medical treatment, have died of syphilis and its side effects.

The study was conducted to determine from autopsies what the disease does to the human body.

PHS officials responsible for initiating the experiment have long since retired. Current PHS officials, who say they have serious doubts about the morality of the study, also say that it is too late to treat syphilis in any of the study’s surviving participants.

In Chapter 4 (Ethics), you can read how the Tuskegee syphilis study led to the Belmont Report, which outlined major changes in how research is evaluated and conducted in the United States. The Tuskegee study violated three distinct ethical principles: Respect for Persons, Beneficence, and Justice. 

a) In your own words, define each of these principles. 

b) Read each of the quoted passages below, from Heller's original story. Decide what ethical problems each one seems to illustrate. Can you map each passage to one of the core ethical principles of the Belmont report (Respect for Persons, Beneficence, or Justice)?

b1) The experiment, called the Tuskegee Study began in 1932 with about 600 black men mostly poor and uneducated, from Tuskegee, Ala., an area that had the highest syphilis rate in the nation at the time.

b2) As incentives to enter the program, the men were promised free transportation to and from hospitals, free hot lunches, free medicine for any disease other than syphilis and free burial after autopsies were performed.

b3) The Tuskegee Study began 10 years before penicillin was discovered to be a cure for syphilis and 15 years before the drug became widely available. Yet, even after penicillin became common, and while its use probably could have helped or saved a number of the experiment subjects, the drug was denied them, Dr. J.D. Millar says.

Don Prince, another official in the venereal disease branch of CDC, said [...] “I don’t know why the decision was made in 1946 not to stop the program,” Prince said. “I was unpleasantly surprised when I first came here and found out about it. It really puzzles me.”

As part of the anniversary, the AP also ran a story-behind-the-story about how Heller researched and broke the story.  Here is Heller reflecting on how she felt as she was breaking the story: 

Normally, reporters celebrate these “Eureka” moments. But Heller felt no such elation.

“I knew that people had died, and I was about to tell the world who they were and what they had,” she says, her voice dropping. “And finding any joy in that ... would have been unseemly.”

Heller also said,

“As much injustice as there was for Black Americans back in 1932, when the study began, I could not BELIEVE that an agency of the federal government, as much of a mistake as it was initially, could let this continue for 40 years,” says Heller. “It just made me furious.”

What were some outcomes of Heller's reporting? 

Nearly four months after the story ran, the study was halted.

The government established the Tuskegee Health Benefit Program to begin treating the men, eventually expanding it to the participants’ wives, widows and children. A class-action lawsuit filed in 1973 resulted in a $10 million settlement.

c) As you reflect on this story, what are you most concerned about? What questions do you still have? 

 

Curbing hate speech with empathy

The study found that empathy-inducing responses like this were more likely to be followed by people deleting their original hateful posts. Photo courtesy of the author

Hate speech abounds on certain corners of social media. Some people post racist, sexist, anti-immigrant, or homophobic comments, and these contribute to a harmful and hateful discourse online. Some companies filter, censor, or censure accounts when they cross the line. Other companies have proved unwilling to ban such content, citing free speech concerns. Recently, some researchers asked if people might engage with such posters online in order to prevent them from posting hate speech. 

The website Optimist Daily ran a story about their work. Here's how they introduced it: 

Led by public policy professor Dominik Hangartner, the research team identified 1,350 English-speaking Twitter users who had posted racist or xenophobic content. The researchers then assigned these users randomly to a control group or one of three test groups. The test groups were then prescribed different counterspeech strategies: empathy-eliciting messages, humor, or warning of consequences.

For example, the empathy eliciting messages might have read "your post is very painful for Jewish people to read...", or "For African-Americans, it really hurts to see people use language like this". Another example can be seen in the comment about immigrants in the photo above, accompanying this post.  

The humorous messages included a meme of an animal with a quip such as "please sir, stop tweeting".

The warning-of-consequences messages read, for example, "Hey, remember that your friends and family can see this tweet too." 

What happened after one of these responses was posted? Here is how the journalist described the results: 

The researchers found that by far, the most effective strategy for getting these users to delete or amend hateful content was counterspeech messages that elicit empathy.

Questions

a) What words in the journalist's story convey that this was an experiment?  I would additionally classify it as a field experiment--can you see why?

b) What was the independent variable? What were its levels? Was the IV manipulated as independent-groups or within groups?

c) There are two dependent variables mentioned in the quotes above. What are they? If you were the researcher, what other DVs would you be interested in? (you can consult the original study to see some of the other DVs)

d) How were the participants in this study selected? Did they use a random sampling method? Or something else? 

e) This study was approved by the researchers' IRB, but the participants in this study apparently did not provide informed consent to participate in the research. Why do you think the IRB approved the study?

Here is a link to the original journal article in PNAS. You can use it to study the graphs of the study's outcome. 

Ethics, justice, and vaccine trials

The journalist writes that Black Americans have been proportionally less willing to participate in vaccine trials, perhaps because researchers in the past have violated their trust. Photo: Andrey_Popo/Shutterstock

At time of writing (late 2020), researchers around the world are racing to test vaccines for the covid-19 virus. To test the efficacy and safety fo these vaccines, studies must recruit thousands of participants and track them over time. They need to know, Do people who get the vaccine have any negative side effects? Do people who get the real vaccine have lower rates of COVID infection compared to people who get a placebo vaccine? 

In this story, New York Times journalist Jan Hoffman explores one legacy of the Tuskegee study: It has led to pervasive distrust of research participation on behalf of African-Americans. This means that Black Americans may not be eager to participate in vaccine trials for covid-19. 

We need to include people of all ethnicities in these clinical trials. Hoffman writes:

It is essential, public health experts say, that research reflect diverse participation not only as a matter of social justice and sound practice but, when the vaccine becomes available, to help persuade Black, Latino and Native American people to actually get it. (The participation of Asian people [in vaccine trials] is close to their share of the population.)

“Historically, we test everything in white men,” said Dr. Michael, a member of the vaccine development team at Operation Warp Speed, the public-private partnership set up by the White House. “But the disease is coming after people of color, and we need to encourage them to volunteer because they have the highest burden of disease.”

Early on in vaccine trials, African-Americans were making up only 3% of the study participants (this group is 15% of the U.S. population). Therefore, recruiters like Carla Arnold were going door to door and meeting with small community groups, trying to recruit African Americans to participate in the trials. Hoffman's article includes this exchange:

...an older woman turned the question back on Carla Arnold, ... who is well-known to people in the Heights:

“Miss Carla, would you feel comfortable allowing them to inject you?”

Ms. Arnold, 62, adjusted her seat to face them down, her eyes no-nonsense above a medical mask.

“They already did,” she replied. 

[...] Ms. Arnold carefully explained to the tenants her decision to participate in the trial for the vaccine being developed by Moderna, a company that has received pledges of $1.5 billion from the Trump administration for its effort.

“I am a proud African-American woman,” she said. “As African-Americans, we always seem to get less out of things that go on. I want us at the forefront of this. I want to make sure that Black people are represented. I’m going by faith that these people won’t do to African-Americans what they did to us in Tuskegee. I’m holding them accountable.”

Questions

a) In Chapter 4 (Ethics) you read about Tuskegee Syphilis study, which violated all three ethical principles of the Belmont Report. One of these is respect for persons. How did the Tuskegee study violate this principle?

b) The second Belmont principle is beneficence. How did the Tuskegee study violate this principle?

c) The third Belmont principle is justice. How did the Tuskegee study violate this principle?

d) Which Belmont principle is most relevant to the composition of the participant pool for a vaccine trial? 

The Tuskegee study isn't the only famous historical example of research that unfairly targeted Black Americans. Hoffman reminds readers of three infamous studies here: 

[examples] include surgeries by Dr. J. Marion Sims, a 19th-century gynecologist, on enslaved Black women, the 40-year-long Tuskegee study, in which doctors deliberately allowed syphilis to progress in Black Alabama sharecroppers, and researchers’ taking of cells without permission from Henrietta Lacks, an African-American cancer patient, in 1951.

e) As Hoffman's article explains, African-Americans' reluctance to participate in vaccine trials probably reflects, in part, an understandable mistrust of researchers, based on these past examples (“'It’s not the science we distrust; it’s the scientists,' said Jamil Bey"). Reflect on this outcome of past unethical practice: Why 

In one sense, it might seem fair that in the current trial, African-Americans are not disproportionately "bearing the burden" of the vaccine trials and its potential risks (that's the justice principle). Nevertheless, by not recruiting African-Americans in the trials, this group risks being left out of its potential benefits, too. It's an issue of ethical fairness and good practice: Access to to the vaccine (even in its trial form) needs to be equal in all communities, both now and in the future. And this is especially true because Black Americans are much more likely to be affected by the virus.

Hoffman's article ends on a positive note:

By the following week, there were signs that the outreach efforts were helping. The portion of people of color in the Pittsburgh area in the vaccine registry had risen to 8 percent, up from 3 percent. Because trial leaders can choose whom they finally enroll, they have been increasing the percentage of nonwhite subjects. Moderna reported that nationwide, as of Sept. 28, 26 percent of those enrolled were Black.

Thank you to Adam Putnam of Furman University for sharing this example with me. 

Teaching with 100 Humans

In 2020, Netflix released an 8-episode series called "100 Humans" in which 100 (American) humans from a variety of ages and social groups participate in a series of studies. It's a gold mine of examples for critical thinking in research methods. You'll find examples of posttest-only, pretest-posttest designs, HARKing, ethics, and more. Most of the studies have flaws, but that makes them more fun to critique. 

I've cataloged ways you can use the series in your teaching: www.tinyurl.com/teachingwith100humans.

Reach out if you use it! Let me know how it works for you. 

 

Testing a COVID drug: Design flaws and ethical lapses

Photo credit: Alissa Eckert, MS; Dan Higgins, MAMS/CDC

Here's a good example of a bad study. It's related to the COVID pandemic. In this story by NPR  journalist Vanessa Romo, you can read about a doctor in Texas who claims to be conducting research on the drug hydroxychloroquine, using COVID patients in a nursing home as participants. The doctor's research includes both design problems and ethical problems.

According to the journalist, 

...a doctor at a Texas nursing home started giving the anti-malaria drug hydroxychloroquine to dozens of elderly patients diagnosed with COVID-19 and tracking the outcomes in what he's calling an "observational study".

According to the doctor conducting this study, 

"It's actually going well. People are getting better," Armstrong told NPR, adding that after just a handful of days, some of the 39 patients on the medication are showing signs of improvement.

Armstrong admits it is difficult to quantify how much of his elderly patients' improvement is due to the malaria drug or how they would have fared without it. Nor can he explain why other patients are not responding to the tablet doses, though he notes many are only halfway through the five-day cycle.

Design questions

a) Why can't the doctor decide "how they would have fared without it"? Your answer should discuss the study's need for a comparison group. 

b) The doctor's study includes one group--all of the patients in the "study" are taking the drug. What kind of comparison group will be needed to conduct a true clinical trial?

c) If you're as far as Chapter 11, you'll recognize the design of this doctor's study. What is it?

Here's another quote from the article:

[The doctor] said he started by screening those he believed would benefit most and added more people each day. He monitored their blood oxygen saturation, temperatures and how well they were breathing. "The people who [I decided to put] on it were getting sicker but were not so sick that they had to go the hospital," Armstrong explained.

d) What design problems, if any, to you see with the doctor selecting only this type of patient for being in his "observational study?" And, what might have happened if he had included more severely ill patients for his study? (How might the results change?)

Ethical questions

According to federal guidelines and the ethical principles of the Belmont Report, when a person claims to be doing "research," they should obtain ethical approval from an IRB. Ethical approval would require such procedures as obtaining informed consent (respect for persons), minimizing harm (beneficence), and selecting participants fairly (justice). Read these next quotes from the story:

He acknowledged that some families were not aware their relatives were put on the drug, saying that "for the most part," he consulted with each nursing home resident prior to giving them on the tablets.

While the "overwhelming majority of them are awake and alert and can actually have a conversation," Armstrong said some suffer from middle stages of dementia. In some cases, he did not discuss prescribing the tablets with anyone at all before doing so. He said it is common for physicians to prescribe new medications to patients without explicit consent from the patient or family members. "It's not required," he said.

e) Review the sections of Chapter 4 on "respect for persons," "beneficence," and "justice," and apply details from these concepts to this example. 

f) Do you think study design flaws such as the ones you've noted in a), b) and c) should be considered ethical violations, too? Why or why not? 

Safety questions

A potential issue with this doctor's decision concerns the safety of the drug hydroxychloroquine: According to the journalist:

The Food and Drug Administration has not approved the drug for the treatment of COVID-19. 
and
[the drug] is known to have serious negative health impacts. That is why so many in the medical community worry about prescribing it without such proof.

g) The safety of the drug is relevant to the ethical principle of beneficence. Can you explain why?

Finally...

Another wider ethical issue (perhaps outside the scope of a research methods course) concerns how the doctor obtained the drug, which has been in short supply:

[The doctor] began making calls to track down a source for the medicine, which is in short supply.That's when his political connections proved useful.

Armstrong, who is a prominent GOP activist, called Republican Lt. Gov. Dan Patrick. He says Patrick reached out to Texas state Sen. Bryan Hughes, also a Republican, who knew someone on the board of the New Jersey-based company Amneal Pharmaceuticals. The company, which makes and distributes the drug, has donated more than a million tablets nationwide, including to the statesof Texas and Louisiana.Two days later, Armstrong had received more than enough medication to begin giving it to patients. 

h) Do you think it is reasonable for a doctor to use political connections to obtain medications that are in restricted supply? Why or why not? 

Suggested answers to selected questions

a) and b) and c) Because the doctor's "study" has only a group of patients who are taking the drug, he doesn't have a comparison group. In Chapter 11 this is called a one-group, pretest-posttest design, otherwise known as the Really Bad Experiment. The design is flawed. If patients improve on the drug, the doctor will not know whether their improvement was caused by the drug itself, or simply because their bodies were healing or because they were getting excellent nursing care, or even some other reason. 

The doctor's behavior in this COVID study reminds me of the Halstead radical mastectomy, which for decades was the dominant treatment for breast cancer in the U.S.. Surgeons at the time were so convinced that this treatment was the best one for breast cancer that they never bothered to compare it to another treatment--they never tested its efficacy. (When a randomized control study was finally conducted in the 1970s, it found that the surgery was no more effective than other, less invasive treatments.)

d) By selecting only relatively less sick patients, the doctor is stacking the deck in favor of finding evidence that the drug is effective. Many of the patients are likely to get better, so the doctor may conclude that the drug has worked. But of course, without a comparison group of equally healthy people who are not getting the drug, we cannot know if the patients would have recovered anyway.

e) f) g) The lack of informed consent--especially for vulnerable patients who have Alzheimer's--is a clear violation of the respect for persons principle. And the use of a risky treatment, one that is known to be dangerous to people with heart conditions, is likely a violation of the beneficence principle. I would argue that when a researcher conducts a flawed study like this, it also a violation of beneficence--the study puts participants at risk, while the benefit of the potential information gained from the research is negligible. We might also argue that the study violates justice: by studying only people at a nursing home, these patients bear the burden of being research participation when anyone can benefit from this research (assuming, of course, that the research were actually well-designed).

Many thanks to Dr. Stephen Chew for pointing me to this example!

Post script: Looking for more examples of bad studies, as described by science writers? Here are two:

Here's a critique of the many problems with another early published study on hydroxychloroquine.

Here's critique of studies supposedly investigating the death rate of COVID-19. 

Temperature data as quasi-experiment

Which government-led social distancing measures are effective at stopping the spread of the novel Coronovirus?  One company has contributed data to answering this question. A private health technology company, Kinsa sells a "smart thermometer" which links to an app that guides people through what to do if they have a fever. When people use the app, they agree to have their temperature information (including fever and other symptoms they report) shared with Kinsa. 

Kinsa has used body temperature data points from all over the U.S. to create maps showing areas of "atypical illness." They define atypical illness on their website as "an unusual incidence of elevated flu-like illness levels."

It's really interesting to explore Kinsa's atypical illness maps of the U.S. here. 

More relevant for a research-methods course are the figures that Kinsa presents that map "flu-like illness" rates over time in certain high-population counties (Boston, New York, Los Angeles). I've pasted two of them here. Take a look at these figures, and then answer the questions below. 

Image courtesy of Kinsa, Inc.

 

Image courtesy of Kinsa, Inc

 

a) These figures should remind you of one of the quasi-experimental designs covered in Chapter 13. Which design is it?

b) The Kinsa company has used these charts to argue that social distancing measures work to reduce cases of COVID-19. Journalists (for example) have used the charts to specifically argue that limiting large gatherings and closing schools alone does not do much to stop the spread; it was only when bars and restaurants were shuttered that these counties saw reduced rates of atypical illness. Look carefully at the three counties above. What parts of the data are relevant to this argument? Do you think the data support the claim that closing bars and restaurants was effective? Can you think of alternative explanations for the pattern, other than "closing bars and restaurants"?

c)  Finally, consider the ethics and scientific openness principles. Do you think it's ethical for Kinsa to use and publicize the private health information of people who buy its products? Consider arguments both for and against the scientific ethics of this practice. Frame your arguments around the ethical principles of respect for persons, beneficence, and justice. 

In your reflections, you might consider that users of Kinsa's products agree to data sharing via its terms and conditions. You might consider that the data from users is kept anonymous and is aggregated over more than a million users. You might consider the public health benefits of sharing the data. And you might also consider the extent to which Kinsa is following open science practices--that is, sharing all the data the way most university researchers now do (relevant Kinsa research information here). 

d) If you're construct-validity-curious, you might be wondering if Kinsa's definition of flu-like illness actually tracks COVID-19 cases. Kinsa do not claim to be diagnosing actual COVID (you need a nasal swab for that). However, they present some maps showing how their temperature data has tracked with confirmed COVID cases in several counties. The data are in the middle of this webpage.  

By the way, here's another example of a company that collects our health data, using it to share national patterns. This one is Fitbit, sharing how sleep time has increased (in most age groups) since the COVID quarantine started.

 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers.