Everyday Research Methods

Evidence-based tips for well-being during a pandemic

Dr. Santos provides evidence-based ways to enhance well-being during this pandemic. Credit: Yale University

How can people enhance their wellbeing during the Coronovirus pandemic? 

Dr. Laurie Santos (pictured) has been meeting with several news outlets, sharing evidence-based tips from her psychology of well-being class at Yale University. A version of the course is also offered on Coursera.

In the Twitter-embedded video here, produced by the World Economic Forum, Dr. Santos covers five tips for coping during the pandemic. In this space, I'm going to quote moments from the video, and you can practice classifying the claims and describing how the studies would have been conducted. The first three of these tips are also covered in this story from the World Economic Forum.

Tip 1: Keep social, even when socially distancing

Dr. Santos states in her segment that "Happy people spend more time with other people in general. They also prioritize time with their friends and family members"

a) What kind of claim is it to say that "happy people spend more time with other people in general"--frequency, association, or cause?  What are the variables in this claim? 

b) What kind of study was likely conducted to support this claim--correlational or experimental? 

c) Sketch a scatterplot with well-labeled axes that could depict this result.

Tip 2: Help others

To explain this tip, Dr. Santos mentions, "The research shows that doing random acts of kindness...can be incredibly powerful." For example, happy people are giving money to charity and volunteering in their communities. 

d) What kind of claim is it to say that "doing random acts of kindness...can make you happy"--frequency, association, or cause?  What are the variables in this claim? 

e) What kind of study would need to be conducted to support this claim--correlational or experimental? 

f) Can you imagine how you  might manipulate the variable, "doing random acts of kindness?"

Tip 3: Move--even if you can't leave your house

In this section, she says, "Exercise and moving your body is really correlated with improved well-being"

g) The verb, "correlated" gives this one away: It's an association claim. Dr. Santos specifies that they are "really correlated," which seems to suggest a strong effect size. What factors do we consider when we're trying to decide if a correlation (an effect size) is strong versus weak?

Tip 4: Savor everyday pleasures

"Research suggests that happy people tend to be more mindful. They tend to savor the present moment; they tend to be in the present moment"  Here Dr. Santos also strongly recommends taking deep breaths to calm the vagus nerve and connect to the present. 

h) How might you design an experiment to test the hypothesis that deep breathing can improve people's well-being during quarantine? Whom might you recruit for such a study? How would you operationalize each of the two variables? What results would you expect to find? 

Tip 5: Think about how you want to look back on this time

Finally, she shares one of my favorite tips for emotional regulation, psychological distancing: "Psychological distancing is the idea that if we think about ourselves and how we will have dealt with this crisis in the future...Think about how you want to tell your grandkids you lived through COVID-19, you don't want to be like, 'I panicked-bought and hoarded toilet paper..no, you want the story to be positive...'"

Experimental research has supported the causal claim that self-distancing--seeing yourself as though you are a fly on the wall, or even imagining looking back on yourself from the future--is associated with improved emotional regulation. It helps us take a cooler, less impulsive view of our emotions and process them instead of ruminating. For example, self-distancing helps people manage feelings of anger in constructive ways (here's a nice summary). 

i) Review the term translational research in your textbook (Ch 1) and explain how we might conduct translational research on psychological distancing during the pandemic. What is the basic research it is based on? What is the real-world context it's being translated to? And what kind of study could translate--that is, bridge the gap--between the two? 

 

 

 

 

 

 

Random sampling in action for public health

Why is random sampling important when trying to estimate antibody levels in the population? Photo: © Eugene Lu/Shutterstock

Here is a timely example of random sampling in action.

Watch this 2-minute video about the CDC's project in Atlanta (produced by NPR). It explains how nurses are visiting about 500 people in the Atlanta metropolitan area to see how widespread the Coronoavirus is.  In the first minute of the video, the narrator reports,

The CDC is sending out employees to conduct antibody tests on a random sample of Atlanta residents. The tests are meant to show how many people have been infected with the Coronavirus.

The CDC wants to get blood samples from 450 households. 

(Note: And here's another story of a similar effort, this time in Corvallis, Oregon, USA. It's about prevalence of live cases, rather than antibodies, but a random sample is still a key player.)

Questions:

a) Why would it be important to conduct antibody tests on a random sample of people from the Atlanta area?

b) The plan is to collect samples from 450 households. Imagine that a friend hears this figure and expresses skepticism--"how can 450 be enough to generalize to the population of Atlanta?"  Explain to your friend why a random sample--even if it has "only" 450 households--can generalize to the population.

c) What are some ways the CDC might conduct this random sample? Review the possibilities in Chapter 7 and explain:

i) how might they obtain a simple random sample of Atlanta residents?
ii) how might they obtain a cluster sample (what could they use as clusters?)
iii) how might they conduct a stratified random sample? (Imagine that they want to use neighborhoods/zipcodes as the strata.)

 d) After conducting the antibody study on a random sample of Atlanta residents, the CDC will have an estimate of the rate of infection in the sample. To whom will they be able to generalize this rate of infection--that is, to which population of interest?

e) What might happen if some of the randomly sampled people in Atlanta refuse to have their blood drawn for this CDC study? Will that affect the sample's generalizability?  Why or why  not?

f) Which big validity is this example concerning? 

Many thanks to Dr. Jess Harnett (of the notawfulandboring blog), who posted this example, and thanks to Kathy Becker-Blease for sharing the Oregon example. 

Testing a COVID drug: Design flaws and ethical lapses

Photo credit: Alissa Eckert, MS; Dan Higgins, MAMS/CDC

Here's a good example of a bad study. It's related to the COVID pandemic. In this story by NPR  journalist Vanessa Romo, you can read about a doctor in Texas who claims to be conducting research on the drug hydroxychloroquine, using COVID patients in a nursing home as participants. The doctor's research includes both design problems and ethical problems.

According to the journalist, 

...a doctor at a Texas nursing home started giving the anti-malaria drug hydroxychloroquine to dozens of elderly patients diagnosed with COVID-19 and tracking the outcomes in what he's calling an "observational study".

According to the doctor conducting this study, 

"It's actually going well. People are getting better," Armstrong told NPR, adding that after just a handful of days, some of the 39 patients on the medication are showing signs of improvement.

Armstrong admits it is difficult to quantify how much of his elderly patients' improvement is due to the malaria drug or how they would have fared without it. Nor can he explain why other patients are not responding to the tablet doses, though he notes many are only halfway through the five-day cycle.

Design questions

a) Why can't the doctor decide "how they would have fared without it"? Your answer should discuss the study's need for a comparison group. 

b) The doctor's study includes one group--all of the patients in the "study" are taking the drug. What kind of comparison group will be needed to conduct a true clinical trial?

c) If you're as far as Chapter 11, you'll recognize the design of this doctor's study. What is it?

Here's another quote from the article:

[The doctor] said he started by screening those he believed would benefit most and added more people each day. He monitored their blood oxygen saturation, temperatures and how well they were breathing. "The people who [I decided to put] on it were getting sicker but were not so sick that they had to go the hospital," Armstrong explained.

d) What design problems, if any, to you see with the doctor selecting only this type of patient for being in his "observational study?" And, what might have happened if he had included more severely ill patients for his study? (How might the results change?)

Ethical questions

According to federal guidelines and the ethical principles of the Belmont Report, when a person claims to be doing "research," they should obtain ethical approval from an IRB. Ethical approval would require such procedures as obtaining informed consent (respect for persons), minimizing harm (beneficence), and selecting participants fairly (justice). Read these next quotes from the story:

He acknowledged that some families were not aware their relatives were put on the drug, saying that "for the most part," he consulted with each nursing home resident prior to giving them on the tablets.

While the "overwhelming majority of them are awake and alert and can actually have a conversation," Armstrong said some suffer from middle stages of dementia. In some cases, he did not discuss prescribing the tablets with anyone at all before doing so. He said it is common for physicians to prescribe new medications to patients without explicit consent from the patient or family members. "It's not required," he said.

e) Review the sections of Chapter 4 on "respect for persons," "beneficence," and "justice," and apply details from these concepts to this example. 

f) Do you think study design flaws such as the ones you've noted in a), b) and c) should be considered ethical violations, too? Why or why not? 

Safety questions

A potential issue with this doctor's decision concerns the safety of the drug hydroxychloroquine: According to the journalist:

The Food and Drug Administration has not approved the drug for the treatment of COVID-19. 
and
[the drug] is known to have serious negative health impacts. That is why so many in the medical community worry about prescribing it without such proof.

g) The safety of the drug is relevant to the ethical principle of beneficence. Can you explain why?

Finally...

Another wider ethical issue (perhaps outside the scope of a research methods course) concerns how the doctor obtained the drug, which has been in short supply:

[The doctor] began making calls to track down a source for the medicine, which is in short supply.That's when his political connections proved useful.

Armstrong, who is a prominent GOP activist, called Republican Lt. Gov. Dan Patrick. He says Patrick reached out to Texas state Sen. Bryan Hughes, also a Republican, who knew someone on the board of the New Jersey-based company Amneal Pharmaceuticals. The company, which makes and distributes the drug, has donated more than a million tablets nationwide, including to the statesof Texas and Louisiana.Two days later, Armstrong had received more than enough medication to begin giving it to patients. 

h) Do you think it is reasonable for a doctor to use political connections to obtain medications that are in restricted supply? Why or why not? 

Suggested answers to selected questions

a) and b) and c) Because the doctor's "study" has only a group of patients who are taking the drug, he doesn't have a comparison group. In Chapter 11 this is called a one-group, pretest-posttest design, otherwise known as the Really Bad Experiment. The design is flawed. If patients improve on the drug, the doctor will not know whether their improvement was caused by the drug itself, or simply because their bodies were healing or because they were getting excellent nursing care, or even some other reason. 

The doctor's behavior in this COVID study reminds me of the Halstead radical mastectomy, which for decades was the dominant treatment for breast cancer in the U.S.. Surgeons at the time were so convinced that this treatment was the best one for breast cancer that they never bothered to compare it to another treatment--they never tested its efficacy. (When a randomized control study was finally conducted in the 1970s, it found that the surgery was no more effective than other, less invasive treatments.)

d) By selecting only relatively less sick patients, the doctor is stacking the deck in favor of finding evidence that the drug is effective. Many of the patients are likely to get better, so the doctor may conclude that the drug has worked. But of course, without a comparison group of equally healthy people who are not getting the drug, we cannot know if the patients would have recovered anyway.

e) f) g) The lack of informed consent--especially for vulnerable patients who have Alzheimer's--is a clear violation of the respect for persons principle. And the use of a risky treatment, one that is known to be dangerous to people with heart conditions, is likely a violation of the beneficence principle. I would argue that when a researcher conducts a flawed study like this, it also a violation of beneficence--the study puts participants at risk, while the benefit of the potential information gained from the research is negligible. We might also argue that the study violates justice: by studying only people at a nursing home, these patients bear the burden of being research participation when anyone can benefit from this research (assuming, of course, that the research were actually well-designed).

Many thanks to Dr. Stephen Chew for pointing me to this example!

Post script: Looking for more examples of bad studies, as described by science writers? Here are two:

Here's a critique of the many problems with another early published study on hydroxychloroquine.

Here's critique of studies supposedly investigating the death rate of COVID-19. 

Temperature data as quasi-experiment

Which government-led social distancing measures are effective at stopping the spread of the novel Coronovirus?  One company has contributed data to answering this question. A private health technology company, Kinsa sells a "smart thermometer" which links to an app that guides people through what to do if they have a fever. When people use the app, they agree to have their temperature information (including fever and other symptoms they report) shared with Kinsa. 

Kinsa has used body temperature data points from all over the U.S. to create maps showing areas of "atypical illness." They define atypical illness on their website as "an unusual incidence of elevated flu-like illness levels."

It's really interesting to explore Kinsa's atypical illness maps of the U.S. here. 

More relevant for a research-methods course are the figures that Kinsa presents that map "flu-like illness" rates over time in certain high-population counties (Boston, New York, Los Angeles). I've pasted two of them here. Take a look at these figures, and then answer the questions below. 

Image courtesy of Kinsa, Inc.

 

Image courtesy of Kinsa, Inc

 

a) These figures should remind you of one of the quasi-experimental designs covered in Chapter 13. Which design is it?

b) The Kinsa company has used these charts to argue that social distancing measures work to reduce cases of COVID-19. Journalists (for example) have used the charts to specifically argue that limiting large gatherings and closing schools alone does not do much to stop the spread; it was only when bars and restaurants were shuttered that these counties saw reduced rates of atypical illness. Look carefully at the three counties above. What parts of the data are relevant to this argument? Do you think the data support the claim that closing bars and restaurants was effective? Can you think of alternative explanations for the pattern, other than "closing bars and restaurants"?

c)  Finally, consider the ethics and scientific openness principles. Do you think it's ethical for Kinsa to use and publicize the private health information of people who buy its products? Consider arguments both for and against the scientific ethics of this practice. Frame your arguments around the ethical principles of respect for persons, beneficence, and justice. 

In your reflections, you might consider that users of Kinsa's products agree to data sharing via its terms and conditions. You might consider that the data from users is kept anonymous and is aggregated over more than a million users. You might consider the public health benefits of sharing the data. And you might also consider the extent to which Kinsa is following open science practices--that is, sharing all the data the way most university researchers now do (relevant Kinsa research information here). 

d) If you're construct-validity-curious, you might be wondering if Kinsa's definition of flu-like illness actually tracks COVID-19 cases. Kinsa do not claim to be diagnosing actual COVID (you need a nasal swab for that). However, they present some maps showing how their temperature data has tracked with confirmed COVID cases in several counties. The data are in the middle of this webpage.  

By the way, here's another example of a company that collects our health data, using it to share national patterns. This one is Fitbit, sharing how sleep time has increased (in most age groups) since the COVID quarantine started.

 

Is "sugar high" a thing?

 

Sugar doesn't make kids hyperactive; at least not according to a null effect study and a meta-analysis. Photo: Alamy

Here's something "everybody says":  Kids will become hyperactive if they eat too much sugar. This folk belief is as much a part of North American cultural wisdom as learning styles and the catharsis hypothesis. But this column in the New York Times explains that, like learning styles and catharsis, the idea of a "sugar high" is also a myth. 

 

The physician who wrote the column traced the modern idea of sugar causing hyperactivity to 1975. But then the author describes a randomized clinical trial on this question. Here's the description: 

An extraordinarily rigorous study settled the question in 1994. Writing in the New England Journal of Medicine, a group of scientists tested normal preschoolers and children whose parents described them as being sensitive to sugar. Neither the parents, the children nor the research staff knew which of the children were getting sugary foods and which were getting a diet sweetened with aspartame and other artificial sweeteners. Urine was tested to verify compliance with the diets. Nine different measures of cognitive and behavioral performance were assessed, with measurements taken at five-second intervals.

The original study was, indeed, ambitious. The authors provided all food to the homes of the children in the study so they could completely control what sweeteners were being used. The study took place over nine weeks in a repeated-measures design--the diets were based on sucrose (sugar) one week, aspartame another week, and saccharine during another week. The researchers measured the children's cognitive performance and mood throughout the study.

The study concluded that sugar does not affect children’s behavior or cognitive function. An editorial that accompanied the study put a fine point on that conclusion, stating “there is no evidence that sugar alone can turn a child with normal attention into a hyperactive child.” 

Questions

a) What were the independent and dependent variables in the study described here? What are the levels of the IV?

b) The IV was manipulated as within subjects, using a repeated-measures design. Describe why counterbalancing would be important here, and how you could counterbalance in this study. Which of the four big validities does counterbalancing support?

c) What questions could you ask about the other three validities for this study? 

d) This study found a null effect (no impact of sucrose on children's behavior). There were only 48 children in the diet study, and this might be considered a low number--there may not have been enough kids in the study to detect a small effect of sucrose. However, the repeated measures design means this study can make a stronger case for the conclusion that sugar doesn't affect kids' behavior. Review the information in Chapter 11 to explain why the repeated measures design helps support the conclusion. 

The author of the piece mentions this, too:

One year later, an analysis that gathered data from all published studies on the subject reached the same conclusion.

e) This sentence is describing a meta-analysis. In your own words, describe what steps they would have taken to conduct this meta-analysis.

f) Why is a meta-analysis stronger evidence for a null effect of sugar on hyperactivity? 

g) Given this information, what would you say to a relative or friend who believes that sugar causes hyperactivity in their own child? 

 

A serendipitous study of long-term memory in birds

Here's a photo of a toutouwai (North Island Robin) sporting its bands for identification. Photo: Shutterstock

We don't include many animal behavior studies in ERM blog, but here's a great one about animal memory. How long do you think wild animals can remember new skills they have learned? In this report in The Scientist, we read that the North Island Robin (known in the Maori language as toutouwai) of New Zealand can apparently remember a skill up to 22 months, without any reminders in the meantime. Here's how the study worked. 

[Originally, scientist Rachel Shaw] demonstrated that wild toutouwai living in the Zealandia Wildlife Sanctuary in Wellington, New Zealand, are capable of learning a novel task. Using a rectangular box with three compartments, each covered by a lid that swivels to the side to reveal its contents, she incrementally taught the birds how to peck the lids aside and retrieve mealworms hidden inside two of the three wells.

(By the way, the story in The Scientist includes a photo of the experimental apparatus and a video of a bird using it.)

Roughly a year later, Shaw was giving a public talk at the sanctuary about conservation and her work with birds. As she explained the memory apparatus she had used in her previous research, a toutouwai appeared.

“I haven’t given this bird this task in over a year,” she recalls telling the audience, “so let’s see how he does.” As if on cue, the bird instantly flew down to the apparatus and started opening the lids. That got her wondering: how many of the other birds could repeat the memory task and how long could they remember what they were taught?

This serendipitous event prompted Dr. Shaw to design a study to find out. Before reading on, how would you study this question? 

In the new study, Shaw and her coauthor, Annette Harvey, tested 32 of the initial 64 birds that were trained in 2015 and 2016 and banded for individual identification, 30 of which performed the memory task by spontaneously pecking the lids and opening them on their first try. None of the trained birds had seen the box between the initial training and retesting, though some of the untrained control birds had encountered the apparatus in a previous research project. The time between when the birds had first learned the lid-opening behavior and subsequent testing ranged from 10 to 22 months. By contrast, the 17 untrained birds were unable to complete the task.

The article mentions several other examples of long-term memory in animals, including pigeons, tortoises, nutcrackers, and chickadees. Read more if you are interested!

Questions

a)  This study might be considered a simple observational study but it might also be classified as a quasi-experiment. What do you think?

b) What role does the untrained comparison group play in this research? 

c)  Do you think the effect size in this study would be considered small or large? Explain your answer.

d) Chapter 6 describes important construct validity considerations for observational studies, such as reactivity, observer bias, and observer effects. Which of these might be relevant in this study on toutouwai? 

Thanks to my friend Dr. Stephen Chew for sharing this story on Twitter!

Claim: "Going to museums and concerts helps you live longer"

Does the study's method support the claim that attending arts events makes you live longer? Photo credit: agefotostock/Alamy

We'll never run out of material in the "correlation is not causation" department. 

This time, a New York Times  journalist led with the headline, "Another benefit of going to museums? You may live longer." 

Let's check out the study behind this causal claim. 

According to the journalist's summary of the empirical article, 

Researchers in London who followed thousands of people 50 and older over a 14-year period discovered that those who went to a museum or attended a concert just once or twice a year were 14 percent less likely to die during that period than those who didn’t.

The chances of living longer only went up the more frequently people engaged with the arts, according to the study, which was published this month in The BMJ, formerly The British Medical Journal. People who went to a museum or the theater once a month or even every few months had a 31 percent reduced risk of dying in that period, according to the study.

a) Starting just from here--what are the two variables they've mentioned so far? Were they manipulated or measured? 

b) Based on what you've read so far, is this study correlational or experimental? 

c) Without reading further, think of a third variable (a "C") that is likely to be correlated with both A (going to museums or concerts) and B (being less likely to die). Any good ideas? 

OK! Time for more information about the study from the journalist:

The study controlled for socioeconomic factors like a participant’s income, education level and mobility, said Andrew Steptoe, a co-author of the study and the head of University College London’s research department of behavioral science and health.

“Even if you take those things into account, we still found that there is a difference in the survival of people who are involved in the arts,” Professor Steptoe said in an interview on Friday.

 

Sounds like multiple regression was used here.

d) What was the criterion (dependent) variable in this study now?

e) What were the predictor ("independent") variables? 

f) Sketch a little regression table similar to those in Chapter 9--put the criterion variable at the top and the predictors at the bottom. Which betas do you think were significant?

g) More importantly, was the "C" variable you thought about in c), above, controlled in their regression?

h) Even though the authors controlled for income, education, and mobility, there are likely to be additional third variables that were NOT controlled for. Think of at least one more alternative explanation. 

 

As you learn in Chapter 9, regression does not support causation because even though it can control for alternative explanations you can think of (such as income, education, or mobility), it doesn't control for everything. If you read the Time's summary, you'll see even more variables they seem to have controlled for , such as marital status and profession. But there could still be unmeasured variables that are responsible for the relationship. 

 

But that doesn't stop the authors from making causal speculations. In fact, here's a mediator hypothesis. i) Sketch this mediator following the diagrams at the end of Chapter 9:

 

The study also noted that engaging in the arts can reduce loneliness, promote empathy and emotional intelligence, and keep people from becoming sedentary — all factors that contribute to a longer life.

 

What kind of study would you need to conduct to test this proposed mediator?

 

 

 

 

 

 

Does this font make me look conservative?

Of these three candidates, Yang is considered more liberal than Klobuchar. Do the fonts used in their posters convey that? Photo credit: Kevin McGovern/Shutterstock

Here's something I never considered--different fonts lean conservative or liberal. 

Serif fonts have little curves and edges (like the "Amy" in the green poster here). Sans-serif fonts are cleaner, without these extra flourishes. 

A study showed multiple fonts to a sample of people and found that people rated the serif fonts as looking more conservative and sans-serif fonts as more liberal.

This research was covered by CNN.com. They summarized the results like this: 

For example, study participants saw Times New Roman as more conservative than Gill Sans. Blackletter, which looks like it belongs on a newspaper masthead, was seen as the most conservative font, while Sunrise, a cartoonish-looking script, was seen as the most liberal.

When they interviewed the researcher, she suggested, 

"If you think about serifs being used in more formal types of print or communications, maybe they're viewed as more traditional and sans serifs are viewed as more modern," Katherine Haenschen, an assistant professor of communications at Virginia Tech and the lead author of the study, told CNN. "There's a small but significant difference in how people perceive these fonts."

The researchers showed each font using a simple sentence, "The quick brown fox jumped over the lazy dog" and asked online participants to rate the font's conservative (vs. liberal) nature. For example:

The quick brown fox jumped over the lazy dog

The quick brown fox jumped over the lazy dog

The quick brown fox jumped over the lazy dog

a) In this study, what was the independent variable (IV)? What were its levels? What was the dependent variable? 

Here is CNN's description:

...987 participants were shown the phrase "The quick brown fox jumped over the lazy dog" in one of six fonts and styles: Times New Roman regular, Times New Roman bold, Times New Roman italic, Gill Sans regular, Gill Sans bold or Gill Sans italic.

b) Based on this description, was the IV manipulated as independent groups or within-groups? 

c) You could do the study either way. What are the advantages and disadvantages of manipulating the font variable as independent groups?What are the advantages and disadvantages of manipulating the font variable as within-groups?

d) What does it mean when the author says "There's a small but significant difference in how people perceive these fonts."? 

At the end of the story, the CNN journalist wrote about the potential applications: 

For example, if Bernie Sanders changed the font on his campaign materials, there probably wouldn't be much of a difference in how people see him because he's already a widely known figure.

The choice of font could, however, make a difference for a new candidate, like someone running for school board, town council or state legislature, Haenschen said. While researchers don't know for sure whether a font would change people's perceptions of a candidate, that's another question that could be explored in the future.

e) How might you conduct an experimental test of the journalist's speculation, above? 

You can see the empirical journal article here. Their Figure 1 shows ratings of conservatism and the 95% CIs for six fonts. 

 

 

Training tools for information literacy

Photo: fotogestoeber/Shutterstock

More than ever, our citizenry needs skills for information literacy. A recent study of almost 4000 U.S. teens, conducted by Stanford's History Education group, concluded that teenagers' ability to think critically about online information is "troubling." Their report included the findings that: 

• Fifty-two percent of students believed a grainy video claiming to show ballot stuffing in the 2016 Democratic primaries (the video was actually shot in Russia) constituted “strong evidence” of voter fraud in the U.S. [...]
• Two-thirds of students couldn’t tell the difference between news stories and ads (set off by the words “Sponsored Content”) on Slate’s homepage.
• Ninety-six percent of students did not consider why ties between a climate change website and the fossil fuel industry might lessen that website’s credibility. [...]

If you're inspired to build skills in your own students, consider assigning an activity designed to develop students' critical reading skills as a supplement to Chapter 2. The following tools have been developed by experts in library science, communication, and social cognition. All of these raise our awareness of the information we may encounter online: 

Games that Teach

1. The free online game Bad News  trains people to spread false ideas with the goal of teaching them how propaganda works. It also includes an (optional) embedded test of people's ability to detect real and fake Twitter accounts. Users earn badges for impersonating experts, exploiting emotion, and polarization. It's a legitimately fun game! What's more, two recent studies have indicated that this game improves people's reasoning about the news (Roozenbeek et al., 2019;  Basol et al., 2020).

2. The Fake News Game (still in development) plans to teach about fake news by challenging people to create their own fake news, building in elements intended to make it go viral. 

Online Curricula

3. Stanford's Civic Online Reasoning (COR) course presents several planned sets of lessons:

• • An three-lesson introduction to skeptical reading, looking for evidence, and fact-checking
  • An six-lesson overview of lateral reading, the strategy used by professional fact-checkers (and a skill associated with better information literacy)
  • A combination package of six lessons offering a little bit of everything (lateral reading, introduction to evidence, and "click restraint"). This might be the package to assign if you're only assigning one set of lessons.

4. Carl Bergstrom and Jevin West's site, Calling Bullshit, offers a syllabus for a full semester college course on misinformation. You can also assign some of their "tools and tricks" to give people practice in the following skills:

• • Spotting misleading axes and formatting in graphs
  • Deciding if a scientific paper is legitimate
  • Deciding if an online face is real or computer-generated
  • They also have case studies in misinformation, sorted by difficulty. 

5. Morton Ann Gernsbacher has developed and shared her free, online methods course focused primarily on scientific information literacy. Check it out for a full set of resources!

My own students have benefitted from working through various modules from these resources, in both methods and content courses. 

I'm grateful to David Dunning (University of Michigan), who shared some of these resources at his NITOP general session. I'm also grateful to UD librarians Meg Grotti and Alison Wessel for sharing resources with me. 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers.