Everyday Research Methods

I like coffee; I like tea. Meta analyses say they're good for me...

Why do researchers consider meta-analyses to be so valuable? Photo: Shutterstock

A health columnist has summarized the results of research on the health benefits of drinking coffee and (in a separate column) drinking tea. Some of these benefits include better mental health, and since they are based on meta-analyses, it makes sense to discuss them here.

First, the coffee column. Here's some introductory material:

Coffee has long had a reputation as being unhealthy. But in almost every single respect that reputation is backward. The potential health benefits are surprisingly large.When I set out to look at the research on coffee and health, I thought I’d see it being associated with some good outcomes and some bad ones, mirroring the contradictory reports you can often find in the news media. This didn’t turn out to be the case.

Here's one example of a meta-analysis the journalist describes:

...a meta-analysis — a study of studies, in which data are pooled and analyzed together — was published looking at how coffee consumption might be associated with stroke. Eleven studies were found, including almost 480,000 participants. As with the prior studies, consumption of two to six cups of coffee a day was associated with a lower risk of disease, compared with those who drank none. Another meta-analysis published a year later confirmed these findings.

That seems like a lot of support for the benefits of coffee on stroke, right? The journalist also lists examples of meta-analyses about coffee and heart disease, diabetes, and neurological disorders (see the complete article for all of them). 

a) Reviewing what your textbook says about meta-analyses in Chapter 14, is the author's definition ("a meta-analysis — a study of studies, in which data are pooled and analyzed together") correct? Would you add anything to it?

b) Why is a meta-analysis generally considered to be excellent evidence for a phenomenon?

c) What is the relationship between meta-analysis and replication studies?

d) Can you think of any downsides to meta-analyses?

e) Read the following statement, and decide, first, what the two primary variables in the meta-analyzed studies would have been; and second, decide if the meta-analyzed studies were likely to have been correlational or experimental studies.

The most recent meta-analyses on neurological disorders found that coffee intake was associated with ...lower cognitive decline...

Indeed, the following statement helps you answer question e), above, and gives a really nice twist to the correlation/causation issue:

I grant you that pretty much none of the research I’m citing... contains randomized controlled trials. It’s important to remember that we usually conduct those trials to see if what we are observing in epidemiologic studies holds up. Most of us aren’t drinking coffee because we think it will protect us, though. Most of us are worrying that it might be hurting us. There’s almost no evidence for that at all.

Now let's turn to the article about tea. The journalist mentions many examples of meta-analyses on tea consumption, including this one:

Tea has been associated with a lower risk of depression. A 2015 meta-analysis of 11 studies with almost 23,000 participants found that for every three cups of tea consumed per day, the relative risk of depression decreased 37 percent.

and these:

A more recent meta-analysis examined 22 prospective studies on more than 850,000 people and found that drinking an additional three cups of tea a day was associated with a reduction in coronary heart disease (27 percent), cardiac death (26 percent), stroke (18 percent), total mortality (24 percent), cerebral infarction (16 percent) and intracerebral hemorrhage (21 percent).

The journalist makes two great points in the tea article. First, he pointed out:

Again, these are all mostly data from observational studies, and as such, they can’t prove causality and should be taken with a grain of salt. We’ve been burned many times before by assuming that what we see in associations in cohort studies will turn out to be truly causal when behavior changes, only to see that fall apart in randomized controlled trials.

f) Summarize, in your own words, what the journalist is arguing here. And then speculate, what are some reasons why  such correlational (epidemiological) findings might turn out not to be causal?

Second, the journalist wrote:

The majority of studies have been done in Asian countries where tea drinking is much more common than in the United States. It’s possible that the people who don’t drink tea in those countries are different from those who do in a way that doesn’t translate to people in the United States.

 g) Which one of the four big validities is the journalist referring to here? Challenge question: how might a moderator analysis in a meta-analysis help answer the journalist's question?

"Screentime drags down grades"

Photo: VGstockstudio/Shutterstock

Several news outlets picked up on a study about high school students' screentime and their grades. The Huffington Post headlined their story, "One Hour Of Extra Screen Time Drags Down Teenagers' Grades"

The summary of the study read like this:

Teenagers who spend an extra hour a day surfing the internet, watching TV or playing computer games risk performing two grades worse in exams than their peers who don't, according to research by British scientists.

An extra hour in front of the TV or online at age 14-and-a-half was linked with 9.3 fewer exam points at age 16 -- equivalent to two grades, for example from a B to a D. Two extra hours was linked to 18 fewer points.

a) Based only on this summary, what kind of study do you think this is--correlational or experimental? What are the two key variables in this study?

b) Since the study is correlational, you might be surprised that the headline was causal, saying that "screentime drags down grades." Apply the three criteria for causation to this claim, given what you know about the methodology. Does it support the claim?

Later in the article, the researcher explains the following about the study:

"We only measured [screen time at age 14.5], but this is likely to be a reliable snapshot of participants' usual behavior, so we can reasonably suggest that screen time may be damaging to a teenager's grades," said Kirsten Corder of Cambridge's Centre for Diet and Activity Research, who co-led the work.

c) In this quote, the researcher seems to be focusing on one of the three criteria for causation. Which one? Which one is she ignoring?

d) Challenge question: In the quote above, the researcher mentions "We only measured [screen time at age 14.5], but this is likely to be a reliable snapshot of participants' usual behavior".  What concept is the researcher discussing here? Is this concept relevant to the causal claim the researchers and journalists wish to make?

e) Does this study appear to be a cross-lag panel design? Why or why not?

Suggested answers:

a) This study is most likely correlational, because it seems very likely that the two key variables in the study--amount of time spent on screens and grades earned--were measured.

b) Covariance: Yes, the study's results do suggest that more screen time is associated with lower grades.

Temporal precedence: Yes, the study measured screen time two years before grades.

Internal validity: It's not clear if any third variables were controlled for in a regression analysis. Even if they were, a regression analysis cannot control for all possible third variables, so this study cannot achieve internal validity. For example, it's possible that boys spend more time on screens and also get lower grades, so gender could be a third variable. Students with less parental support might both spend more time on screens and achieve lower grades, so parental support could be a third variable, too.

c) By emphasizing that screen time was measured at age 14.5, the authors seem to be emphasizing the study's temporal precedence, but not its internal validity.

d) The comment is referring to the test-retest reliability of the screen time measure. The researcher is saying that she assumes that screen time will be a stable trait, and that it would have good test-retest reliability. Of course, test-retest reliability is an empirical question--you have to collect data to find out if it exists for some measure. Assuming is not enough. In addition, the test-retest reliability of a measure is a consideration for construct validity.  Having good construct validity is important for a study, but construct validity does not, by itself, help a study support a causal claim.

e) This is probably not a cross-lag panel design. In such a design, you need to measure the two key variables (screen time and grades) twice: once at Time 1 and again at Time 2. By saying that they only measured screen time once, the researcher is also revealing that this study is not a cross-lag panel design.