Everyday Research Methods

Coffee drinking and depression

A study recently reported that among a large sample of nurses (the sample included only women), women who drank more cups of coffee per day were less likely to be diagnosed with depression.

Here's how the story was covered by MSNBC.

The journalist reported the following about the study:

Participants were followed from 1996 to 2006 to see whether they were diagnosed with depression. None of the participants had depression at the study's start. Women were considered depressed if they had been given a diagnosis of clinical depression by their physician and they started taking antidepressants.

Over the 10-year period, 2,607 new cases of depression were reported. Women who drank four or more cups of coffee per day were 20 percent less likely to develop depression than those who drank one or fewer cups of coffee per week.

a.) Graph the association described in the quotes above. (Is a scatterplot or a bar graph more appropriate?)

b.) According to the journalist's report, does this study establish temporal precedence for a causal link between coffee and depression? If so, how?

c.) What third variables might you want to control for in a regression equation--to see if the relationship between coffee and depression is attributable to some outside factor?

Here's more from the journalist's article:

[An] assistant professor of psychiatry at the University of Rochester Medical Center, calls the findings "interesting," but urged caution regarding the link.

For one, the women included in the study were nurses and their caffeine consumption may not be reflective of the U.S. women population in general, Robertson-Blackmore said.

d) What kind of validity is this expert addressing?

e) Should the fact that the study included only nurses be reason enough to "urge caution regarding the link?" Consider what you read in Chapter 13.

 

Early in the article, the journalist betrays a lack of scientific training with the use of the word, "prove":

the new study only shows an association between coffee consumption and depression risk , and cannot prove that drinking coffee reduces risk of depression in women.

f) What do you think the journalist is trying to say here?  Instead of "prove" in the quote above, what might you say instead, and why?

g) Finally, what questions can you ask about the construct and statistical validity of this study? Where might you find the answers?

 

 

Suggested answers:

a)  Either a scatterplot or bar graph could work.  If you did a scatter, then the x-axis would be labeled with "number of cups of coffee per day."  If you did a bar graph, then you might use the category labels "one cup or less per week" and "four cups or more per day." 

b)  The study began by selecting a sample of women who did not have a diagnosis of depression and then followed them for several years.  So we can be reasonably sure that the coffee drinking habits came first, followed by the depression diagnoses.

c) To be a plausible threat to internal validity, a third variable must correlate positively with both variables in the right way--that is, with more coffee drinking and lower depression (or conversely, with less coffee drinking and more depression).  Some might say "stress" is a third variable, but that would mean stress would  be associated with both more coffee drinking and less depression, which seems implausible (because usually stress is related to depression).  Social life might be a plausible third variable:  Women might drink more coffee in social contexts, and social life might also buffer depression symptoms.  It is best to consult the original article to see which third variables were controlled for. 

d) By saying that the results "may not be reflective of the U.S. women population in general," the expert is addressing external validity.

e)   Without additional data, we can never say for certain whether research with one sample can generalize to other populations.  The study included nurses, who may be more healthy and educated than many American women.  However, it may still be reasonable to assume that nurses are similar enough in important ways to the average woman, that we can generalize their results.  

f) By stating that this study "only shows an association," the journalist probably meant to emphasize that the study design does not allow a causal claim. (Remember that in science, a study never "proves" anything.)

g) To interrogate construct validity, you can evaluate how depression was measured.  As the journalist reports, depression was measured  by a doctor's diagnosis and a prescription for antidepressants.  Is this a valid and reliable measure of depression?  In addition, you can ask about the construct validity of the coffee consumption variable.  Did women report on their own coffee drinking?  Can they report on that accurately?

To ask about statistical validity, you can ask about the effect's statistical significance and its effect size.  In fact, effect size information is carried in the statement, "Women who drank four or more cups of coffee per day were 20 percent less likely to develop depression than those who drank one or fewer cups of coffee per week." This sounds like a relatively strong effect size. 

Ben Goldacre on TED

Here's Ben Goldacre in a 20-minute TED video talking about his favorite topic, Bad Science.  He addresses the journal-to-journalism cycle, the flaws of authority, the placebo effect, industry-sponsored trials, publication bias, correlation/causation, ethics, and more.  The overall message is the importance of publishing all data--both for and against a drug, therapy, or intervention. 

Ben Goldacre on TED.

Can a fake smile be bad for you?

Here's a study on the moods of a sample of bus drivers. Bus driving is a profession in which people must be courteous to others throughout the day. What impact might such job-mandated courtesy have on bus drivers? The article reports that,

... on days when the smiles were forced, the subjects’ moods deteriorated and they tended to withdraw from work. ...But on days when the subjects tried to display smiles through deeper efforts — by actually cultivating pleasant thoughts and memories — their overall moods improved and their productivity increased.

a. What are the independent and dependent variables in this study?

b. Can you graph the outcomes described? (Note--there are two DV's, so choose only one for your sample graph.

c. From the journalist's description, it's hard to tell if the "forced smile" vs. "deeper smile" variable is measured or manipulated. Why does this matter: What kinds of conclusions could you draw if this variable was manipulated? What kinds of conclusions can you draw if this variable was only measured?

Photo: Tina Fineberg, Associated Press

Later on, the authors state the following:

Women were affected more than men.

d. Redraw your graph, adding this gender variable. In the graph that you drew, do you notice any main effects? Interactions?

Shrimp on a treadmill? What's research really about?

Sometimes journalists or politicians call out research projects that they consider to be a waste of money, such as a line of research in biology and ecology in which shrimp ran on a treadmill. This news story from National Public Radio covers some examples of research whose measures or instruments have been criticized as silly, because they seem frivolous--at least on the surface.

According to the story,

The treadmills were just a small part of it, a way to measure how shrimp respond to changes in water quality. Burnett says the first treadmill was built by a colleague from scraps and was basically free, and the second was fancier and cost about $1,000. The senator's report was misleading, says Burnett, "and it suggests that much money was spent on seeing how long a shrimp can run on a treadmill, which was totally out of context."

In the language of the textbook, the behaviors of shrimp on a treadmill can be called an operationalization, in this case, as a way to operationalize the variable "reaction to water quality." On the surface, some research operationalizations can seem silly, but they almost always represent something more serious.

Rather than criticize how frivolous an operationalization is, we should ask how well that operationalization represents the construct it is meant to measure (that is, we should interrogate its construct validity).

The same story gives another example of a "silly study" that was attacked by some critics:

[One] group has also criticized other government-funded studies, including a National Institutes of Health grant for measuring nicotine exposure in toenail clippings. "They used recovery money, money that was meant to more or less stimulate the economy," says Lafferty. "Interesting use of money — mailing in toenail clippings."

That use of recovery funds was defended by Lawrence Tabak, deputy director of the National Institutes of Health. And he says the toenail study has an important goal that its critics didn't mention — trying to assess people's risk of lung cancer.

Consider the example above. What variable are the researchers operationalizing by measuring the qualities of toenail clipplings? What other measures, or operationalizations, could they also use to meausre the same construct? What evidence would support the construct validity of the toenail clipping measure?

On the political trail: Can we say a theory "hasn't been proven"?

When asked about global warming, one of the U.S. presidential candidates in the Republican field was quoted last month as saying:

"I don't think from my perspective that I want to be engaged in spending that much money on still a scientific theory that has not been proven and from my perspective is more and more being put into question." (Texas Governor Rick Perry, August 17, New Hampshire, Politics and Eggs breakfast).

No matter what your own view is on this candidate, and no matter what your understanding is of global warming , is it appropriate to state, as Perry says here, that the theory of human-made global warming "has not been proven"?

According to the material in Chapter 1, a scientific theory can never be proven; it can only be strengthened by data or weakened by the data. Bloggers Nancy Abrams and Joel Premack address the idea very clearly in their article here. For example, Abrams and Premack state,

"[a theory] can never be proved true because that would mean it couldn't be refuted; and if it can't be refuted, by definition it's not a scientific theory -- it's faith, not science."

Political debates on global warming would benefit from research methods lessons from Chapter 1. Appropriate questions to ask about any theory include, "What is the strength of the evidence in support of this theory?" and "Is the theory falsifiable?" We should not be asking, "Is this theory proven?" because in science, no theory ever is.

Curly or straight? A journalist experiments on her hair...

In this ABC news video, a reporter shows what happened when she presented herself with her naturally curly hair or her straightened hair, in three situations.

1. Consider the first experiment--with the panel of men.

a. What were the independent and dependent variables in the reporter's study?

b. How did she manipulate the independent variable--as within groups or between groups?

c. Interrogate the internal validity of this study. What confounds do you notice and how do they affect her ability to conclude that "men like women with straight hair better"? How might you redesign her study to make better conclusions?

d. Interrogate the statistical validity of her report---what questions are worth asking?

e. What about external validity? Ask questions about two aspects of this study's external validity.

 

Journalist Taryn Winter Brill demonstrated what happened when she straightened her natually curly hair. Photo by Suzanne Kreiter/The Boston Globe via Getty Images

 

2. Now consider the second experiment--with the group of children.

a. What were the independent and dependent variables in this second study?

b. How did she manipulate the independent variable this time--as within groups or between groups?

c. Interrogate the internal validity of this study. What problems do you notice and how do they affect her ability to conclude that "kids prefer women with straight hair"? What should she change in a future study?

3. Finally, consider the last experiment--the job interview.

a. What were the independent and dependent variables in this third study?

b. How did she manipulate the independent variable this time--as within groups or between groups?

c. Interrogate the internal validity of this study. What problems do you notice and how do they affect her ability to conclude that "employers are more likely to hire women with curly hair"?