Everyday Research Methods

Measuring charter schools

Are charter schools effective?  Philadelphia Inquirer columnist Joel Naroff wrote about the data we might use to decide whether charter schools are doing well. He describes the pitfalls of using achievement test scores to measure school effectivness. His article uses the language of multiple regression and internal validity to describe how difficult this process is.

Specifically, Naroff mentions the following about charter schools, discussing

....what researchers call "selection bias." Parents have to choose to send their children to a charter school, meaning that charters start with students who have high parental involvement.

a) To translate his comment into our own research methods terms:  Imagine a study that compared achievement test scores in a city's charter schools with the city's traditional public schools. Sketch a simple bar graph showing that charter schools score higher in achievement than traditional schools. Does this result mean that charter schools cause better school performance?  What are some internal validity issues here?  Apply the term  "selection bias" or "selection effect," as the author is doing above.

This article also states that

Unfortunately, we don't adjust test scores for differences in factors such as intelligence, income, parental involvement, or school facilities. Instead, we use misleading, unadjusted test-score comparisons. That means some schools that test well might not be doing all they can, while those with failing scores might be making outstanding progress. We are largely clueless about what is happening.

In this paragraph, the author seems to be advocating for using multiple regression to evaluate the effectiveness of charter schools. 

b) Why would multiple regression help us rule out the selection bias problem the author mentioned earlier?

c) Sketch a possible table of multiple regression results, as the author suggests above: Assume that achievement test score is the dependent variable of interest. What would be your predictor variables? What results would show that charter schools really are more effective than traditional schools, even controlling for these selection effect variables?  What results would show that charter schools are not more effective than traditional schools?

d) Do you agree that "we are largely clueless" about what is going on? Do you think the regression approach he mentions is a good idea? Are there any downsides to the regression approach?

 

Suggested answers to question c:

Below is a possible table of (fabricated) results that would show that charter schools really are more effective than traditional schools, even controlling for the selection effect variables the author mentioned.

DV: Achievement test scores

Predictor                                                                                    Beta

Traditional vs. Chater (0 = traditional, 1 = charter)       0.15**
Entering achievement test score                                         0.59*
IQ of child                                                                                0.20**
Child's family income                                                            0.11**
Child's parental involvement                                               0.18*
School's level of facilities                                                       0.08

Note: Data are fabricated for teaching purposes.

* represents a statistically significant effect. This table suggests that charter schools are associated with better achievement test scores, even when controlling for earlier scores, IQ of child, income, parental involvement, and school facilities.

 

Now, below is a possible table of (fabricated) results that would show that charter schools are not more effective than traditional schools, even controlling for these selection effect variables

DV: Achievement test scores

Predictor                                                                                    Beta

Traditional vs. Chater (0 = traditional, 1 = charter)       0.03
Entering achievement test score                                        0.57*
IQ of child                                                                               0.24**
Child's family income                                                           0.13**
Child's parental involvement                                               0.16*
School's level of facilities                                                       0.06

Note: Data are fabricated for teaching purposes.

* represents a statistically significant effect. This table suggests that charter schools are not associated with better achievement test scores, when controlling for earlier scores, IQ of child, income, parental involvement, and school facilities.

Trashing your thoughts reduces their power

Another article from the mindfulness desk, this time graphable as an interaction.  Read on.

A recent study by psychologist Richard Petty and his colleagues investigated the situations in which people's thoughts have more power over them. Specifically, they wanted to know if thoughts would affect people less when they wrote them down and then threw them away--as in literally, in a trash can! Here's how the study was covered by the Huffington Post.

This study is a factorial design. Read the journalist's description below very carefully, attempting to locate the two independent variables and the dependent variable. 

[The first experiment] included 83 high-schoolers in Spain who were given three minutes to write their negative or positive thoughts about their own body image.

After writing down these thoughts, all of them were asked to read them back over and think about them. Half of them were then asked to throw away those written thoughts in the trash, while the others were not instructed to throw away their thoughts and were instead asked to proof-read what they had written. Then, researchers had the study participants rate their attitudes on their own body image on a scale -- for example, if they liked or disliked their bodies, thought they were attractive or unattractive, etc.

Researchers found that for the students who were not asked to throw away their written thoughts on their self-body image, what they had written down seemed to have an effect on how they rated their body image afterward. For example, someone who wrote down a lot of positive thoughts about themselves were likelier to rate themselves higher on the body image scale.

However, for the students who were asked to throw away their written thoughts, what they wrote down didn't seem to have any effect on how they rated themselves afterward.

a) What are the two IV's in this study?  Name each IV. Then list the levels of each IV. Are the levels of each IV manipulated as independent groups or within-groups?  

b) What is the main DV in this study?

c) Sketch a graph of the outcome of this study.

d) Estimate the main effects and interactions that Petty and his colleagues probably obtained.

The journalist also describes a second study in the paper. Here's the journalist's description:

 Some of the study participants were then asked to drag that file into the computer's recycling bin; others were instructed to just drag the file to a storage disk. Some of them were also asked to just imagine that the file was moved to the recycling bin.

The researchers found that those who actually dragged the file to the recycling bin were less affected by the thoughts they'd typed out, compared with the ones who just saved to them another disk, or those who just imagined moving them to the recycling bin.

e) The study above is a conceptual replication of the first. Instead of using a real trash can to operationalize the concept of "trashing," they used a virtual one--on a computer. Can you think of another way to operationalize "trashing?"

 f) The blogger's description of this second study is much less detailed than the blogger's description of the first. What details are left out of this description?

 

Suggested answers

a) The first IV is whether people wrote about positive or negative thoughts. It has two levels (positive or negative) and is independent groups.

The second IV is whether people proofread what they wrote or threw it in the trashcan. It has two levels (proofread or trash) and is independent groups.

b) The DV is body image, operationalized by a body image scale.

c) Based on my reading of the results, the graph might look something like this:

If your library has access to the journal Psychological Science, then you can look up the real graph of the results for this study. The study shows them in Figure 1.

d) The graph I sketched shows a probable main effect for the variable thought type, such that positive thoughts lead to higher body image than negative thoughts. There is probably no main effect for the trashed vs. proofread IV. There is an interaction, such that when people trashed their thoughts, positive messages led to only a little more positive body image than negative ones; but when people did not trash their thoughts, positive messages led to a lot more positive body image than negative ones.

e) Another way to operationalize "trashing" might be with a paper shredder, or by burning the paper you write your thoughts on. What are your ideas?

f) The second study description doesn't mention what topic participants wrote about (body image again? or something new?). The article mentions one IV (which has three levels--whether people dragged the file to the recycle bin, dragged it to the save icon, or just imagined dragging it to the recycle bin). But the article doesn't mention if there was another IV (such as positive vs. negative thoughts). It also doesn't mention how they operationalized the DV--they just say it was about being "affected by the thoughts." You can read the original published article in Psychological Science to look up these details (see p. 5).

 

Exercise and Depression: No effect after all?

A science blog reviewed a study that showed a null effect for exercise and depression. Although some past research--and certainly popular belief--have indicated that exercise can improve depressive symptoms, this study showed no effect:

Danish researchers Krogh and colleagues randomly 115 assigned depressed people to one of two exercise programs. One was a strenuous aerobic workout - cycling for 30 minutes, 3 times per week, for 3 months. The other was various stretching exercises.

The idea was that stretching was a kind of placebo control group on the grounds that, while it is an intervention, it's not the kind of exercise that gets you fit. It doesn't burn many calories, it doesn't improve your cardiovascular system, etc. Aerobic exercise is the kind that's most commonly been proposed as having an antidepressant effect.

So what happened? Not much. Both groups got less depressed but there was zero difference between the two conditions. The cyclists did get physically fitter than the stretchers, losing more weight and improving on other measures. But they didn't feel any better.

Chapter 10 addresses what we should think about when an experiment shows no significant effect. We can apply those categories to this example.

a) Specifically, does it seem that there was enough variability between levels?  Did the experimenter use a strong enough manipulation of exercise? What other questions might you ask here?

b) Or what if there was too much variability within each group? What questions might you ask to evaluate this possibility?

c) The blogger seems to conclude that perhaps exercise really doesn't affect depression. Do you think that is a reasonable conclusion too?

 

 

Suggested Answers

a) The study seems to have used a manipulation check, because they measured the physical fitness of the two groups after the period of exercise was over. The aerobic group was in better shape than the stretchers were, so it does seem that exercise was manipulated appropriately.

b) As the book explains, researchers use several techniques to reduce within-group variability. In this study, we might ask if the researchers used the most precise measure of depression that they could. We could also ask if the researchers attempted to keep situation noise to a minimum. If those steps were impossible, then another way to counteract within-groups variability is to use a within-groups design. This study used an independent-groups design, so researchers did not use this approach. 

When a within-groups design is not possible and a researcher is trying to control unsystematic variability due to situation noise and individual differences, the most common fix is to simply test more people. This study had about 65 people in each group. Although the number may seem small to you, a sample of this size is often enough to find a medium-sized effect of exercise. On the other hand, if the true effect of exercise on depression is very small, the researchers might have found a significant effect if they had used more than 100 people in each group.

(Appendix B in the book uses this analogy: If you're looking for an elephant in a dark room, you need only a small light--a small sample--to find it. If you're looking for an earring in that dark room, you'd need a large, powerful light--a large sample--to find it.)

c) If exercise had a medium strong effect on depression, it seems that this study would have found it. What do you think? What kind of further evidence would convince you that exercise does not help depression?

 

 

Cognitive benefits to being bilingual

Researchers have found that bilingualism is associated with delayed onset of dementia symptoms. This story was covered in many North American news outlets, including the CBC:

Researchers at Toronto’s York University reviewed the hospital records of patients who were either monolingual or bilingual and who all had been diagnosed with dementia. A wide range of languages were spoken by the study participants.

They discovered that bilingual people were diagnosed three to four years after people who spoke one language....

a.) According to the statement above, what kind of study was conducted here? (Correlational or experimental?)  What are the variables in the study?  Were they measured or manipulated?

b.) Apply the three causal rules to this finding. Can we conclude that being bilingual prevents dementia onset? 

A comment by the lead research specifies a possible mediator for this relationship:

"Lifelong experience in managing attention to two languages reorganizes specific brain networks, creating a more effective basis for executive control and sustaining better cognitive performance throughout the lifespan," says Ellen Bialystok, lead author, in a release.

c) What kind of study would the researchers need to conduct in order to test this mediational hypothesis? 

 

Suggested answers:

a) This is a correlational study. The two measured variables were whether each person was bilingual or not, and the age at which each person was diagnosed with dementia.

b) There is covariance here--bilingualism was associated with a later age at diagnosis. There is temporal precedence, too--we know that people were bilingual first, and diagnosed with dementia later. What about internal validity?  It is possible that some other third variable is associated with both dementia onset and bilingualism. Perhaps education level or job status are third variables? In one part of the press release, we read that these particular third variables are not responsible. Consider this quote:

Interestingly, educational level and occupational status favoured the monolingual group, suggesting that those factors were not as critical in the development of dementia as linguistic ability, according to the study.

This information indicates that education and job status could not be the third variable, because although these two variables usually delay dementia onset, they were actually higher in the monolingual group. However, there may be other third variables, unmentioned here, that could be responsible for the relationship. Therefore, while we can predict that bilinguals will have a later onset of dementia, we cannot say for sure that bilingualism is the causal factor--the "active ingredient."

c) A study to test this mediational hypothesis would measure not only bilingualism and age at diagnosis, but would also measure brain networks involved in executive control. To support mediation, the results should show that bilinguals have richer such networks, and that such networks are associated with delayed dementia onset. Furthermore, the relationship between bilingualism and delayed onset of dementia should get weaker when the measure of brain networks is controlled for.

 

Do magazine personality tests tell us anything?

Have you ever seen those magazine articles entitled, "What does your _____ say about you?" Magazines like to make us think we can use people's pets, music preferences, Saturday wardrobe, or cars as secret keys to making personality judgments about them. Is that really possible?

Here's an example of this kind of article, from Men's Health, claiming to be able to tell you "What your drink order says about you."

Of course, magazine articles like these aren't meant to be scientifically rigorous--they are mainly for entertainment. Nevertheless, what is this particular article able to say? Use your research methods skills to interrogate this piece.

a. Read the first three paragraphs--what sources of evidence does the author use for his claims? 

b. Read the fourth paragraph--the one about the personalities of people who order light beers. The paragraph states,

Domestic Light Beer (Bud Light, Miller Light)

You’re impulsive and quick to take risks. You’re respectful of authority, accepting of most people, and generally easy to get along with.

Can you turn the above statements into an association claim? (Hint--identify two variables at a time.)

c. Another part of this article states that people who order vodka "care about fashion and appearance." How might a study be conducted that would be able to test whether people who order vodka are more fashion-conscious?  Make a graph of data that would support this claim.

 

Suggested answers

a. It seems that the authors used three sources of data. One was a published study conducted by the University of Pittsburgh; this study is mentioned in the first sentence but its findings are not discussed further in this article. Another source of data was market research reports which had been produced by some market research companies. A third source was anecdotal--that is, experiential evidence--gathered from bartenders. (You might pause to think about the credibility and accuracy of these three types of data, using guidelines from Chapter 2.)

b. This paragraph is describing bivariate assocations between the variable, "preference for light beer" and various personality traits. One association claim you could make is, "preference for light beer is associated with impulsivity." Another could be, "preference for light beer is associated with risk taking," and so on.

c.  In order to show that people who order vodka "care about fashion and appearance," researchers would select a sample of people and measure both their tendency to order vodka and their degree of interest in fashion and appearance. Then they would see whether the two variables are correlated in the sample. Picture the correlation as a scatterplot with "tendency to order vodka" on one axis and "degree of interest in fashion and appearance" on the other axis. The dots (each representing one person) would arrange themselves in a cloud of points sloping upward, representing a positive association between these two variables.

Follow up question--using the answer to Question c, decide how you could test the association claims made in Question b. How could you graph these claims?

 

Links between spanking and later mental health

A recent article in Yahoo! news network describes research on spanking and mental health.  The journalist reports:

Corporal punishment was associated with increased odds of anxiety and mood disorders, including major depression, panic disorder, post-traumatic  stress disorder, agoraphobia and social phobia. Several personality  disorders and alcohol and drug abuse were also linked to physical  punishment, the researchers found.

The study, originally published in the journal Pediatrics, intended to test the effects of physical punishment separately from more severe physical or sexual abuse. The journalist reported that the method involved

about 34,000  individuals aged 20 or older gathered from the U.S. National Epidemiologic  Survey on Alcohol and Related Conditions. Participants were questioned face-to-face and asked, on a scale of "never" to "very often," how often  they were ever pushed, grabbed, shoved, slapped or hit by their parents or another adult living [in] their home.

a. The journalist's story mentions that while the study shows that physical punishment is associated with some mental illnesses, the results cannot "prove that one causes the other." Why not?  What are three possible causal paths for this association? (Hint:  Think about both temporal precedence and the third variable problem)

Part of the story reports that

The researchers adjusted the data to take into account  socio-demographic factors and any family history of dysfunction.

b. Was family history of dysfunction one of your third variables in question a? If so, what does the above statement mean?

One of the experts who was interviewed about this study said,

"While it's a well-done study, looking at a national data sample, there are limitations in the way the study was done," said Dr. Andrew Adesman, chief of developmental and behavioral pediatrics at Steven and Alexandra Cohen Children's Medical Center in New Hyde Park, N.Y. "There are  limitations to relying on adults recalling childhood experiences...."

c.  This expert is evaluating two of the big validities. Which of his quoted statements evaluate which validities, and why?

Suggested answers

a. The two variables in this association are a history of physical punishment and a diagnosis of mental illness. It could be the case that physical punishment causes mental illness. It could also be the case that kids who show early signs of mental illness (somehow) cause their parents to be more physical with them. Or, it is possible that a third variable, such as family dysfunction or poverty, is associated with both mental illness and physical punishment.

b. The statement means that the researchers used a multivariate correlational design, testing whether there is still a link between spanking and mental illness even when controlling for family dysfunction.  Since the link is still there, it means that family dysfunction can actually be ruled out as a third variable explaining the relationship.

c. The expert praised the external validity of the study when he said, "While it's a well-done study, looking at a national data sample".  But he criticized the construct validity of the study when he said, "there are limitations to relying on adults recalling childhood experiences". By mentioning that people may not accurately recall their childhoods, this expert is suggesting that the study's operationalization of spanking may have been flawed.

 

When do people cheat?

Dan Arieli, a behavioral economist at Duke University, has publshed a book for popular audiences called, The (Honest) Truth about Dishonesty. Some of the research from his book was summarized recently in the Wall Street Journal.

Dr. Arieli and his colleagues have concluded from their research that almost everybody has the capacity to cheat and lie--just a little. Their research investigates the situational forces that make cheating more or less likely. But how do you study cheating in a laboratory? Arieli and his colleagues have creatively operationalized cheating using a math test technique. Here's how he describes it:

Test subjects (usually college students) are given a sheet of paper containing a series of 20 different matrices...and are told to find in each of the matrices two numbers that add up to 10. They have five minutes to solve as many of the matrices as possible, and they get paid based on how many they solve correctly. When we want to make it possible for subjects to cheat on the matrix task, we introduce what we call the "shredder condition." The subjects are told to count their correct answers on their own and then put their work sheets through a paper shredder at the back of the room. They then tell us how many matrices they solved correctly and get paid accordingly.

...In the control condition, it turns out that most people can solve about four matrices in five minutes. But in the shredder condition, something funny happens: Everyone suddenly and miraculously gets a little smarter. Participants in the shredder condition claim to solve an average of six matrices—two more than in the control condition. This overall increase results not from a few individuals who claim to solve a lot more matrices but from lots of people who cheat just by a little.

Like most programmatic research, Arieli's is theory-driven. Specifically, their theory states that people are motivated to rationalize cheating to themselves: They try to balance the gains of cheating against their own self-respect. Importantly, their theory also states what is NOT happening: People are not driven to cheat simply because they compute the gains and losses of cheating: they don't rationally think about how much money they will gain and how likely it is they will be caught.

Read this description of one of their studies to see how their data supports the rationalization theory, and not the gain-loss theory:

Would putting more money on the line make people cheat more? We tried varying the amount that we paid for a solved matrix, from 50 cents to $10, but more money did not lead to more cheating. In fact, the amount of cheating was slightly lower when we promised our participants the highest amount for each correct answer. (Why? I suspect that at $10 per solved matrix, it was harder for  participants to cheat and still feel good about their own sense of integrity.)

a) How does this result support their "rationalization" theory, rather than the "rational gain-loss theory?"

 

Arieli's laboratory has conducted a number of studies to find out which situational forces are likely to affect people's cheating--either to make it worse, or make it better.

b) For each example below, identify the manipulated variable and the measured variable.

c) What causal statement can each study make? Explain how the experimental nature of each of the studies helps support the causal claim.

Example i.

One thing that increased cheating in our experiments was making the prospect of
a monetary payoff more "distant," in psychological terms. In one variation of
the matrix task, we tempted students to cheat for tokens (which would
immediately be traded in for cash). Subjects in this token condition cheated
twice as much as those lying directly for money.

Example ii.

Another thing that boosted cheating: Having another student in the room who was
clearly cheating. In this version of the matrix task, we had an acting student
named David get up about a minute into the experiment (the participants in the
study didn't know he was an actor) and implausibly claim that he had solved all
the matrices. Watching this mini-Madoff clearly cheat—and waltz away with a wad
of cash—the remaining students claimed they had solved double the number of
matrices as the control group. Cheating, it seems, is infectious.

Example iii.

We took a group of 450 participants, split them into two groups and set them
loose on our usual matrix task. We asked half of them to recall the Ten
Commandments and the other half to recall 10 books that they had read in high
school. Among the group who recalled the 10 books, we saw the typical widespread
but moderate cheating. But in the group that was asked to recall the Ten
Commandments, we observed no cheating whatsoever.

Suggested answers

Example i:
b) The manipulated variable was whether people could earn tokens or actual dollars for solving math problems.  The measured variable was how many math problems people said they solved.

c) The causal claim they can make is that "having people earn tokens rather than real dollars causes people to cheat more on this math task."

They can support this causal claim because they have covariance (people in the token condition claimed solving twice as many problems as those in the cash condition). They also have temporal precedence (the token/cash condition was administered before subjects had the chance to cheat). And we might assume internal validity, too. People were probably randomly assigned to the token and cash conditions; both groups of people were solving the same math problems in the same settings, with the same absolute value assigned to the payoff. The only thing that differed was the form that the payoff took (cash or token). This causal claim seems justified!

California high schoolers eat fewer calories...

This recent report from the New York Times focuses on California's high school students. About five years ago, the state banned junk foods like soda, chips, and candy from high schools in the state. Now a study has reported that California students eat fewer calories than students in other states.

The study found that California high school students consumed on average nearly 160 calories fewer per day than students in other states, the equivalent of cutting out a small bag of potato chips.

To study the effect of this policy, the researchers examined data from the Centers for Disease Control and Prevention on the eating habits of high school students in California, comparing it with data on students from 14 states that did not have nutrition standards for vending machine snacks and other foods sold outside of school lunches and other meal plans.

a) This study's design seems to be a quasi-experiment. What makes it so?

b) Sketch a graph of the study and its results.

c) What internal validity threats might be present in this study? If you were the researcher, what would you do to control for these threats?

d) Overall, do you think the evidence presented here supports the claim that banning junk food in schools can help kids eat fewer calories?

Suggested Answers

a) This is a quasi experiment--probably a nonequivalent groups, post-test only design. It is a quasi experiment because the independent variable (being in a state that bans junk food in school) is not manipulated. Students cannot be randomly assigned to live in a state with a junk food ban (i.e., CA) or to a state without junk food ban (i.e., the other 14 states).

b) Your sketch would probably have "Calories consumed daily" on the y-axis and "state" on the x-axis. On the x-axis you could plot all 15 states separately (showing CA with the lowest calorie consumption), or you could plot California and an average of the remaining 14 states (showing CA with the lowest calorie consumption).

c) We might worry that compared to the other 14 states in the study, CA students come disproportionately from one ethnic background (for example, there are more Hispanic students there), or that CA and the comparison states may also have different requirements for physical education in school, which might affect eating habits. To find out how the researchers controlled for these factors, you could check the original source in the Archives of Pediatrics and Adolescent Medicine.

d) If, after reading the original report, you were convinced that the junk food ban was the only "active ingredient" separating CA from the 14 comparison states, then you might well be convinced that the junk food ban is playing a role in controlling child and adolescent obesity.

Can ties with teachers prevent drug abuse in kids?

Some adolescents experiment early with alcohol and drugs, a behavior that puts them at later risk for substance abuse and other health problems. A recent study looked at this risky behavior. Journalistic coverage by a Yahoo! news reporter started with this encouraging headline:

Caring Teachers May Help Keep Kids From Trying Alcohol, Drugs

The full story appears here. The basic details of the study were summarized by Yahoo! news as follows:

The researchers found that students in middle school who felt more emotional support from teachers had a lower risk of early alcohol and illicit drug use. The students defined teacher support as feeling close to a teacher or being able to discuss problems with a teacher.

a. What are the two main variables in this study, as described above? Can you tell how they were operationalized? Do you think these two variables were measured or manipulated?

b. Take a careful look. I bet you'll notice a difference in how the researcher, Carolyn McCarty, describes her own study, when she said:

"this is the first data-driven study which shows that teacher support is associated with lower levels of early alcohol use."

...compared to how the Yahoo! journalist describes it in the headline:

"teachers may help keep kids from trying alcohol, drugs."

b. What kind of claim is each person making?

c. Is the journalist's claim justified by the study?

Suggested answers.

a. One variable in this study is "early alcohol and drug use." It's hard to tell how this was operationalized---probably through self-report. Since early alcohol and drug use is almost impossible to manipulate ethically, this is almost certainly a measured variable. The other variable is "emotional support from teachers." This variable appears to be operationalized through a self-report of 'feeling close to a teacher or being able to discuss problems with them'. This is a measured variable, too. They simply asked kids about their relationships with teachers.

b. The researcher is making an association claim (she clearly says "is associated with"). But the journalist's verb choice, "keep from," is a causal verb--making it a causal claim.

c. The journalist's causal claim is not justified by the study. Only an experiment (with one manipulated and one measured variable) can potentially support causation. This study, in contrast, is a corelational study because all of the variables were measured.

Chocolate consumption and BMI

More good diet news from the New York Times, along with a chance to apply lessons from Chapter 8 in the text.

A recent study showed that people who eat chocolate more frequently tend to weigh 5 to 7 pounds less than people who don't.

The people who ate chocolate the most frequently, despite eating more calories and exercising no differently from those who ate the least chocolate, tended to have lower B.M.I.’s.

Importantly, the variable concerning chocolate consumption was about frequency, not quantity. It was the frequency of eating chocoloate, not how much people ate, that was associated with lower body weight.

 

“It’s not the case that eating the largest amount of chocolate is beneficial; it’s that eating it more often was favorable,” Dr. Golomb said. “If you eat 10 pounds of chocolate a day, that’s not going to be a favorable thing.”

Can we make a causal statement here? Can eating chocoloate more frequently cause people to lose weight? Here's what the article says:

Dietary studies can be unreliable, since so many complicating factors can influence results, and it is difficult to pinpoint cause and effect. But the researchers adjusted their results for a number of variables, including age, gender, depression, vegetable consumption, and fat and calorie intake. “It didn’t matter which of those you added, the relationship remained very stably significant,” said Dr. Golomb.

a. In the quoted text you should recognize the telltale signs of multiple regression. As an exercise, create a regression table with a dependent variable indicated up top, and then a list of independent variables that corresponds to what you read above. Then estimate beta weights that might correspond to what the researcher is describing above.

b. Imagine that a friend says, "It's probably not the chocolate. It's just that the frequent chocolate eaters somehow consume fewer calories." What should you say in reply?

Suggested answers:

a. The dependent variable in this analysis is BMI. The independent variable of interest is frequency of chocolate consumption. The other independent variables include age, gender, depression, vegetable consumption, and fat and calorie intake. Your invented beta for the frequency of chocolate consumption variable should be negative and significant (indicating that as frequency of chocolate consumption goes up, BMI goes down, controlling for the other variables in the equation).

(For more fun, you can even check your invented values against the original article, linked here!)

b. Your friend isn't right, because even when the authors controlled for fat and calorie intake, the beta for chocolate is still significant. Chocolate consumption is still related to lower BMI, even taking that variable into account.