Everyday Research Methods

Really? "Sleep this way to add 5 years to your life"

A friendly reminder that when all the variables are measured, causal claims aren't supportable. Photo credit: Sleeping: MicrostockAsia/Deposit Photos

CNN's headline was "sleep this way to add 5 years to your life." This headline should immediately kick in your "correlation is not causation" spidey-sense.

a) What makes this headline causal?

b) What are the two variables in the headline? 

c) Why might you suspect this headline is based on a correlational study (rather than an experiment)?

Now that these preliminaries are out of the way, let's find out if the study behind the headline  can support causation.

Here's the splashy intro:

Want to live longer? Then prioritize sleep in your life: Following five good sleep habits added nearly five years to a man's life expectancy and almost 2.5 years to a woman's life, a new study found.

The article then provides advice such as getting at least 7 hours of sleep, and so on. The journalist links that advice to the following research summary:

"Recent studies have shown irregularity in sleep timing and duration have been linked to metabolic abnormalities ...," he said. "Encouraging maintenance of regular sleep schedules with consistent sleep durations may be an important part of lifestyle recommendations for the prevention of heart disease."

d) In the example above--What are the two variables in the study? What makes the study correlational? 

e) Apply the three causal criteria to the example in d). The study can support covariance (why?). What about temporal precedence? What about internal validity? What specific third variables might be responsible for the link between sleep timing and metabolic abnormalities?

 

Later in the piece, CNN shares data from another study, this one using multiple regression:

The preliminary study, presented Thursday at an annual meeting of the American College of Cardiology, analyzed data from over 172,000 people who answered sleep questionnaires between 2013 and 2018 as part of the National Health Interview Survey. The annual survey is done by the CDC and the National Center for Health Statistics.

 

Each of the five healthy sleep habits — falling asleep easily, staying asleep, getting seven to eight hours of zzz's, waking up rested and foregoing sleep meds — was assigned a number. People were scored on how many of the five habits they had.

 

About four years later, researchers compared those scores with National Death Index records to see if their sleep behaviors contributed to an early death from certain diseases or any cause.

 

The team then factored out other potential causes for a higher risk of dying, such as alcohol consumption, lower socioeconomic status and existing medical conditions.

 

f) Make a table of the variables in this study:

Variable name	How was this variable operationalized?	What are the variable's possible levels?	Is this manipulated or measured?
			(add rows as needed)

 

g) The study directly above meets both covariance and temporal precedence....can you explain why? 

h) Imagine that a critic says, about the study directly above, "Hmmm....it could be that healthy sleep habits predicted early death because of alcohol consumption--maybe people who drink more alcohol are more likely to have unhealthy sleep, and they are also more likely to die early." What do you say? 

 

Finally, if you're looking for an example of a moderator, we've got you:

 

Men who followed all five of the healthy sleep habits had a life expectancy that was 4.7 years greater than people who had none or only one of the five elements of low-risk sleep, the study found.

 

The impact of healthy sleep habits was much lower for women: Those who followed all five sleep habits gained 2.4 years compared with those who did none or only one.

 

i) Put this into a moderator sentence: _____ moderates the relationship between _____ and _____ such that ____.

 

Thanks Stephen Chew for sharing yet another great example from the "correlation is not causation" files.

 

Selected answers

a) The headline is causal because it takes the form of advice: If you do X, it will cause Y (here, If you sleep better, it will cause increased life span)

b) Type of sleep and lifespan are the two variables.

c) You might suspect this headline is based on a correlational study (rather than an experiment) because lifespan is definitely a measured variable (you can't manipulate it), and sleep seems likely to be measured as well. Of course, it's possible to randomly assign volunteers to one of two sleep patterns---you'll want to read the article closely to see!

d) One variable is irregularity in sleep timing and duration; the other is metabolic abnormalities. 

e) The study can support covariance because the results showed a correlation between sleep irregularity and metabolic abnormalities. The study may not have temporal precedence--it's not clear if the variables were measured at the same time, or if sleep irregularity was actually measured first. What about internal validity? It's possible that some pre-existing medical condition (such as depression or addiction) goes with both sleep irregularities and with metabolic abnormalities.  Other third variables might work, too--make sure you explain how the variable you are thinking of links to both sleep irregularities and with metabolic abnormalities.

f)

Variable name	How was this variable operationalized?	What are the variable's possible levels?	Is this manipulated or measured?
Healthiness of sleep habits	Survey; summed five habits (falling asleep easily, staying asleep, etc.)	0 to 5	Measured
Early death	National Death Index records	Alive or dead	Measured
Alcohol consumption	Doesn't say how this was operationalized	Not clear--lo to hi	Measured
SES	Doesn't say how this was operationalized	Not clear--lo to hi	Measured

 

g) It meets covariance because the results show an association between healthiness of sleep habits and early death. It meets temporal precedence because the sleep habits were measured first, and then people were followed for four years, when early death was measured. 

h) This argument doesn't work as a third variable critique because the researchers controlled for alcohol consumption in their multiple regression (The CNN article said, "The team then factored out other potential causes for a higher risk of dying, such as alcohol consumption, lower socioeconomic status and existing medical conditions").

 

i)  Gender moderates the relationship between sleep quality and early death such that the relationship is present for both, but is stronger for men than for women. 

 

Correlates of depression in boys and girls

The study found a moderated relationship. What are the three variables involved? Photo: cozyta/Deposit Photos

This story comes from the website studyfinds.org--a website that covers a wide range of scientific studies (and therefore a great place to practice critical reading of how journalists cover science news).

The StudyFinds journalist describes a newly-published Dutch study on body image and depression in teenage boys and girls. The result is an example of a moderator, so let's practice moderator reasoning.

Here is some of the journalist's description:

The study by researchers at Erasmus University in The Netherlands tracked 4,000 teenagers for several years. Researchers found that young men are a staggering 285 percent more likely to have a severe depressive episode if they were unhappy with their bodies as teens. 

Their female counterparts were also affected, but were 84 percent more likely to suffer the same problems if they had been insecure about their appearance. 

You might be wondering what it means to be "84 percent more likely" or "285 percent more likely?"  These numbers seem huge. You should convert the percentage to a decimal (i.e., 0.84 or 2.85), and then multiply that times some baseline rate. So in the case of depression, let's imagine that for girls, the baseline rate of depression diagnosis is 10 (out of 100)--we'd then assume that the risk for girls who are insecure about their appearance would have a rate of about 18.4 (out of 100). For boys, the baseline rate of depression might also be 10 (out of 100); we'd calculate that the risk for boys who are insecure about their appearance would have a rate of depression diagnosis of about 28.5 (out of 100). 

a) It takes three variables to have a moderator--two for a core bivariate relationship and one more variable as the moderator. Identify the three players in this example:

Core bivariate relationship (2 variables):
Moderator variable:

b) In this example, are the variables manipulated or measured? 

c) We can depict a moderator two ways: with a table or with a figure. Let's do a table first--this will be  similar to Table 8.5 and 8.6. Note that you don't have to use the numbers provided (285% or 84%)--just think about how strong the relationships are, relative to each other. 

On each line, enter one level of the moderator variable	What is the core bivariate relationship and how strong is it? (strong, weak, or zero)

d) We can also depict a moderator with a graph. Sketch a pair of bar or line graphs. Label one graph for boys and one for girls. On the x-axis, put the "body dissatisfaction" variable; put the other variable on the y-axis. If you are making line graphs, use steepness to depict the strengths of the two relationships (which graph will be steeper--boys or girls?). If you are making bar graphs, use the difference between the bars to depict the strength of the relationship (which will be stronger--boys or girls?). 

e) Now write a sentence similar to those in Figure 9.14: "_________ moderates the relationship between _______ and _________, such that ___________."

 

Refining your understanding: A moderator is a relationship changer: it changes the strength or direction of some core bivariate relationship. In the present example, it's also true that there were gender differences in one of the variables--relative level of body dissatisfaction: 

Well over half of teens, 61 percent, experience body dissatisfaction.... Nearly one in three girls and around one in seven boys were insecure about their weight, and around one quarter girls and one in seven boys were dissatisfied with their figure. 

Sometimes students confuse simple differences (i.e., boys are lower than girls on body dissatisfaction) with moderators (i.e., boys with body dissatisfaction are more vulnerable to depression than girls are). Take a moment and study the difference. How many variables are involved in each statement:

boys are lower than girls on body dissatisfaction
boys with body dissatisfaction are more vulnerable to depression than girls with body dissatisfaction are

f) Challenge yourself to create a scatterplot in which body dissatisfaction is on one axis and risk of severe depression is on the other. Use one color of dot to represent boys; pick another for girls. Carefully place dots on this scatterplot to represent the two major findings--that boys are generally lower than girls on body dissatisfaction, AND that boys who do have body dissatisfaction are more at risk for severe depression. (Consult Figure 8.17 in the 4th edition for a model.)

Musical skill: Why does an early start seem to help?

People might start musical training early because they are living in musical families or show early evidence of musical talent. 
Photo credit: Artem Furman, Alamy stock photo

What accounts for musicality? Some people are highly skilled and accomplished musicians, while others express no aptitude. A recent correlational study tried to understand why some people are more musically accomplished. A summary of this research appears on the APS website, and the original empirical article appears in the journal Psychological Science. 

The researchers used a sample of 310 professional musicians as well as a large sample of twins (which they called "amateur musicians"). The twin sample came "from an existing research project, the Study of Twin Adults: Genes and Environment (STAGE)." The twin sample enabled the researchers to estimate genetic and environmental similarities on musicality.

The participants completed measures of musical aptitude (e.g., tests of their accuracy at pitch and rhythm) and musical achievement (e.g., having performed publicly or won prizes). Researchers also asked each person what age they had started learning music. They asked how many hours per week they'd practiced at different points in their lives. The journalist summarized one of the core results:

By comparing the results from these two independent studies, the researchers were able to show that an earlier start age is associated with musical aptitude, both in amateurs and professional musicians...

Questions for students appear in gray throughout this post:
a) Sketch two scatterplots of this result described in the green quoted text above--one scatterplot for professional musicians and one for amateurs. The x and y axes will be the same on the two scatterplots--be sure to label these carefully. Will your scatterplots show positive or negative relationships? 

 

Now, it might make sense that starting musical training at an earlier age is associated with better musical aptitude simply because of total practice time. That is, people who start training earlier have practiced more hours, and practicing more hours is usually associated with more musical skill. In other words, we might sketch this, third-variable proposal:

According to the study, was total practice time a third variable, explaining why starting age was associated with greater aptitude? Consider this full quote from the journalist's summary: 

By comparing the results from these two independent studies, the researchers were able to show that an earlier start age is associated with musical aptitude, both in amateurs and professional musicians, even after controlling for accumulated practice time.

This quote means that even though practice time is usually associated with greater aptitude, this was not the reason why starting earlier was associated with greater aptitude. The results from a regression table might have looked like this:

DV/criterion variable: Musical aptitude
IV/predictor variable:                 Beta
Age starting musical training          -.21**
Total practice time                               .03           
(The betas above are simplified versions of those reported in the original journal article, Table 3, for the musician sample)

Question
b) Write a sentence describing what the beta of -.21** means. 

Therefore, it wasn't total practice time that accounted for the benefit of starting musical training early. On to the next explanation:

[The researchers] then evaluated starting age in a manner that accounted for the genetic data from the STAGE study.  

The results indicate that genetic factors—possibly related to musical interest and talent—have a substantial influence on the age individuals start music practice and their future musical aptitude. When controlling for familial factors, namely shared genetic and environmental influences, such as a home environment that is steeped in music, there was no additional association between an earlier start age and musicality. 

So, in other words, the researchers were proposing this explanation:

And they found results that supported this explanation. The statistical calculations that are used in twin studies are actually beyond the scope of this course and book--so you'll have to trust the researchers here. Suffice it to say that when they controlled for genetic interest or talent, there was no longer an association between age at starting training and musical aptitude.

c) Consider the construct validity of two of this study's variables: Musical aptitude and age at starting training. What questions could you ask to evaluate the construct validity of these measures? 

d) Here is a challenge question: In the empirical journal article (Table 2) the researchers reported that the correlation between between total practice hours and musical aptitude in the professional sample was r = .05, while the same association in the amateurs (i.e., the twins) was r = .30. One reason the correlation was weaker in the professional musician sample might have been restriction of range. Sketch two scatterplots--one for each sample but with the same x and y axes--that illustrates how restriction of range might have limited the correlation in the professional musician sample. 

Claim: "Going to museums and concerts helps you live longer"

Does the study's method support the claim that attending arts events makes you live longer? Photo credit: agefotostock/Alamy

We'll never run out of material in the "correlation is not causation" department. 

This time, a New York Times  journalist led with the headline, "Another benefit of going to museums? You may live longer." 

Let's check out the study behind this causal claim. 

According to the journalist's summary of the empirical article, 

Researchers in London who followed thousands of people 50 and older over a 14-year period discovered that those who went to a museum or attended a concert just once or twice a year were 14 percent less likely to die during that period than those who didn’t.

The chances of living longer only went up the more frequently people engaged with the arts, according to the study, which was published this month in The BMJ, formerly The British Medical Journal. People who went to a museum or the theater once a month or even every few months had a 31 percent reduced risk of dying in that period, according to the study.

a) Starting just from here--what are the two variables they've mentioned so far? Were they manipulated or measured? 

b) Based on what you've read so far, is this study correlational or experimental? 

c) Without reading further, think of a third variable (a "C") that is likely to be correlated with both A (going to museums or concerts) and B (being less likely to die). Any good ideas? 

OK! Time for more information about the study from the journalist:

The study controlled for socioeconomic factors like a participant’s income, education level and mobility, said Andrew Steptoe, a co-author of the study and the head of University College London’s research department of behavioral science and health.

“Even if you take those things into account, we still found that there is a difference in the survival of people who are involved in the arts,” Professor Steptoe said in an interview on Friday.

 

Sounds like multiple regression was used here.

d) What was the criterion (dependent) variable in this study now?

e) What were the predictor ("independent") variables? 

f) Sketch a little regression table similar to those in Chapter 9--put the criterion variable at the top and the predictors at the bottom. Which betas do you think were significant?

g) More importantly, was the "C" variable you thought about in c), above, controlled in their regression?

h) Even though the authors controlled for income, education, and mobility, there are likely to be additional third variables that were NOT controlled for. Think of at least one more alternative explanation. 

 

As you learn in Chapter 9, regression does not support causation because even though it can control for alternative explanations you can think of (such as income, education, or mobility), it doesn't control for everything. If you read the Time's summary, you'll see even more variables they seem to have controlled for , such as marital status and profession. But there could still be unmeasured variables that are responsible for the relationship. 

 

But that doesn't stop the authors from making causal speculations. In fact, here's a mediator hypothesis. i) Sketch this mediator following the diagrams at the end of Chapter 9:

 

The study also noted that engaging in the arts can reduce loneliness, promote empathy and emotional intelligence, and keep people from becoming sedentary — all factors that contribute to a longer life.

 

What kind of study would you need to conduct to test this proposed mediator?

 

 

 

 

 

 

Cuter kids get higher grades

One of the proposed mediators of the effect was that teachers report better relationships with attractive students.Photo credit: FatCamera/Getty Images

File this one under: Depressing, but we suspected it all along. The Washington Post ran a summary of research that looked at the link between school achievement and physical appearance. 

A new study finds that good looking kids do better in school than their less striking peers.

The research, by Barnard College economist Daniel Hamermesh and colleagues, finds that people whose looks are “one standard deviation above average” attain nearly five more months of schooling than an “otherwise identical average-looking individual.”

The study in question found the effect in two large, national studies, one in the U.S. and one in the U.K. You might be wondering, how did they decide how good looking the kids were? What do you think of the construct validity of this operationalization: 

For each kid in the American study, looks were assessed by a panel of at least 10 undergraduates who watched segments of video interviews with the children collected over the course of the survey. The raters assessed each one’s looks on a scale from 1 (not at all cute/very unattractive) to 5 (very cute/very attractive).

For the U.K. study, attractiveness was assessed by the children’s teachers, who assigned each child to one of four categories: attractive, unattractive, “abnormal feature” or “underfed or scruffy and dirty.”

Here is a description of the results:

Ratings in hand, the researchers then analyzed the relationship between a child’s looks and academic achievement, as measured via various standardized tests administered throughout the two studies. The studies revealed the same general pattern: Better looking children performed better on tests of academic achievement, even when controlling for ethnicity, gender and parents’ education and income.

The language above should cue you to realize that this study evaluated the core relationship (between looks and achievement) but used multiple regression techniques to control for these potential internal validity threats. 

a) What is the criterion variable in this study? What are the predictor variables?

b) Sketch a little regression table with results that would be consistent with what is mentioned above. Put the criterion variable at the top, and list the predictor variables. Which betas would be significant here? 

(Note: You can see the actual results in Table 7 of the scientific article.)

The journalist continues the summary with this statement:

Economists who’ve studied academic performance have found that children who perform better on tests tend to get more schooling: A student who aces all her reading and math tests is much more likely to go on to college than one who does not. Hamermesh and his colleagues estimate that good-looking kids’ heightened test performance relative to their average-looking peers works out to about 0.4 years, or five months, of additional schooling.

c) The above is a statement about effect size. Do you think this effect is a large one? Is it important?  What more do you need to know to evaluate this effect? 

d) Next comes a statement about potential mediators of the relationship. Read the following and sketch out one of the mediation paths they are describing (use Figure 9.11 as a model)

To determine why, Hamermesh and his colleagues tested a number of theories. They found some evidence that teachers report better relationships with the more attractive students, which explains some of the achievement gap. They also found that youngsters rated as unattractive were somewhat more likely to report being bullied by their peers, with detrimental spillover on their academic performance.

Similarly, they found that there were fewer reports of behavioral problems at school among the highly rated children.

 

Are there other potential mediators you can think of?  

 

Guess what! This is my 200th post! Thanks for reading and making this blog possible. 

Claim: Mothers' friendships are good for babies' brains

The study showed that moms with larger social support networks had kids with higher cognitive test scores. Photo: Noriko Cooper / Alamy Stock Photo

The  Washington Post headline reads, "Mothers' friendships may be good for babies' brains."  The story is about an empirical study on mothers' social networks. Although it says "babies' brains", the study measured the babies' cognitive skill (there were no baby brain scans). The empirical article came out in JAMA Network Open. Does the original study support the claim? Let's find out.Before reading on, answer these questions:

 

a) What are the two variables in the Washington Post headline claim? What makes the claim a causal one? 

b) Of course, we need to conduct an experiment to definitively support a causal claim. Would an experiment be possible in this particular case?  

Here are some details about the study, as summarized by the journalist (not the researcher):

For the current study, researchers examined data on 1,082 mother-child pairs. They questioned women about their family structure, friendships and relationships in their communities and also looked at test results from cognitive assessments done when children were 2 years old.

Overall, mothers had an average of 3.5 friends in their social support networks. The kids of mothers with more than that tended to have higher cognitive test scores than the kids of those who had fewer, suggesting “network conditions were significantly associated with early cognitive development in children,” the study authors wrote.

c) Sketch a  simple scatterplot of the result described above. What will you put on the two axes? How many dots should your scatterplot have? What does one dot represent? 

Read the following details if you've studied Chapter 9: 

The researchers accounted for some factors that might influence child cognitive development, including mother’s age and IQ, father’s education level, and the child’s birth weight. 

In addition, "researchers also accounted for poverty levels"

d) The paragraph above is describing a multiple regression analysis. What would be the dependent (criterion variable) in this analysis? What are the predictor variables? (Hint: There are six predictor variables).

e) Imagine that someone proposes income as a possible internal validity problem. Given what you know about the study, can we say that "Perhaps friendship network size and child cognitive skills are correlated because of the third variable of income: That is, higher-income mothers may have more friends and more cognitively skilled children"?  Would this criticism be reasonable? Why or why not?

f) The journalist writes: "...they did not have data to assess other personal differences, such as history of depression, between mothers with large networks of friends and those with fewer."
This means history of depression could be a possible third variable explaining the relationship. How could the researchers rule out this third variable? 

The journalist writes: 

All of the families in the study lived in the area around Memphis. It is possible that results might be different elsewhere.

g) Which big validity is the journalist addressing in the statement above? 

h) Do you think the correlation between friendship and cognitive test scores would actually come out differently in a different city or country? Why or why not? 

 

Suggested answers:

a) The two variables are Amount of mothers' friendships and Level of health of babies' brains. The language of "may be good for" makes this a causal headline (remember we ignore adverbs such as "may" when classifying claims). 

b) Probably not; you'd have to randomly assign some mothers to have more friends and others to have fewer friends.

c)  One axis should have "number of friends" and the other should have "cognitive test score". There should be about 1082 dots on the scatterplot (better count your dots now...), and each dot represents a mother-child dyad.

d) The criterion variable is cognitive test score. The predictor variables are number of friends, poverty levels, mother’s age, mother's IQ, father’s education level and the child’s birth weight. 

e) Since the researchers did control for poverty levels, we can actually rule out wealth as a possible third variable. 

f) the researchers could obtain history of depression data on each mother and then add this variable as a predictor variable to the regression analysis. If the beta associated with "number of friends" is still significantly predicting cognitive test score, then we can rule out this possible confound (/third variable/ internal validity problem).

g) external validity

h) In my opinion, it seems unlikely that such a relationship is limited only to Memphis residents. 

Note: You can read the original empirical article in JAMA, here. Note to savvy instructors: The p-curve on the results in this paper suggests that there may be little evidentiary value. 

Which comes first? Cross-lag study on depression and social media use

Which comes first, social media use or depression? Photo:Africa Studio/Shutterstock

We have seen multiple studies that have tried to estimate how strong the link is between social media use and depression among young adults. We've covered some of these studies on this blog, including the effect size of the correlation and even a brain imaging study that manipulated "likes."

One challenge with studying real-world links between social media use and depression is that most use correlational designs. Therefore, we can't be sure whether any link between social media use and depression means that social media leads to more depression, or if depression leads to more social media use. And of course, in correlational studies there may even be a third variable, such as age, which is associated with both variables and causing them to appear correlated. 

The latest study used a longitudinal, cross-lag panel design, such as those described in Chapter 9, to study this question. Here's a writeup of the study by a journalist at Earth.com.

First, here are some comments about the design: 

The researchers point out that previous studies that found links between depression and social media use among tweens are limited in their results because they only captured a snapshot of teen social media use.

“You have to follow the same people over time in order to draw the conclusion that social media use predicts greater depressive symptoms,” said Taylor Heffer, the lead author of the study. “By using two large longitudinal samples, we were able to empirically test that assumption.”

Here are some more details about the method. They actually used two longitudinal samples: 

For this study, the researchers conducted a two-year longitudinal study surveying 6th, 7th, and 8th graders in Canada. The team also surveyed a series of undergraduate students once a year for six years as they progressed through college.

What were the results? 

Higher depression symptoms predicted social media use among teen girls which indicates that unhealthy social media use may be a symptom depression, not a catalyst for it.

“This finding contrasts with the idea that people who use a lot of social media become more depressed over time. Instead, adolescent girls who are feeling down may turn to social media to try and make themselves feel better,” said Heffer.

Questions

a) In a cross-lag panel design, researchers measure the same two variables in the same sample of people over time. What are the two variables they focused on here?

b) Sketch a series of boxes depicting the design of the study, similar to the figures in Chapter 9's cross-lag panel section. How many time periods would you need to depict for the undergraduate sample? 

c) Focus only on the girls. What do you think the autocorrelations might have been in this design? Fill those in on your sketch.

d) Still focusing only on the girls, given the description of the results, estimate what the two cross-lag correlations might have been. Which correlation should be larger--the one from Time 1 depression to Time 2 social media, or the one from Time 1 social media to Time 2 depression?

 

Do kids do better in private schools?

What evidence would it take to convince us that it's the schools, not other factors, that are responsible for the outcomes of private school students? Photo: Image Source / Alamy Stock Photo

A large study has compared the outcomes of children who've attended private schools to those who've attended public schools.  A journalist summarized the report in the Washington Post. The study provides a nice example of how multivariate regression can be used to test third variable hypotheses. 

When we look simply at the educational acheivement of students in private schools vs. public schools, private school students have higher achievement scores. However, all such studies are correlational, because the two variables--Type of School and Level of Achievement--are measured. 

Therefore, such studies how covariance, because the results depict a relationship. The study may even show temporal precedence, because attending school presumably precedes the measure of achievement. However, such studies are weak on internal validity. We can think of several alternative explanations for why children in private schools are scoring higher. 

One major alternative explanation is socioeconomic status. Children from wealthier families are more likely to afford private schools. And in general, children from wealthier families tend to score higher on achievement tests. 

The Washington Post journalist quoted one of study's authors, Robert Pianta, who summed up the study's results this way:

“You only need to control for family income and there’s no advantage,” Pianta said in an interview. “So when you first look, without controlling for anything, the kids who go to private schools are far and away outperforming the public school kids. And as soon as you control for family income and parents’ education level, that difference is eliminated completely.”

Questions

a) Draw little diagrams similar to those in Figure 8.15 (in the 3rd ed.) to depict the arguments being made in this study. What would A be? What about B? In the quote from Pianta, above, what would the C variable(s) be? 

b) The researchers used type of school (private vs. public), which is a categorical variable. But in some analyses, the researchers also used "number of years in private school" as an alternative version of this variable. Is "number of years in private school" categorical, ordinal, interval, or ratio data? 

c) Sketch a mock-up regression table with the criterion variable  at the top and predictors below (Use Table 9.1 as a model). Which variable do you think the researchers selected as the criterion (dependent) variable in their analyses? Which variable(s) would have been the predictors? 

d) Now that you know what the results were, think about how the beta associated with "number of years in private school"  would change when parental SES is added and removed from the regression analyses. 

Original journal article seems to be open-access. Give it a try!

Suggested answers

a) A and B would be Type of School and Level of Acheivement. It doesn't really  matter which one is A and which one is called B. C would be Family Income and/or Parental Education.

b) Ratio data (zero is meaningful in this scale because you could attend zero years of private school)

c) The criterion variable would be Achievement, and the predictors would be Number of Years of Private School, Family Income, and Parental Education.

d)When the Number of Years of Private School is on the table (in the analysis) by itself, its beta is likely to be positve and significant (more years of private school goes with higher achievement). When Family Income and Parental Education are added to the table, the beta for Number of Years of Private School should drop to zero.  This pattern of results is consistent with the argument that Family Income and Parental Education are the alternative explanation for the original relationship .

Delay of gratification (marshmallow study replication)

Her school achievement later in life can be predicted from her ability to wait for a treat (or by her family's SES). Photo: Manley099/Getty Images

There's a new replication study about the famous "marshmallow study",  and it's all over the popular press. You've probably heard of the original research: Kids are asked to sit alone in a room with a single marshmallow (or some other treat they like, such as pretzels). If the child can wait for up to 15 minutes until the experimenter comes back, they receive two marshmallows. But if they eat the first one early, they don't. As part of the original study, kids were tracked over several years. One of the key findings was that the longer children were able to wait at age 4, the better they were doing in school as teenagers. Psychologists have often used this study as an illustration of how self-control is related to important life outcomes. 

The press coverage of this year's replication study illustrates at least two things. First, it's a nice example of multiple regression.  Second, it's an example of how different media outlets assign catchy--but sometimes erroneous--headlines on the same study.

First, let's talk about the multiple regression piece. Regression analyses often try to understand a core bivariate relationship more fully. In this case, the core relationship they start with is between the two variables, "length of time kids waited at age 4" and "test performance at age 15." Here's how it was described by Payne and Sheeran in the online magazine Behavioral Scientist:

The result? Kids who resisted temptation longer on the marshmallow test had higher achievement later in life. The correlation was in the same direction as in Mischel’s early study. It was statistically significant, like the original study. The correlation was somewhat smaller, and this smaller association is probably the more accurate estimate, because the sample size in the new study was larger than the original. Still, this finding says that observing a child for seven minutes with candy can tell you something remarkable about how well the child is likely to do in high school. 

a) Sketch a well-labelled scatterplot of the relationship described above. What direction will the dots slope? Will they be fairly tight to a straight line, or spread out?

b) The writers (Payne and Sheeran) suggest that a larger sample size leads to a more accurate estimate of a correlation. Can you explain why a large sample size might give a more accurate statistical estimate? (Hint: Chapter 8 talks about outliers and sample size--see Figures 8.10 and 8.11.)

Now here's more about the study: 

The researchers next added a series of “control variables” using regression analysis. This statistical technique removes whatever factors the control variables and the marshmallow test have in common. These controls included measures of the child’s socioeconomic status, intelligence, personality, and behavior problems. As more and more factors were controlled for, the association between marshmallow waiting and academic achievement as a teenager became nonsignificant. 

c) What's proposed above is that social class is a third variable ("C") that might be associated with both waiting time ("A") and school achievement ("B"). Using Figure 8.15. draw this proposal. Think about it, too: Why does it make sense that lower SES might go both with lower waiting time (A)? Why might lower SES go with lower school achievement (B)? 

d) Now create a mockup regression table that might fit the pattern of results being described above. Put the DV at the top (what is the DV?), then list the predictor variables underneath, starting with Waiting time at Age 4, and including things like Child's Socioeconomic Status and Intelligence. Which betas should be significant? Which should not?

Basically, here we have a core bivariate relationship (between wait time and later achievement), and then a critic suggests a possible third variable (SES). They used regression to see if the core relationship was still there when the third variable was controlled for. The core relationship went away, suggesting that SES was a third variable that can help explain why kids who wait longer do better in school later on. 

Next let's talk about some of the hype around this replication study. The Behavioral Scientist piece (quoted above) is one of the more balanced descriptions. Its headline was, Try to Resist Misinterpreting the Marshmallow Test. It emphasized that the core relationship was replicated. It also explains in some detail why SES is related to self-control, and how the two probably cannot be meaningfully separated--it's a nuanced report. But other press coverage had a doomsday feel:

Vox's subtitle read:
The famous psychology test gets roasted in the new era of replication.

The Guardian blared:
Famed impulse control 'marshmallow test' fails in new research

And the Quartz website said:
We learned the wrong lesson about self-control from the famous marshmallow test

One person on Twitter even wrote,
"The marshmallow/delayed gratification study always felt "wrong" to me - this year it was reported to be hopelessly flawed"

Are these headlines and comments fair? Probably not. As Payne and Sheeran write in Behavioral Scientist, 

The problem is that scholars have known for decades that affluence and poverty shape the ability to delay gratification. Writing in 1974, Mischel observed that waiting for the larger reward was not only a trait of the individual but also depended on people’s expectancies and experience. If researchers were unreliable in their promise to return with two marshmallows, anyone would soon learn to seize the moment and eat the treat. He illustrated this with an example of lower-class black residents in Trinidad who fared poorly on the test when it was administered by white people, who had a history of breaking their promises. Following this logic, multiple studies over the years have confirmed that people living in poverty or who experience chaotic futures tend to prefer the sure thing now over waiting for a larger reward that might never come. But if this has been known for years, where is the replication crisis?

e) Give at least 2 reasons why online sources might use incorrect headlines when they cover psychological science. 

f) Check out at least two different press stories about the marshmallow replication effect, and see how similar they are to each other and to the summary you analyzed above, by Payne and Sheeran. 

You can read the published paper for yourself! The full replication paper is in Psychological Science and is open access. 

Engagement with Disney Princesses

The study's results showed that the more children were engaged with Disney Princess media, the more they adhered to stereotypically female play later on. Photo: Shutterstock

There's a lot to think about in a recent study on Disney Princess Media. The study was covered by the Huffington Post here, and by another source in Oregon.  (You'll find other press coverage if you search for it).  The study illustrates some of the concepts in Chapter 8 and 9.

Let's point out the causal claims made by the journalists first. The Huffington Post journalist's headline reads:

Disney Princesses May Impact Gender Stereotypes For Girls (But Not Boys), Study Finds

And the Oregon paper's headline reads:

Disney 'princess culture' damaging to young girls, study says

Did you notice the causal verbs, "impact" and "damaging"? That's probably inappropriate, because this was a correlational study. In the first paragraph, the HuffPo journalist continues the dramatic causal language: 

Brigham Young University family life professor Sarah M. Coyne conducted a study with 198 preschoolers, both male and female, to rank their interaction with Disney Princess culture — including toys and movies — and how that affected their behavior one year later with reports from parents and teachers. 

a) Look back at that quote and find the verb that makes the causal claim. 

 

Now let's talk about the moderator in the headline, Disney Princesses May Impact Gender Stereotypes For Girls (But Not Boys).

b) This headline describes a moderator. Review what you learned about moderators in Chapter 8. What is the core relationship in this example (what are the two core variables)? And what is the moderator variable? How would you sketch this headline in a table or figure? 

 

Now, here are some more points made about the study, quoted from the Oregon news source:

Linder and her co-authors found that 96 percent of girls and 87 percent of boys had taken in some form of princess culture. Roughly 61 percent of girls reported playing with a princess doll in the last week, but only 4 percent of boys said they had played with the dolls.

When the subjects were examined a year after the initial analysis, both girls and boys who played with the dolls showed an increased likelihood to adhere to female gender stereotypes, a fact that could prove especially problematic for girls, said Sara Coyne, a co-author of the study and professor of family life at Brigham Young University.

Here's how the HuffPo journalist described this part of the results:

For both boys and girls, greater interactions with princesses predicted more gender-normative behavior — like wanting to play with traditional female stereotypical toys and activities (e.g., dolls, tea sets, playing house), and behaviors (e.g., avoiding getting dirty, avoiding taking risks)

c) What are the variables in this study, as described above? Are they measured or manipulated? And given your answer, does that make the study correlational or experimental? Did you notice that the design is longitudinal? (What signaled you to that?) Which design in Chapter 9 does this design seem closest to? 

d) Now, do these descriptions of the results actually show a relationship between Disney Princess media and gender-normative behavior that is different for boys and girls, as the one headline suggested? 

 

OK. If the relationship between Disney culture and stereotypical behavior was actually the same for boys and girls, where did the moderator in the headline come from? Turns out that when the researcher was interviewed about the results, she indicated that the gender stereotypes could, in the future, lead to different outcomes for boys and girls. Therefore, the moderator in the headline came more from the researcher's discussion than from the study itself. The researcher said, for example:

"We know that girls who strongly adhere to female gender stereotypes feel like they can't do some things," she told the university's news site. "They're not as confident that they can do well in math and science. They don't like getting dirty, so they're less likely to try and experiment with things."

And for boys:

The effects of princess culture were less troubling for boys, the authors said, as more stereotypically feminine behavior could serve as a counterbalance to the hyper-masculine superhero culture many young boys take in on a daily basis.

 

If your college or university subscribes to the journal, Child Development,you can see the full report of the study and learn more about the full pattern of results!