Everyday Research Methods

Should scientists change out of pajamas?

Does the study's design support the causal claim the journalist attached to it?
Photo credit: R.O.M/Shutterstock

Some journalists simply cannot resist attaching a causal claim to a study where it doesn't belong. Here's an example from the trade magazine, Inside Higher Ed, which reprinted a piece from Times Higher Education. The headline reads, "Scientists urged to change out of pajamas." 

The first paragraph begins:

Academics who are tempted to remain in pajamas during the working day should think again, says an Australian study that has linked the practice to a deterioration in mental health.

 

a) There are actually two claim-like statements in the first paragraph. What are they? (Remember that advice is a form of causal claim). 

b) What seem to be the two variables in the association and causal claims above? 

The journalists summarizes the study this way: 

The study, which drew on 163 responses from staff at five medical institutes in Sydney, also delved into the reality of working at home for researchers. [...]

Some 28 percent of scientists said they wore pajamas at least once a week -- a cohort who were twice as likely to report worsened levels of mental health than those who dressed normally each day, according the study, by David Chapman and Cindy Thamrin [...]

Given the survey format of this study, we can assume that all the variables in this study were self-report. To learn more about this relatively simple study, you can visit the open-access original journal article here, and you can see the full, original text of the survey here. 

Let's work through the four big validities. 

c) Construct validity: Look through the original survey (here) and find the variables that measured pajama-wearing and mental health assessment. What do you think--how well do these variables seem to be measured? (Guess what? You're assessing face validity.) 

d) Construct validity: When you view the original text of the survey (still here)  you might be a bit surprised by the lighthearted tone of some the survey questions. For example, they ask about the "typical home working environment" they included the option, "hiding in the bathroom." When they asked people what they wear during remote meetings, one of the options was, "none of your business, camera turned off."  How might this casual tone affect the construct validity of the variables being measured? Do you think it will it affect the accuracy of self-reports?

e) External validity: The journalist refers to "scientists", so that seems to be the population of interest. The sample was described this way by the journalist:

The study, which drew on 163 responses from staff at five medical institutes in Sydney...

In the empirical article, there is more detail about the sample: 

An invitation to participate was emailed to all staff, students and affiliates of the Woolcock Institute of Medical Research, Sydney, and later extended to other medical research institutes in Sydney (Garvan Institute, Children’s Medical Research Institute, Centenary Institute, Brain and Mind Centre).

Can this study generalize from this sample to the population of interest? Why or why not? (Don't be tempted to focus on the size of the sample here. Remember that it's not the size that matters for external validity)

f) Statistical validity: Here is the effect size of the relationship between wearing pajamas and mental health: 

[people who wore pajamas at least once per week] were twice as likely to report worsened levels of mental health than those who dressed normally each day

In the empirical article, the rates of poorer mental health were described as 59% (for pajama-wearers) vs. 26% (for non-wearers). What do you think of the strength of this effect? What more information would you like that is relevant to statistical validity?

g)  Internal validity. Let's evaluate whether the study can support the causal claim, "wearing pajamas put the scientists at risk for poorer mental health?" In order to support a causal claim, we need to conduct an experiment. Was this an experiment or a correlational study?  Explain your answer. 

h) Internal validity: Now let's apply the three criteria for causation. 

The study does show covariance, because people who report working in pajamas did have twice the rate of mental health decline, compared to those who did not.

What about temporal precedence? Does the method establish which variable came first in time? 

What about internal validity? Can you think of a third variable (some "C" variable) that might be associated with both wearing pajamas and having worse mental health? 

Now make a conclusion--does the study support the advice to "change out of pajamas?"

i) How might you design a true experiment to study the potential causal effect of pajamas on mental health? What variable would you need to manipulate? What variable would you need to measure? 

 

 

 

Evidence-based tips for procrastination

What do the studies suggest about why people procrastinate? Photo credit: Timothy Hodgkinson/Shutterstock

The BBC magazine, Science Focus, collected a set of studies on procrastination that might help students at the beginning of a new school year.  The journalist opens with a couple of simple frequency claims. 

[If you procrastinate,] you’re not alone: an estimated 20 per cent of adults (and above 50 per cent of students) regularly procrastinate.

Then the article addresses three topics: 1. Is procrastination a sign of poor time management? 2. Is procrastination unhealthy? 3. How can we stop? Let's take each topic and apply some research methods content as we go.

1. Is procrastination a sign of poor time management?

The journalist quoted psychologist Fuschia Sirois, who explained that contrary to what many people believe, procrastinators aren't more likely to be bad at time management. Instead, she explains, it's about emotion regulation:

“At its core, procrastination is about not being able to manage your moods and emotions. Although many think impulsivity and self-control are the problems – and they do play a factor – underneath is a poor emotional response.”

As Sirois explains, every person faces stressful situations, demanding tasks that trigger brain activity that involves a brain region known amygdala. And it’s the amygdala that processes emotions and signals threats, capable of prompting a ‘fight or flight’ response linked to procrastination.

“Interestingly, people who say they are chronic procrastinators tend to have larger grey matter volume in the amygdala,” says Sirois.

“This means they will also be more sensitive to the potential negative consequences of their actions, leading to more negative emotions and procrastination.”

a) What were the variables in the amygdala study? Was this study correlational or experimental? 

b) Sketch a scatterplot (or bar graph) of the results Dr. Sirois describes here.

 

2. Is procrastination "bad for your health"?

The journalist implies causality with this headline, and with lines like these: 

...procrastination can cause a lot more problems than missed deadlines. Over decades Sirois has examined the impact of chronic procrastinating on a person’s health, her findings worrying at best – and downright terrifying at worst.

Let's see if the evidence can support these causal claims. He quotes Dr. Sirois again, who stated, 

“People who chronically procrastinate – people who make it a habit – have higher levels of stress and a greater number of acute health problems. They are more likely to have headaches or insomnia or digestive issues. And they’re more susceptible to the flu and colds.”

Even more alarming, Sirois has found that procrastination is a factor that can lead to hypertension and cardiovascular disease, with chronic procrastinators more likely to put off healthy behaviour such as exercise.

c) Is Dr. Sirois describing correlational or experimental evidence here? (Explain your answer by specifying the variables she mentions).

d) Why can't the evidence described by Dr. Sirois support the journalist's assertions about "the impact of chronic procrastinating"? Specify the temporal precedence problem and the internal validity problems here (when you discuss internal validity, specify a particular third, or "C" variable that might correlate with both A and B). 

3. How can we stop procrastinating?

In this section, the journalist describes some intervention studies targeting procrastination.

For example, one compelling Psychological Science paper described how downsizing larger metrics of time (think 48 hours instead of 2 days, or 10,950 days instead of 30 years) can make events seem more immediate, prompting people to engage in upcoming tasks.

You can follow the underlined link in the quoted text (above) to see that this paper described several experiments (for example, the abstract states, "we manipulated time metric".

e) Given the journalist's description (and the empirical study's abstract), name the independent variable in this experiment and name its levels.

f) Name the dependent variable in this experiment.

g) Sketch a bar graph of the study's result, as described by the journalist. 

h). Because this study was an experiment, it is more likely to support causation.
Think about something you have to do in the near future. How can you apply the results from the Lewis and Oyserman study in Psychological Science to that task, to prevent yourself from procrastinating on it? 

 

Selected Answers

a) One variable was "saying you are a chronic procrastinator or not" and the other was "volume of grey matter in the amygdala". Both variables must have been measured, so this was a correlational study.  

b) One axis should have "grey matter volume in amygdala" and the other should have "level of chronic procrastination"; the scatterplot should have a positive slope, but you don't have information on how strong the relationship was, so you can't really know how spread out the dots should be. 

c) This is likely correlational evidence. They are measuring (not manipulating) people's procrastination habits. They are measuring (one cannot manipulate) people's acute health problems, susceptibility to the flu, hypertension, and cardiovascular disease. 

d) We can take acute health problems as an example. Even if there is an association between chronic procrastination (A) and acute health problems (B), we still don't know if the procrastination led to the health problems (A to B), or if the health problems lead to more procrastination (B to A). In addition, there could be some outside variable such as life stressors or poverty (C) that is associated with both procrastination (A) and health problems (B). 

e) The independent variable is apparently "time metric" and the levels were hours vs. days, or perhaps days versus years. 

f) The dependent variable seems to be "how immediate events seem to be"

g) The x-axis should have hours and days, and the y-axis should indicate "how immediate events seem to be". The bar for "hours" should be higher than the bar for "days". 
This study seems to suggest that if you have homework due in two days, you might be more likely to get started on it if you say to yourself, "My homework is due in 48 hours." 

 

Really? Can "hot baths protect your heart?"

The data came from Japan, where hot baths (whether at home, in local bathhouses, or in hot springs) are a well-loved practice. Photo credit: nitis.s/Shutterstock

Sigh....even the imprimatur of Harvard Medical School doesn't stop writers from tacking causal headlines onto correlational  studies. 

In this example, the journalist writes, "The study is only observational [i.e., correlational] and doesn't prove that daily tub bathing staves off heart problems".  Yet the headline, "Can hot baths protect your heart?" sounds pretty causal to me. 

Here's the journalist's summary of the method:

Researchers analyzed self-reported health and lifestyle information from more than 30,000 middle-aged people in Japan. Participants responded to a questionnaire at the start of the study and were then followed for about 20 years.

And here's the summary of the major results:

Compared with people who didn't take a tub bath more than twice a week, people who took a daily warm or hot bath had a 28% lower risk of cardiovascular disease and a 26% lower risk of stroke. 

This article provides a good opportunity to practice the four big validities. 

a) Name the three major variables in the study, as included in the journalist's summary. 

b) How do you know that the study was correlational? (remember to focus on whether the variables were measured or manipulated)

c) Ask at least one construct validity question about this study. (Hint--start your question with "How well...")

d) Now ask an external validity question about this study. (Reminder--your question should address sampling method, not sample size)

When you consider external validity, it might be relevant to consider that Japan has a well-developed bath culture. Most homes have soaking tubs, and some towns still support a neighborhood sento, or public bathhouse. In addition, many tourist areas and resorts feature beautifully landscaped hot spring baths called onsen. In this study, 2/3 of the participants reported that they bathe in a tub almost daily. What might this bathing culture suggest about the external validity of this study? Would this relationship generalize to a less-bathy culture like the U.S.?  

e) What about statistical validity? The authors report two estimates: One estimate concerns the link between bathing and cardiovascular disease (28%), and another concerns the link between bathing and risk of stroke (26%). The journalist didn't provide 95% CIs--but what does the sample size tell you about the 95% CI's of these estimates? (i.e., will they be less precise or more precise?)

f) Now let's walk through why this study can't support a causal claim. This study does have covariance (baths were associated with lower risk of cardiovascular disease and stroke).

What about temporal precedence--what is this, and why does this study achieve it?

And what about internal validity? Come up with a third variable ("C") that is reasonably associated both with taking more baths ("A"), and with having lower rates of disease ("B"). 

Bonus: The empirical article is open-access and you can read it here. Instructors--it's likely that students would understand this article well enough to read it and critique it.

 

Claim: "Drinking any amount of alcohol causes damage to the brain"

Photo credit: Dangubic/Deposit Photos

Here's a link to CNN's coverage of a study on alcohol use and gray matter in the brain. The headline reads, "Drinking any amount of alcohol causes damage to the brain, study finds." Let's consider how well the study supports the causal claim the journalist (or the editor) attached to it.  Here's one of CNN's introductory statements: 

In an observational study, which has not yet been peer-reviewed, researchers from the University of Oxford studied the relationship between the self-reported alcohol intake of some 25,000 people in the UK, and their brain scans.

a) The term, "observational study" in this context is synonymous with "correlational study." What are the key variables in the study? What does the quote above tell you about whether the variables were manipulated or measured? 

Here's the next paragraph in the CNN summary: 

The researchers noted that drinking had an effect on the brain's gray matter -- regions in the brain that make up "important bits where information is processed," according to lead author Anya Topiwala, a senior clinical researcher at Oxford.

"The more people drank, the less the volume of their gray matter," Topiwala said via email.

b) The CNN journalist erroneously wrote, "drinking had an effect on the brain's gray matter".  Let's walk through whether this correlational study here can support the journalist's causal claim. The first criterion is covariance, which is about the study's results. Does the study show covariance?

c) The second criterion is temporal precedence. The journalist doesn't provide information about whether the drinking behavior (variable A) was measured first in time, before brain scans measured people's gray matter (variable B). What are the two potential directions this relationship might go?

d) The third criterion is internal validity. There could be some third variable, C, which is associated with both A and B. See if you can come up with a potential C variable that could reasonably correlate with both amount of drinking and reduction in gray matter. 

You might have noticed that the opening to the CNN summary mentions that the study "has not yet been peer-reviewed". 

e) What happens during the peer review process? Do you think journalists should summarize, for the general public, studies that have not yet been peer-reviewed?

f) The empirical report has not yet been published, but you can read more about the study where it is posted on the preprint site, medarxiv. What are some pros and cons of putting one's research on "arxiv" sites like these? Consider issues such as openness of science, peer review, and public interest. 

 

Thanks again, Stephen Chew, for sharing the example!

Interrogating Buzzfeed: Does Instagram really have "huge effects" on our psychology?

Do the studies really support cause and effect? And are those effects "huge"? Photo: Julia_Tim/Deposit Photos

This Buzzfeed headline is eyecatching and dramatic: Instagram Has Some Pretty Huge Effects On Our Psychology. Here's Everything We Know So Far.

I see two important issues to interrogate in this headline. First, it's a causal claim ("Instagram has some...effects on our psychology"). Second, it's making an argument about effect size ("pretty huge effects"). Let's see if they pan out. 

The article focuses on body image and mental health. Here's an example:

The research into how Instagram affects our psychology is relatively new, but there is a growing body of evidence to say that it has a profound effect on the way we perceive ourselves and the world around us.

Does the research support such a causal claim? Here's one study that the journalist summarized: 

Earlier this year, the UK's Royal Society for Public Health (RSPH) and the Young Health Movement published a report that looked at the impact of different social media platforms on mental health.

The report was based on a survey of over 1,400 British 14-to-24-year-olds that asked about the positive and negative implications of each social media platform for health and wellbeing.Instagram scored well on its ability for self-expression, but it was linked with some of the worst scores for sleep loss, body image concerns, and FOMO.

It's not completely clear from this summary what questions were included on this survey. However, a survey is unlikely to have manipulated exposure to Instagram, so it can't support the claim that "Instagram has implications for how people perceive themselves and the world around them."

What about the effect size of the associations? The journalist didn't report the data from the study, so we can't evaluate whether or not the effect was really "huge" or "profound." We'd have to search out the original paper for that. 

How about this other study, summarized later in the article: 

In a study published last year, Cohen and her co-authors found that it wasn’t so much social media use itself that impacted how people perceived their bodies but rather how they were using it.

Following body- and appearance-focused accounts on Instagram — as opposed to neutral accounts dedicated to things such as travel — was associated with idealizing thin bodies, obsessive surveillance of one’s own body, and striving for thinness.

a) Four variables are listed in the study above (Following body-....) What are they?

b) Were these variables measured or manipulated, do you think?  (Check your hunches here, by reading the abstract of the Cohen study.)

c) Why can't this study support a claim such as "following body-focused accounts impacts how people perceive their bodies"? Apply the three criteria for causation. 

d) Does the journalist provide any indication of the effect size of this association? If not, what data might have helped you decide if the link was "huge"?

Let's read a bit further in the journalist's article to see if we can find some experimental evidence. How about this one:

However, Cohen is hopeful that the positive body movements emerging on the platform will have far-reaching benefits for how women reflect on their own physiques.

She has recently submitted a paper for publication demonstrating that among 195 young women exposed to “body-positive” images, mood, body satisfaction, and body appreciation were all improved significantly.

The wording in the example above suggests that there was a manipulated variable, because it reads "among 195 young women exposed to 'body-positive' images'". When I read that, I picture that some people were exposed to these images, and some were not. 

e) Imagine you were going to conduct a conceptual replication of this experiment. What was the independent variable? What would its levels be? 

f) What would the dependent variable(s) be? (Hint: The journalist mentions three).

g Sketch a little bar graph of the results of your experiment--picking one of the DVs to graph. In your view, how far apart would the means and error bars have to be for you to be convinced that the effect is "huge"?

h) Would this experiment have the potential to support a causal claim? What confounds would you need to control for? 

Suggested answers to selected questions

c) Covariance: Yes--apparently this questionnaire study found a link between following more body-focused accounts and idealizing thin bodies more. 

Temporal precedence: Nope--this was a questionnaire study that measured both variables at the same time. We can't know if the IG accounts came first, or if people who idealize thin bodies more are more likely to want to follow body-focused accounts.

Internal validity: Also nope. In this correlational study we can't rule out every possible third variable. For example, maybe people with a history of eating disorders are both more likely to follow body-focused accounts and are more likely to idealize thin bodies.

d) No correlation was reported here--too bad! It would be nice if the journalist had given us some benchmarks indicating how much more likely people were to idealize thin bodies....perhaps a percentage change or a correlation coefficient, and a 95% CI for that change or coefficient. 

 

Thanks to UD student Emily Hogan for suggesting this article for the blog!

 

 

Too sleepy to choose that major

These students might misattribute their early-morning fatigue to their interest in the subject. Photo credit: Gorodenkoff/Shutterstock

Podcasts have been gaining in popularity in the last few years, and several feature psychological research. I'm going to be devoting my next few posts to some podcasts. We'll start with this very short, 3-minute episode/ interview from the NPR podcast, Hidden Brain. The episode is called "How does the way you feel shape your life?" and it's about how students might be influenced by situational factors (such as how early a class is) when they select their majors. 

The podcast opens with a story:

...behavioral economist Kareem Haggag at Carnegie Mellon University... sometimes teaches early morning classes. And he's worried that when his students are tired or sleepy, they might draw the wrong conclusion about him and about the class.

The researcher himself states (around minute 1)

DR. KAREEM HAGGAG: Our hypothesis was that students who are assigned to an early morning section of the class, or to multiple back-to-back classes before a class, might mix up how tired they are in that class with how much they like the subject, thus leading them to be less likely to choose the subject as their major.

The host, Shankar Vedantam, reports:

This is a psychological phenomenon called misattribution. You know, you go to an amusement park on a sunny day and you think the park is great. You go when it's raining - you think the park is terrible. You're unconsciously confusing your feelings about the weather for the quality of the park.

The researchers used thousands of data points from students at the U.S. Military Academy at West Point to test their hypothesis. As Vedantam points out:

... students at West Point are randomly assigned classes that are scheduled at different times of the day, which means that some students are taking this introductory class first thing in the morning. Some students are taking the very same class later in the day. The researchers then measure the chance that students would later on choose that subject as their college major.

The researchers, led by Dr. Haggag, found that students were more likely to major in the topic of that introductory class if they'd taken it at a later time of the day: 

HAGGAG: We find that students who are randomly assigned to the first period, 7:30 a.m. section, are about 10% less likely to choose the corresponding major compared to a student who takes that class later in the day. 

Questions

a) What are the two main variables of interest in this study (the independent and dependent variables)? What are the levels of each variable?

b) Is this a true experiment or a quasi-experiment? If it's a true experiment, what kind is it (prettest-posttest, posttest-only, repeated measures, concurrent measures)?

If it's a quasi-experiment, which of the four types is it (nonequivalent control groups, nonequivalent control group prettest-posttest, nonequivalent control groups interrupted time series, nonequivalent control groups interrupted time series)?

c) Which is the IV and which is the DV here?

d) Sketch a small bar graph of the results they describe.

e) Can these researchers support the causal claim that "having an early morning introductory class makes you less likely to want to major in that discipline?" Apply the three rules of causation: covariance, temporal precedence, and internal validity in your response.

f) Ask a question about this study's internal, external, statistical, and construct validity.

Bonus content:

In the study above, the main IV was "time of day of the introductory class". As a bonus, here is another independent variable they studied--"number of breaks before the introductory class":

DR. KAREEM HAGGAG: We also compared two students who are sitting in the exact same classroom, but one of whom just had a free period as a break before, and the other came from one or more back-to-back classes. We find that each additional back-to-back class reduces the likelihood that that student enrolls in the major by about 12%.

g) Which is the IV and which is the DV here?

h) Sketch a small bar graph of the results they describe.

i) Can these researchers support the causal claim that "having breaks before your introductory class makes you more likely to want to major in that discipline?" Apply the three rules of causation: covariance, temporal precedence, and internal validity.

 

 

Empathy and social justice

The more privilege people have, the less empathy they have for people lower in status. But that can change with increased perspective-taking. 
Photo credit: Clay Banks/Unsplash

The spring of 2020 has seen sustained, worldwide, democratic protests, led by Black Lives Matter and demanding social justice and police reform in the U.S.  After this spring, American support for Black Lives Matter and police reform are suddenly higher than it has ever been.  Psychologist Jamil Zaki asks in this opinion piece, Why now? Why did Americans react more strongly in 2020, after police killed George Floyd, compared to other highly publicized instances when police killed other unarmed Black people, including Breonna Taylor, Tamir Rice, Philando Castille, Michael Brown, and Stephon Clark?  

Dr. Zaki points to empathy as the answer. He writes, 

In the past few weeks, many people have opened their eyes to suffering they had previously ignored. Much credit for this should go to the activists and organizers who have made it harder to look away. 

Dr. Zaki supports his argument with evidence from psychological research on empathy, power, and structural change. For example, teams of psychological researchers have investigated how being in powerful positions is associated with being less able to understand others' suffering:

In one series of studies, the psychologist Michael Kraus and his colleagues measured people’s socio-economic status, as well as their ability to decipher emotions in pictures and in-person interactions. People higher in status were less accurate about other people’s feelings. More recent work has replicated these results and also found that high-status individuals make more errors when trying to take other people’s perspective.

a) What are the variables in Dr. Kraus's research? Were these variables measured or manipulated? (Hint: There are two studies--take one at a time.)

b) Given your answer to question a), were these studies correlational or experimental?

Here are two more brief research descriptions (the links go to the original empirical studies): 

Higher-status individuals display less interest when talking with strangers, and report less concern for the suffering of others. These gaps play out in racial contexts as well. In another study, Kraus found that high-income white Americans overestimate racial economic equality more than black Americans or low-income white Americans.

c) What about the studies above--what are the variables? Are the studies correlational or experimental? (Hint: There are two studies--decide the variables in each study separately.)

Zaki argues that one reason higher status people have less empathy is because the powerful are less motivated to think about others. In contrast, less-powerful people need to pay attention to others in order to function well in society. He writes, "This is one way privilege works its way into our minds. Not only are privileged people exempt from material struggles, they can comfortably ignore everyone else’s."

Here are two more brief study descriptions that Zaki uses in support of his argument:

[Power] expands the change a person could make while narrowing the aperture of who they truly see. But this is not inevitable. When powerful people choose to empathize, they become more cooperative and more invested in justice. In one particularly relevant series of studies, Emile Bruneau and his colleagues asked members of low-power groups to “perspective give,” sharing their stories, and high-power individuals to perspective take, paraphrasing what they’d heard. These dialogues increased connection and positive regard between groups — not by ignoring existing power structures, but by reversing them.

d) In the study by Bruneau and colleagues, what were the variables? Can you tell if the variables were manipulated or measured?  

e) What do you think--was Bruneau's study an experiment or a correlational study? Can it support the causal claim that "these dialogues increased connection and positive regard between groups?"

f) Dr. Zaki's descriptions of the studies are short on detail, likely the result of working under a word-limit and writing for a popular audience. Try following the links to one of the studies linked in Dr. Zaki's article. Are you able to access the full text of the study easily? If not, can you access the journal using your university's library? When you read the full text of the original study, were you correct about its correlational or experimental nature? 

 

 

Saturated fat can impair concentration

Breakfast of (concentration) champions? Maybe not....Photo credit: Kyndall Ramirez/Unsplash

A headline at Psych Central reads, "High fat meal may impair concentration." Does the study behind the headline support that causal claim?

At first, you might assume that this came from a correlational study on diet and concentration. However, it turns out the research summarized in the article was experimental: It manipulated the saturated fat content of one specific meal. In the study, researchers created two versions of a breakfast--one made with saturated fat and the other made with sunflower oil, which is also high in dietary fat but is not the saturated type. 

The journalist describes the method of the study this way: 

Women in the study [n = 52] completed a baseline assessment of their attention during a morning visit to the lab. The tool, called a continuous performance test, is a measure of sustained attention, concentration and reaction time based on 10 minutes of computer-based activities.

The high-fat meal followed: eggs, biscuits, turkey sausage and gravy containing 60 grams of fat, either a palmitic acid-based oil high in saturated fat or the lower-saturated-fat sunflower oil. Both meals totaled 930 calories and were designed to mimic the contents of various fast-food meals such as a Burger King double whopper with cheese or a McDonald’s Big Mac and medium fries.

Five hours later, the women took the continuous performance test again.

Between one and four weeks later, they repeated these steps, eating the opposite meal of what they had eaten on the first visit.

After eating the meal high in saturated fat, all of the participating women were, on average, 11 percent less able to detect target stimuli in the attention assessment.

(The original empirical journal article is here; it might be paywalled)

Questions:

a) What was the independent variable (IV) in this study and what were its levels?

b) What was the dependent variable?

c) Was the IV manipulated as independent groups or within-groups?

d) This design is difficult to classify because it has a pretest and posttest, but the women also experienced both levels of the IV. Therefore it shares features with two of the four designs in Chapter 10. What are those two designs it shares features with? (Note to students: As you can see in this example, the four designs you learned about in Chapter 10 are not the only types of experiments there are, and this study provides an example of alternative designs.) 

e) Sketch the results of this study--using "reduction in concentration" (i.e., a difference score) as the dependent variable. 

Next let's work through the four big validities. 

f) Here's some text related to external validity of this study:

For this work, Madison [the lead author] conducted a secondary analysis of data from Kiecolt-Glaser’s study assessing whether high-fat meals increased fatigue and inflammation among cancer survivors.

and

[the experimental meals]  were designed to mimic the contents of various fast-food meals such as a Burger King double whopper with cheese or a McDonald’s Big Mac and medium fries.

Given the information above, to what kinds of people and what kinds of situations might this research be generalized? How do you know? 

g) The researchers found an 11 percent drop in concentration after the meal high in saturated-fat. What further information would you like to know in order to evaluate the study's statistical conclusions? 

h) Can this study support the causal claim that "A high-fat meal may impair concentration"? Apply the three criteria to the study: Covariance (what did the results show?), temporal precedence (did the method ensure that the meal came before the drop in concentration?), and internal validity (What potential confounds did they control for in the study?) 

i) Here's some information about the study's construct validity: 

Women in the study completed a baseline assessment of their attention during a morning visit to the lab. The tool, called a continuous performance test, is a measure of sustained attention, concentration and reaction time based on 10 minutes of computer-based activities.

What questions would you need to ask to determine if this test is a good measure of people's ability to concentrate? 

Instructors: If you teach Chapter 12, you can also use this example as a mixed factorial design, because the study also included a participant variable of endotoxemia, a condition called "leaky gut." From the journalist's summary: 

After eating the meal high in saturated fat, all of the participating women were, on average, 11 percent less able to detect target stimuli in the attention assessment. Concentration lapses were also apparent in the women with signs of leaky gut: Their response times were more erratic and they were less able to sustain their attention during the 10-minute test.

“If the women had high levels of endotoxemia, it also wiped out the between-meal differences. They were performing poorly no matter what type of fat they ate,” Madison said.

Students could identify the IV, PV, and DV and practice graphing the results described above. 

Thanks to my colleague Stephen Chew, who shared this example.

Claim: "Going to museums and concerts helps you live longer"

Does the study's method support the claim that attending arts events makes you live longer? Photo credit: agefotostock/Alamy

We'll never run out of material in the "correlation is not causation" department. 

This time, a New York Times  journalist led with the headline, "Another benefit of going to museums? You may live longer." 

Let's check out the study behind this causal claim. 

According to the journalist's summary of the empirical article, 

Researchers in London who followed thousands of people 50 and older over a 14-year period discovered that those who went to a museum or attended a concert just once or twice a year were 14 percent less likely to die during that period than those who didn’t.

The chances of living longer only went up the more frequently people engaged with the arts, according to the study, which was published this month in The BMJ, formerly The British Medical Journal. People who went to a museum or the theater once a month or even every few months had a 31 percent reduced risk of dying in that period, according to the study.

a) Starting just from here--what are the two variables they've mentioned so far? Were they manipulated or measured? 

b) Based on what you've read so far, is this study correlational or experimental? 

c) Without reading further, think of a third variable (a "C") that is likely to be correlated with both A (going to museums or concerts) and B (being less likely to die). Any good ideas? 

OK! Time for more information about the study from the journalist:

The study controlled for socioeconomic factors like a participant’s income, education level and mobility, said Andrew Steptoe, a co-author of the study and the head of University College London’s research department of behavioral science and health.

“Even if you take those things into account, we still found that there is a difference in the survival of people who are involved in the arts,” Professor Steptoe said in an interview on Friday.

 

Sounds like multiple regression was used here.

d) What was the criterion (dependent) variable in this study now?

e) What were the predictor ("independent") variables? 

f) Sketch a little regression table similar to those in Chapter 9--put the criterion variable at the top and the predictors at the bottom. Which betas do you think were significant?

g) More importantly, was the "C" variable you thought about in c), above, controlled in their regression?

h) Even though the authors controlled for income, education, and mobility, there are likely to be additional third variables that were NOT controlled for. Think of at least one more alternative explanation. 

 

As you learn in Chapter 9, regression does not support causation because even though it can control for alternative explanations you can think of (such as income, education, or mobility), it doesn't control for everything. If you read the Time's summary, you'll see even more variables they seem to have controlled for , such as marital status and profession. But there could still be unmeasured variables that are responsible for the relationship. 

 

But that doesn't stop the authors from making causal speculations. In fact, here's a mediator hypothesis. i) Sketch this mediator following the diagrams at the end of Chapter 9:

 

The study also noted that engaging in the arts can reduce loneliness, promote empathy and emotional intelligence, and keep people from becoming sedentary — all factors that contribute to a longer life.

 

What kind of study would you need to conduct to test this proposed mediator?

 

 

 

 

 

 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers. 

 

Dog people live longer

The study found an association between dog ownership and longevity. Which third variables were controlled for? Which third variables might still be responsible for the relationship? Photo: Shutterstock

I can't resist blogging about research on dogs. Here's a story on NPR that summarized several studies that have tested the correlation between longevity and dog ownership. The journalist introduces the topic this way: 

Dogs and humans that interact with one another get a jolt of oxytocin, the so-called "cuddle hormone." And, if you get to look at dogs and hug them every day, you just might live longer than people who don't have to clean animal hair off their clothes, according to a pair of studies out this month.

Here is a summary of one of the studies:

[researchers] looked over a 12-year period at dog owners in Sweden who have cardiovascular disease. They adjusted for the kinds of things we know affect cardiovascular health — age, demographics, socioeconomic status, marital status, number of children at home — and even after adjusting for all of that, they found a benefit of dog ownership [on longevity].

....they followed over 100,000 people and found a 21% reduction in deaths from any cause.

The study above used multiple regression. 

a) What was the dependent (criterion) variable in this study?  What were the predictor variables? 

b) Sketch a little regression table with results that would be consistent with what is mentioned above. Put the criterion variable at the top, and list the predictor variables. Which betas would be significant here? 

c) The researchers controlled for age, demographics, SES, marital status, and children--but regression doesn't control for all third variables. What else could be associated with both dog ownership and longevity? 

d) They mention a "21% reduction in deaths from any cause."  Which of the four big validities is this concerning? 

One of the experts who was interviewed for this story said:

In particular, I found it very convincing and striking that the benefits seems to be larger among individuals who live alone compared to multiperson households. That suggests the companionship of a dog is possibly very important to their heart health.

e) This is a moderator! Identify the variables in this relationship and complete this sentence: ____ moderates the relationship between ____ and _____.

 

In part of the article, the journalist presents quotes from an expert, who seems to equivocate on the causality issue:

"When you look at the big picture and look at all the evidence around dog ownership and cardiovascular health, it's pretty clear the signal is real and likely causal," he says.

In other words, he's convinced getting a dog improves your health — especially for those with heart disease.

The research is not definitive, though, Kazi adds. "They're not randomized trials, the gold standard for what we would do to evaluate a new drug," he says.

f) It is frustrating to read that the expert believes the "signal is real and likely causal" and then admits that "it's not definitive." You can set this expert straight. Why can't this correlational study support causation?  Apply covariance, temporal precedence, and internal validity. When you address internal validity, check out answer c) above!

 

 

Suggested answers

a) The criterion variable is longevity (being alive at the end of the study). The predictor variables are dog ownership, age, demographics, SES, marital status, and children living at home. 

b) The table might look like this:

DV: Longevity (or being alive after 12 years)

IV                            Beta (sig)
Dog ownership         .12*
Age                            -.11
SES                              .13*
Married                        .08*
Children at home        .07*

d) Current health and mobility might be associated with both dog ownership and longevity. After all, you  may need to have a certain level of health and mobility to own a dog, and healthier folks are more likely to be alive at the end of the study. 

e)  Living status (being alone or married) moderates the relationship between dog ownership and longevity.  

f) There is covariance--there is a link between owning a dog and living longer. 

There is temporal precedence-- the study took place over several years and dog ownership was measured first, and death rates later. 

There is not full internal validity. They controlled for age, demographics, SES, marital status, and children living at home, so they can rule those out. But they did not control for everything. One possibility is that current health and mobility might be associated with both--after all, you  may need to be at a certain level of health and mobility to own a dog, and those folks are more likely to be alive at the end of the study.