Everyday Research Methods

Anxiously waiting? Tetris might help

 

Which version of Tetris helped the most? Photo: incamerastock / Alamy Stock Photo

NPR's journalists covered a study that tested whether different forms of Tetris would help people feel less anxious.  

The journalist writes that

...research suggests that Tetris can ease us through periods of anxiety by getting us to a blissfully engrossed mental state that psychologists call "flow."

"The state of flow is one where you're completely absorbed or engaged in some kind of activity," Sweeny explains. "You lose your self-awareness, and time is just flying by."

Here's more on the detail: 

Sweeny and her collaborators gathered a group of more than 300 college students and told them their peers would be evaluating how attractive they were. "I know, it's kind of cruel, but we found it's a really effective way to get people stressed out," Sweeny says. While the participants awaited their attractiveness scores, the researchers had them play Tetris.

Some played a painfully slow, easy version of the game — which bored them. Some played an extremely challenging, fast version — which frustrated them. And everyone else played the classic version, which adapts to each player's individual skill level and gets them into that state of flow.[People were randomly assigned to the three groups.]

In the end, everyone experienced a degree of worry. But the third group reported slightly higher levels of positive emotions (on average, about a quarter of a point higher on a five-point scale) and slightly lower levels of negative emotions (half a point lower on a five-point scale).

"It wasn't a huge difference, but we think it's noticeable," Sweeny says. "And over time, it can add up."

Questions:

a) In this study, they decided to manipulate the conceptual variable, "degree of flow." How did they operationalize this variable? 

b) What were the dependent variables in this study? (there seem to be two DVs here)

c) What was the independent variable? What were its levels? 

d) Does this seem to be an experiment or a correlational study? How do you know? 

e) Sketch a graph of the results.

f) The journalist mentions details about the results (e.g., "about a quarter of a point higher on a five-point scale" and "half a point lower on a five-point scale"). Which aspect of statistical validity is being discussed here?

g) What questions would you ask to decide if this study was internally valid? Which of the internal validity threats in Table 11.1 could you rule out? Which could you ask about? 

h) What about the external validity of this study? How might you see if this effect might generalize to other flow-related activities (other than Tetris)? 

 

Corrections on effect size content

Here's an update to some effect size content in the textbook.

Figure S1.12

First, in Supplemental Chapter S1, there's a comparison of psychological to medical effect sizes (page 477 of the 3rd Edition). Figure S1.12 reproduces a chart by Bushman and Anderson (2001) comparing the correlation between violent media and aggression to effect sizes from various medical studies, such as the association between smoking and lung cancer. The chart makes psychology look good--our effect sizes appear larger than common medical wisdom! 

I've learned that this figure is problematic.  One issue with the figure concerns its medical study comparators. The examples suggest that effect sizes for medical treatments and risks are rather small (such as the link between asbestos and cancer or lead exposure and IQ). But over the years, critics have argued that the examples in Bushman and Anderson's figure are artificially low or inaccurately computed. (Notably, Bushman and Anderson have defended their figure, more than once.)

Another issue is that the effect sizes for violent media and aggression are based on meta-analytic estimates  that vary depending on the techniques used. Certain researchers report medium sized effects, but others argue that media effects on violence are negligible. (This debate led me to remove a media violence meta-analysis from Chapter 14 when I prepared the 3e.)

Ultimately, I am persuaded by Ferguson's (2009) explanation of how effect sizes of medical treatment studies vary dramatically, depending on whether they include "hypothesis irrelevant" cases. Especially when effect sizes are based on population-based epidemiological frequency counts, they can drastically underestimate the effect size of a vaccine, drug, or risk factor. 

Chapter 8: Aspirin and heart attack

Chapter 8 uses the classic example of "aspirin prevents heart attack" as an illustration of how a very small effect size (r = .03) can, in fact, have large life-or-death impact. But if the aspirin-heart attack effect size came from a study that included hypothesis-irrelevant cases, it probably underestimated the true effect size. Indeed, the Physician's Health Study estimated that aspirin reduces the risk of a first heart attack by 44%--a much larger effect. 

As always, I am grateful to colleagues who read my text carefully and alert me to problematic examples.  Thanks!

Friends and brain health

What do you think about the external validity of the study? Photo credit: Rawpixel.com/Shutterstock

This story's headline reads, "social ties could preserve memory, slow brain aging" and it's illustrated with an image similar to the one shown here. The press release sets out the goal of the study:

“We know that in humans there’s a strong correlation between cognitive health and social connections, but we don’t know if it’s having a group of friends that’s protecting people or if it’s that people with declining brain health withdraw from their human connections,” [Study researcher] Kirby said.

[The n]ew research ...found that mice housed in groups had better memories and healthier brains than animals that lived in pairs.

 a) Before reading on, reflect: Why would a researcher probably need an animal model to test this question experimentally? 

 

Here's some more detail about the experiment:

Some mice lived in pairs, which Kirby refers to as the “old-couple model.” Others were housed for three months with six other roommates, a scenario that allows for “pretty complex interactions.”

The mice were 15 months to 18 months old during the experiment – a time of significant natural memory decline in the rodent lifespan.

In tests of memory, the group-housed mice fared better. 

One test challenged the mice to recognize that a toy, such as a plastic car, had moved to a new location. ...“With the pair-housed mice, they had no idea that the object had moved. The group-housed mice were much better at remembering what they’d seen before and went to the toy in a new location, ignoring another toy that had not moved,” Kirby said.

In another common maze-based memory test, mice are placed on a well-lit round table with holes, some of which lead to escape hatches. Their natural tendency is to look for the dark, unexposed and “safe” escape routes.

The “couples” mice didn’t get faster at the test when it was repeated over the course of a day.“But over the course of many days, they developed a serial-searching strategy where they checked every hole as quickly as possible. It’d be like walking as quickly as possible through each row of a parking lot to look for your car rather than trying to remember where your car actually is and walk to that spot,” Kirby said.

The group-housed mice improved with each trial, though. “They seemed to try to memorize where the escape hatches are and walk to them directly, which is the behavior we see in healthy young mice,” Kirby said. “And that tells us that they’re using the hippocampus, an area of the brain that is really important for good memory function.”

b) What was the independent variable in this study? How was it operationalized?

c) What was the dependent variable? What were the two ways it was operationalized?

d) How does this experiment help us decide which comes first--social life or better memory? (note: This is temporal precedence!)

e) Do you think the journalist is justified in generalizing this study's results from mice to older adult humans? Why or why not? 

f) Chapter 3 explains how internal validity and external validity are often in a trade-off. Describe how this study with mice illustrates that trade-off. 

 

A test of dog empathy

Which of the two dependent variables showed that dogs were affected by their owners' sadness? Photo: adogslifephoto/istock/Getty Images Plus

Here's a short video with footage from an experiment that tested dogs' empathy. Watch the video with these questions in mind.

a) The study was an experiment. What was the independent variable (IV)? 

b) Was the IV manipulated as between-groups (independent groups) or within-groups? What keywords in the video description helped you answer this question? 

c) The dependent variable (DV) was operationalized in two ways. What were they? 

d) One of the DVs did not support the hypothesis, but the other DV did. Explain the results that they found. (You can sketch two little bar graphs, too.)

e) What do you think--does "opening a door to release a crying owner" indicate "empathy?" (P.S., That's a construct validity question.)

f) Does the study support the claim that "hearing their owners ask for help while crying causes dogs to help their owners faster"?  Apply the three causal criteria to support your answer. 

 

Suggested answers:

a) The IV was whether the owners were saying "help" while pretending to cry, or saying "help" in a neutral tone, while humming Twinkle Twinkle Little Star.

b) This was independent-groups--the reporter used keywords such as "some owners" and "other owners". This was a posttest-only design. 

c) One operationalization of the DV was whether the dogs opened the door, or not. The other operationalization of the DV was how long each dog took to open the door.

d) Only the "time taken" DV showed the predicted effect.

e) Answers will vary

f)  The results show covariance because dogs whose owners were crying opened the door three times as fast as dogs whose owners were calm. Temporal precedence is ensured by the methodology--by randomly assigning owners to cry (vs. sing), they ensured that this condition came before opening the door. The study would have good internal validity if they randomly assigned owner/dog pairs to the two conditions--this would take care of selection threats such as having more "already helpful" dogs in the crying condition, or having owners who are better at acting in one condition or the other. As far as design confounds, we might ask about whether the owners in the two conditions acted exactly the same in all ways except their assigned conditions. 

Concentration blunts your sense of smell

He is concentrating so much that he might not notice the coffee smell.... Photo: baranq/Shutterstock

At times, we've all been so engrossed in a task that we've lost awareness of our surroundings. Maybe you didn't hear someone calling your name when you were finishing your paper, or maybe you missed the oven timer when you were reading that mystery book.  Now researchers Sophie Forester and Charles Spence have reported that concentration impacts our sense of smell. Here's how the research was described on the APS website:

They set up a room to be distinctively aromatic, hiding three small containers of coffee beans around the room overnight. Over the course of two experiments, they led 40 college students into the room one at a time to perform a tough visual-search task on a computer, finding the letter “X” or “N” in a circle of similar-looking letters (“W,” “M,” “K,” “H,” “Z,” and “V”). 40 other students completed an easier version of the same task; searching for the letter “X” or “N” among a circle of lowercase “o”s. [Students had been randomly assigned to either the difficult or easy task.]

The experimenters then took the students into another room and asked them some follow up questions that grew increasingly leading :

1. “Describe the room you just completed the task in. Try to describe it using all of your senses.”
2. “Did you notice any odors in the room, if so what?”
3. “Could you smell coffee in the room?”

Students assigned to the difficult search task were far less likely to report having picked up the aroma (25% of participants said they noticed a coffee smell) compared to the participants assigned to the easy task (60%-70% percent of participants). When the experimenters led the students back into the test room, all of them said they could smell it. Some of them even commented that the room smelled like a cafe.

Questions

1. What kind of study was this--experimental or correlational? How do you know? 
2. What was the independent variable? What was the dependent variable? 
3. Think about construct validity: What do you think of the way they measured their dependent variable? Is this a good measure?
4. Now think about statistical validity: How large does this effect seem to be? Take another look at the results and make a comment on the practical effect size.
5. What about external validity? To whom might these results generalize? Do you think the pattern for coffee and letter detection might generalize to other smells? To other tasks?
6. Now consider internal validity. The authors claim that it was concentration that caused people to not notice the smell. Can you think of any confounds in this design? 

 

You're not as smart as you think

College graduates were more likely than those who'd not been to college to report  they are "smarter than average." Is their perception overconfident, or not? Photo: PeopleImages/Getty Images

It seems to be conventional wisdom that people are overconfident in their own abilities. People tend to think they are nicer, smarter, and better looking than most other people. But what's the evidence? The scientist-authors of this Wall Street Journal summary  explain, 

The claim that "most people think they are smarter than average" is a cliche of popular psychology, but the scientific evidence for it is surprisingly thin. Most research in this area has been conducted using small samples of individuals or only with high school or college students. The most recent study that polled a representative sample of American adults on the topic was published way back in 1965.

The authors, Patrick Heck and Christopher Chabris, worked with a third colleague. 

..[W]e conducted two surveys: one using traditional telephone-polling methods, the other using internet research volunteers. Altogether we asked a combined representative sample of 2,821 Americans whether they agreed or disagreed with the simple statement "I am more intelligent than the average person." 

Here are some of the results:

We found that more than 50% of every subgroup of people -- young and old, white and nonwhite, male and female -- agreed that they are smarter than average. Perhaps unsurprisingly, more men exhibited overconfidence (71% said they were smarter than average) than women (only 59% agreed).

Perhaps "overconfidence" is really accuracy? Consider this pattern of results: 

In our study, confidence increased with education: 73% of people with a graduate degree agreed that they are smarter than average, compared with 71% of college graduates, 62% of people with "some college" experience and just 52% of people who never attended college.

The accessible Wall Street Journal summary is paywalled, but the original empirical publication is open-access in PLOS One.

Questions

a) What kind of study was this? Survey/poll? Correlational? Experimental? What are its key variables? 

b) The authors found that more than 50% of every subgroup of people considered themselves smarter than average. Why is this result a sign of overconfidence? 

c) The authors of this piece state that their combined sample was "representative". Re-read the section on how they got their sample and then make your own assessment--is the sample representative? (i.e., how is its external validity?). What population of interest do they intend to represent? 

d) Sketch a graph of this result: 

73% of people with a graduate degree agreed that they are smarter than average, compared with 71% of college graduates, 62% of people with "some college" experience and just 52% of people who never attended college.

e) In concluding their article, the authors wrote, "Our study shows that many people think they are smarter than they really are, but they may not be stupid to think so." What do you think? To what extent does this study's results support this conclusion? 

e) Ask a question about this study's construct, internal, external, and statistical validity. 

Replication Update: When do people cheat?

A blog post in this space (2012) once included a summary of a set of studies on people's cheating behavior. One of those studies found that recalling the Ten Commandments led people to do less cheating. That study has recently been revisited as a registered replication report. The new researchers were not able to replicate the Ten Commandments finding.  Here's a description of the replication study.

 

Do kids do better in private schools?

What evidence would it take to convince us that it's the schools, not other factors, that are responsible for the outcomes of private school students? Photo: Image Source / Alamy Stock Photo

A large study has compared the outcomes of children who've attended private schools to those who've attended public schools.  A journalist summarized the report in the Washington Post. The study provides a nice example of how multivariate regression can be used to test third variable hypotheses. 

When we look simply at the educational acheivement of students in private schools vs. public schools, private school students have higher achievement scores. However, all such studies are correlational, because the two variables--Type of School and Level of Achievement--are measured. 

Therefore, such studies how covariance, because the results depict a relationship. The study may even show temporal precedence, because attending school presumably precedes the measure of achievement. However, such studies are weak on internal validity. We can think of several alternative explanations for why children in private schools are scoring higher. 

One major alternative explanation is socioeconomic status. Children from wealthier families are more likely to afford private schools. And in general, children from wealthier families tend to score higher on achievement tests. 

The Washington Post journalist quoted one of study's authors, Robert Pianta, who summed up the study's results this way:

“You only need to control for family income and there’s no advantage,” Pianta said in an interview. “So when you first look, without controlling for anything, the kids who go to private schools are far and away outperforming the public school kids. And as soon as you control for family income and parents’ education level, that difference is eliminated completely.”

Questions

a) Draw little diagrams similar to those in Figure 8.15 (in the 3rd ed.) to depict the arguments being made in this study. What would A be? What about B? In the quote from Pianta, above, what would the C variable(s) be? 

b) The researchers used type of school (private vs. public), which is a categorical variable. But in some analyses, the researchers also used "number of years in private school" as an alternative version of this variable. Is "number of years in private school" categorical, ordinal, interval, or ratio data? 

c) Sketch a mock-up regression table with the criterion variable  at the top and predictors below (Use Table 9.1 as a model). Which variable do you think the researchers selected as the criterion (dependent) variable in their analyses? Which variable(s) would have been the predictors? 

d) Now that you know what the results were, think about how the beta associated with "number of years in private school"  would change when parental SES is added and removed from the regression analyses. 

Original journal article seems to be open-access. Give it a try!

Suggested answers

a) A and B would be Type of School and Level of Acheivement. It doesn't really  matter which one is A and which one is called B. C would be Family Income and/or Parental Education.

b) Ratio data (zero is meaningful in this scale because you could attend zero years of private school)

c) The criterion variable would be Achievement, and the predictors would be Number of Years of Private School, Family Income, and Parental Education.

d)When the Number of Years of Private School is on the table (in the analysis) by itself, its beta is likely to be positve and significant (more years of private school goes with higher achievement). When Family Income and Parental Education are added to the table, the beta for Number of Years of Private School should drop to zero.  This pattern of results is consistent with the argument that Family Income and Parental Education are the alternative explanation for the original relationship .

Can greening up a vacant lot ease depression in a community?

The study randomly assigned lots like this one to be cleaned up, turned into green space, or left alone. Photo: 1000 Words/Shutterstock

When I first saw the headline, "Replacing Vacant Lots With Green Spaces Can Ease Depression In Urban Communities" I thought it was just another journalist putting a causal claim on a correlational study. So I was surprised to read this statement about the design:

[Researcher Eugenia South] and her colleagues wanted to see if the simple task of cleaning and greening these empty lots could have an impact on residents' mental health and well-being. So, they randomly selected 541 vacant lots [in the city of Philadelphia] and divided them into three groups.

They collaborated with the Pennsylvania Horticultural Society for the cleanup work.

The lots in one group were left untouched — this was the control group. The Pennsylvania Horticultural Society cleaned up the lots in a second group, removing the trash. And for a third group, they cleaned up the trash and existing vegetation, and planted new grass and trees. The researchers called this third set the "vacant lot greening" intervention.

Here's more:

The team surveyed residents living near the lots before and after their trial to assess their mental health and wellbeing. "We used a psychological distress scale that asked people how often they felt nervous, hopeless, depressed, restless, worthless and that everything was an effort," explains South.

The scale alone doesn't diagnose people with mental illness, but a score of 13 or higher suggests a higher prevalence of mental illness in the community, she says.

People living near the newly greened lots felt better. "We found a significant reduction in the amount of people who were feeling depressed," says South.

As one commentator noted:

Previous research has shown that green spaces are associated with better mental health, but this study is "innovative," says Rachel Morello-Frosch, a professor at the department of environmental science, policy and management at the University of California, Berkeley, who wasn't involved in the research.

"To my knowledge, this is the first intervention to test — like you would in a drug trial — by randomly allocating a treatment to see what you see," adds Morello-Frosch. 

Questions

a) How do we know that this is an experiment and not a correlational study?

b) What were the Independent and Dependent variables? 

c) What was the design: Posttest only? Pretest-posttest? Repeated measures? Or concurrent measures?

d) Sketch a graph of the results described above. 

e) Ask at least one question each about this study's construct, internal, external, and statistical validities.

f) Because this article was published in the open-access journal JAMA Network Open anyone can read the paper. Take a look carefully at the tables in the paper (especially Table 2). How strong do the results seem to you? Are the differences between the conditions large? Do you see improvements on all the measured variables, or just on a few of them? 

g) Why does the design of this study help support the causal claim that "Replacing vacant lots with green spaces can ease depression...."? (Apply the three causal criteria.) 

 

Social media effect sizes

Is social media use responsible for depressed mood? Photo: Ian Allenden/Alamy stock

Do smartphones harm teenagers? If so, how much? In this blog, I've written before about the quasi-experimental and correlational designs used in research on screen time and well-being in teenagers. In that post you can practice identifying the different designs we can use to study this question. 

Today's topic is more about the size of the effect in studies that have been published. A recent Wired story tried to put the effect size in perspective. 

One side of the argument, as presented by Robbie Gonzalez in Wired, scares us into seeing social media as dangerous.

For example, first

...there were the books. Well-publicized. Scary-sounding. Several, really, but two in particular. The first, Irresistible: The Rise of Addictive Technology and the Business of Keeping Us Hooked, by NYU psychologist Adam Alter, was released March 2, 2017. The second, iGen: Why Today's Super-Connected Kids are Growing Up Less Rebellious, More Tolerant, Less Happy – and Completely Unprepared for Adulthood – and What That Means for the Rest of Us, by San Diego State University psychologist Jean Twenge, hit stores five months later.

In addition,

...Former employees and executives from companies like Facebook worried openly to the media about the monsters they helped create. 

But is worry over phone use warranted? Here's what Gonzalez wrote after talking to more researchers:

When Twenge and her colleagues analyzed data from two nationally representative surveys of hundreds of thousands of kids, they calculated that social media exposure could explain 0.36 percent of the covariance for depressive symptoms in girls.

But those results didn’t hold for the boys in the dataset. What's more, that 0.36 percent means that 99.64 percent of the group’s depressive symptoms had nothing to do with social media use. Przybylski puts it another way: "I have the data set they used open in front of me, and I submit to you that, based on that same data set, eating potatoes has the exact same negative effect on depression. That the negative impact of listening to music is 13 times larger than the effect of social media."

In datasets as large as these, it's easy for weak correlational signals to emerge from the noise. And a correlation tells us nothing about whether new-media screen time actually causes sadness or depression. 

 There are several things to notice in the extended quote above. First let's unpack what it means to, "explain 0.36% of the covariance". Sometimes researchers will square the correlation coefficient r  to create the value R². The R²  tells you the percentage of variance explained in one variable by the other (incidentally, they usually say "percent of the variance" instead of "percent of covariance."). In this case, it tells you how much of the variance in depressive symptoms is explained by social media time (and by elimination, it tells you what percentage is attributable to something else). We can take the square root of 0.0036 (that's the percentage version of 0.36%) to get the original r between depressive symptoms and social media use. It's r = .06. 

Questions

a) Based on the guidelines you learned in Chapter 8, is an r of .06 small, medium, or large? 

b) Przybylski claims that the effect of social media use on depression is the same size as eating potatoes. On what data might he be basing this claim? Illustrate your answer with two well-labelled scatterplots, one for social media and the other for potatoes. Now add a third scatterplot, showing listening to music. 

c) When Przybylski states that the correlation held for the girls, but not the boys, what kind of model is that? (Here are your choices: moderation, mediation, or a third variable problem?)

d) Finally, Przybylski notes that in large data sets, it's easy for weak correlation signals to appear from the noise. What statistical concepts are being applied here? 

e) Chapter 8 presents another example of a large data set that found a weak (but statistically significant) correlation. What is it? 

f) The discussion above between Gonzalez and Przybylski concerns which of the four big validities? 

g) Finally, Przybylski mentions that "a correlation tells us nothing about whether new-media screen time actually causes sadness or depression".  Why not? 

 

Suggested answers:

a) An r of .06 is probably going to be characterized as "small" or "very small" or even "trivial."  That's what the "potatoes" point is trying to illustrate, in a more concrete way. 

b) One scatterplot should be labeled with "potato eating" on the x axis and "depression symptoms" on the y axis. The second scatterplot should be labeled with "social media use" on the x axis and "depression symptoms" on the y axis. These first two plots should show a positive slope of points with the points very spread out--to indicate the weakness of the association. The spread of the first two scatterplots should be almost the same, to represent the claim the two relationships are equal in magnitude. The third scatterplot should be labeled with "listening to music" on the x axis and "depression symptoms" on the y axis, and this plot should show a much stronger, positive correlation (a tighter cloud of points).

c) It is a moderator. Gender moderates (changes) the relationship between screen use and depression. 

d) Very large data sets have a lot of statistical power. Therefore, large data sets can show statistical significance for even very, very, small correlations--even correlations that are not of much practical interest. A researcher might report a "statistically significant' correlation, but it's essential to also ask about the effect size and its practical value (the potatoes argument).  Note: you can see the r = .06 value in the original empirical article here, on p. 9. 

e) The example in Chapter 8 is the one about meeting one's spouse online and having a happier marriage--that was a statistically significant relationship, but r was only .03. That didn't stop the media from hyping it up, however. 

f) Statistical validity

g) The research on smartphones and depressive symptoms is correlational, making causal claims (and causal language) inappropriate. That means that we can't be sure if social media is leading to the (slight) increase in depressive symptoms, or if people who have more depressive symptoms end up using more social media, or if there's some third variable responsible for both social media use and depressive symptoms. As the Wired article states, 

...research on the link between technology and wellbeing, attention, and addiction finds itself in need of similar initiatives. They need randomized controlled trials, to establish stronger correlations between the architecture of our interfaces and their impacts; and funding for long-term, rigorously performed research.

Finally, the Wired article quotes (the seemingly skeptical) Pyrbylski as saying, 

"Don't get me wrong, I'm concerned about the effects of technology. That's why I spend so much of my time trying to do the science well," Przybylski says.

Good science is the best way to answer our questions.