Everyday Research Methods

How well do fire alarms work for kids?

 Which fire alarm worked best to wake young children up? Photo: YMHhappyfamily/Shutterstock

An important applied psychology question concerns smoke alarms. Does the loud, screeching sound of a traditional smoke alarm work on kids? If not, what would work better? There's a Reuters press release about the findings of the study. 

Sleeping children, as it turns out, are fairly impervious to the screeching of a smoke alarm. Researchers found that most children ages 5 through 8 took more than five minutes to wake up with a standard alarm, as compared with around four seconds when they heard the sound of their mother’s voice, according to the results published in The Journal of Pediatrics.

“The thing that was most remarkable to us was to see a child sleep five minutes through a very loud high-pitched tone, but then sit bolt upright in bed when their mothers voice sounded through the alarm,” said the study’s lead author, Dr. Gary Smith.

In this research, experimenters invited families into a sleep laboratory on four different nights. Here's how a journalist summarized the study: 

Smith and his colleagues recruited 176 children between the ages of 5 and 12 and brought them into the sleep lab for testing. The lab had rooms that were set up to look like a typical bedroom, Smith said.

The children’s brain activity was monitored with electrodes on their scalp and face, and they were allowed to fall into the deepest stage of sleep. Then one of four types of simulated smoke alarm was sounded and researchers measured how long it took for a child to wake up and follow a previously rehearsed escape plan.

The experiment was run four times with each child, each about a week apart, so that all the children were exposed to all four types of alarm: standard high-pitched tone, the mother’s voice with instructions like “wake up” and “get out of bed,” the mother’s voice saying the child’s name, and the mother’s voice saying the child’s name and then giving instructions.

There was no significant difference in time to waking between the three alarms that used the mother’s voice. But there was a very large difference between all three of the maternal voice alarms and the standard alarm.

 

Questions: 

a) This is a simple experiment. What is the independent variable? What are its levels? Is it manipulated as independent groups or within groups? 

b) What is the dependent variable? 

c) Which of the four simple experiments is this: Posttest only? Prettest-posttest? Repeated measures? Concurrent measures? 

d) Sketch a graph of the results of this study.

e) What are the advantages to using a within-groups design in this study? 

f) Which internal validity threat do we most worry about in a design like this? How should the researchers control for that internal validity threat?

 

 

Suggested answers:

a) The independent variable is which smoke alarm they tried, and it has four levels: Traditional beep, Mom's voice saying the child's name; Mom's voice saying the name and giving instructions, and Mom's voice giving instructions. It was manipulated within groups. 

b) The DV is time to wake up (and whether they followed the right instructions)

c) This is a repeated measures design. 

d) Sketch should have "time to wake up" on the y axis and the four alarm conditions on the x-axis. 

e) The strongest advantage here is that each child can serve as their own control. Each child's normal level of "wakeability" is held constant across the four alarm conditions. 

f) We mainly worry about order effects in most repeated measures designs. The researchers should counterbalance the order of the four conditions to help control for this threat. To find out if they did that, refer to Method section of the empirical journal article, here. 

"Join a team and get sweaty" to help with depression

Photo credit: isitsharp/Getty Images

A health story's headline reads, "A New Prescription For Depression: Join A Team And Get Sweaty"

Psychologists have known for a while that depression levels are linked to exercise. In fact, some comprehensive treatments for depression include an exercise component in addition to psychotherapy or medication. 

This recent NPR story focused on which type of exercise that might work best: 

..a recent study in the journal Lancet Psychiatry found that popular team sports may have a slight edge over the other forms of physical activity.

The researchers analyzed Centers for Disease Control and Prevention survey data from 1.2 million adults and found — across age, gender, education status and income — people who exercised reported fewer days of bad mental health than those who didn't. And those who played team sports reported the fewest.

a. What are the two central variables in this research, based on what you've read so far?

b. Is this study correlational or experimental? 

c. Sketch a graph of the result. 

d) The journalist writes: 

Now, this study only shows an association between group exercise and improved mental health, and can't prove that the one causes the other. 

Why can't the study support that "one causes the other"?  Apply the three causal criteria (covariance, temporal precedence, and internal validity) to this study. 

 

Anxiously waiting? Tetris might help

 

Which version of Tetris helped the most? Photo: incamerastock / Alamy Stock Photo

NPR's journalists covered a study that tested whether different forms of Tetris would help people feel less anxious.  

The journalist writes that

...research suggests that Tetris can ease us through periods of anxiety by getting us to a blissfully engrossed mental state that psychologists call "flow."

"The state of flow is one where you're completely absorbed or engaged in some kind of activity," Sweeny explains. "You lose your self-awareness, and time is just flying by."

Here's more on the detail: 

Sweeny and her collaborators gathered a group of more than 300 college students and told them their peers would be evaluating how attractive they were. "I know, it's kind of cruel, but we found it's a really effective way to get people stressed out," Sweeny says. While the participants awaited their attractiveness scores, the researchers had them play Tetris.

Some played a painfully slow, easy version of the game — which bored them. Some played an extremely challenging, fast version — which frustrated them. And everyone else played the classic version, which adapts to each player's individual skill level and gets them into that state of flow.[People were randomly assigned to the three groups.]

In the end, everyone experienced a degree of worry. But the third group reported slightly higher levels of positive emotions (on average, about a quarter of a point higher on a five-point scale) and slightly lower levels of negative emotions (half a point lower on a five-point scale).

"It wasn't a huge difference, but we think it's noticeable," Sweeny says. "And over time, it can add up."

Questions:

a) In this study, they decided to manipulate the conceptual variable, "degree of flow." How did they operationalize this variable? 

b) What were the dependent variables in this study? (there seem to be two DVs here)

c) What was the independent variable? What were its levels? 

d) Does this seem to be an experiment or a correlational study? How do you know? 

e) Sketch a graph of the results.

f) The journalist mentions details about the results (e.g., "about a quarter of a point higher on a five-point scale" and "half a point lower on a five-point scale"). Which aspect of statistical validity is being discussed here?

g) What questions would you ask to decide if this study was internally valid? Which of the internal validity threats in Table 11.1 could you rule out? Which could you ask about? 

h) What about the external validity of this study? How might you see if this effect might generalize to other flow-related activities (other than Tetris)? 

 

Corrections on effect size content

Here's an update to some effect size content in the textbook.

Figure S1.12

First, in Supplemental Chapter S1, there's a comparison of psychological to medical effect sizes (page 477 of the 3rd Edition). Figure S1.12 reproduces a chart by Bushman and Anderson (2001) comparing the correlation between violent media and aggression to effect sizes from various medical studies, such as the association between smoking and lung cancer. The chart makes psychology look good--our effect sizes appear larger than common medical wisdom! 

I've learned that this figure is problematic.  One issue with the figure concerns its medical study comparators. The examples suggest that effect sizes for medical treatments and risks are rather small (such as the link between asbestos and cancer or lead exposure and IQ). But over the years, critics have argued that the examples in Bushman and Anderson's figure are artificially low or inaccurately computed. (Notably, Bushman and Anderson have defended their figure, more than once.)

Another issue is that the effect sizes for violent media and aggression are based on meta-analytic estimates  that vary depending on the techniques used. Certain researchers report medium sized effects, but others argue that media effects on violence are negligible. (This debate led me to remove a media violence meta-analysis from Chapter 14 when I prepared the 3e.)

Ultimately, I am persuaded by Ferguson's (2009) explanation of how effect sizes of medical treatment studies vary dramatically, depending on whether they include "hypothesis irrelevant" cases. Especially when effect sizes are based on population-based epidemiological frequency counts, they can drastically underestimate the effect size of a vaccine, drug, or risk factor. 

Chapter 8: Aspirin and heart attack

Chapter 8 uses the classic example of "aspirin prevents heart attack" as an illustration of how a very small effect size (r = .03) can, in fact, have large life-or-death impact. But if the aspirin-heart attack effect size came from a study that included hypothesis-irrelevant cases, it probably underestimated the true effect size. Indeed, the Physician's Health Study estimated that aspirin reduces the risk of a first heart attack by 44%--a much larger effect. 

As always, I am grateful to colleagues who read my text carefully and alert me to problematic examples.  Thanks!

Friends and brain health

What do you think about the external validity of the study? Photo credit: Rawpixel.com/Shutterstock

This story's headline reads, "social ties could preserve memory, slow brain aging" and it's illustrated with an image similar to the one shown here. The press release sets out the goal of the study:

“We know that in humans there’s a strong correlation between cognitive health and social connections, but we don’t know if it’s having a group of friends that’s protecting people or if it’s that people with declining brain health withdraw from their human connections,” [Study researcher] Kirby said.

[The n]ew research ...found that mice housed in groups had better memories and healthier brains than animals that lived in pairs.

 a) Before reading on, reflect: Why would a researcher probably need an animal model to test this question experimentally? 

 

Here's some more detail about the experiment:

Some mice lived in pairs, which Kirby refers to as the “old-couple model.” Others were housed for three months with six other roommates, a scenario that allows for “pretty complex interactions.”

The mice were 15 months to 18 months old during the experiment – a time of significant natural memory decline in the rodent lifespan.

In tests of memory, the group-housed mice fared better. 

One test challenged the mice to recognize that a toy, such as a plastic car, had moved to a new location. ...“With the pair-housed mice, they had no idea that the object had moved. The group-housed mice were much better at remembering what they’d seen before and went to the toy in a new location, ignoring another toy that had not moved,” Kirby said.

In another common maze-based memory test, mice are placed on a well-lit round table with holes, some of which lead to escape hatches. Their natural tendency is to look for the dark, unexposed and “safe” escape routes.

The “couples” mice didn’t get faster at the test when it was repeated over the course of a day.“But over the course of many days, they developed a serial-searching strategy where they checked every hole as quickly as possible. It’d be like walking as quickly as possible through each row of a parking lot to look for your car rather than trying to remember where your car actually is and walk to that spot,” Kirby said.

The group-housed mice improved with each trial, though. “They seemed to try to memorize where the escape hatches are and walk to them directly, which is the behavior we see in healthy young mice,” Kirby said. “And that tells us that they’re using the hippocampus, an area of the brain that is really important for good memory function.”

b) What was the independent variable in this study? How was it operationalized?

c) What was the dependent variable? What were the two ways it was operationalized?

d) How does this experiment help us decide which comes first--social life or better memory? (note: This is temporal precedence!)

e) Do you think the journalist is justified in generalizing this study's results from mice to older adult humans? Why or why not? 

f) Chapter 3 explains how internal validity and external validity are often in a trade-off. Describe how this study with mice illustrates that trade-off. 

 

A test of dog empathy

Which of the two dependent variables showed that dogs were affected by their owners' sadness? Photo: adogslifephoto/istock/Getty Images Plus

Here's a short video with footage from an experiment that tested dogs' empathy. Watch the video with these questions in mind.

a) The study was an experiment. What was the independent variable (IV)? 

b) Was the IV manipulated as between-groups (independent groups) or within-groups? What keywords in the video description helped you answer this question? 

c) The dependent variable (DV) was operationalized in two ways. What were they? 

d) One of the DVs did not support the hypothesis, but the other DV did. Explain the results that they found. (You can sketch two little bar graphs, too.)

e) What do you think--does "opening a door to release a crying owner" indicate "empathy?" (P.S., That's a construct validity question.)

f) Does the study support the claim that "hearing their owners ask for help while crying causes dogs to help their owners faster"?  Apply the three causal criteria to support your answer. 

 

Suggested answers:

a) The IV was whether the owners were saying "help" while pretending to cry, or saying "help" in a neutral tone, while humming Twinkle Twinkle Little Star.

b) This was independent-groups--the reporter used keywords such as "some owners" and "other owners". This was a posttest-only design. 

c) One operationalization of the DV was whether the dogs opened the door, or not. The other operationalization of the DV was how long each dog took to open the door.

d) Only the "time taken" DV showed the predicted effect.

e) Answers will vary

f)  The results show covariance because dogs whose owners were crying opened the door three times as fast as dogs whose owners were calm. Temporal precedence is ensured by the methodology--by randomly assigning owners to cry (vs. sing), they ensured that this condition came before opening the door. The study would have good internal validity if they randomly assigned owner/dog pairs to the two conditions--this would take care of selection threats such as having more "already helpful" dogs in the crying condition, or having owners who are better at acting in one condition or the other. As far as design confounds, we might ask about whether the owners in the two conditions acted exactly the same in all ways except their assigned conditions. 

Concentration blunts your sense of smell

He is concentrating so much that he might not notice the coffee smell.... Photo: baranq/Shutterstock

At times, we've all been so engrossed in a task that we've lost awareness of our surroundings. Maybe you didn't hear someone calling your name when you were finishing your paper, or maybe you missed the oven timer when you were reading that mystery book.  Now researchers Sophie Forester and Charles Spence have reported that concentration impacts our sense of smell. Here's how the research was described on the APS website:

They set up a room to be distinctively aromatic, hiding three small containers of coffee beans around the room overnight. Over the course of two experiments, they led 40 college students into the room one at a time to perform a tough visual-search task on a computer, finding the letter “X” or “N” in a circle of similar-looking letters (“W,” “M,” “K,” “H,” “Z,” and “V”). 40 other students completed an easier version of the same task; searching for the letter “X” or “N” among a circle of lowercase “o”s. [Students had been randomly assigned to either the difficult or easy task.]

The experimenters then took the students into another room and asked them some follow up questions that grew increasingly leading :

1. “Describe the room you just completed the task in. Try to describe it using all of your senses.”
2. “Did you notice any odors in the room, if so what?”
3. “Could you smell coffee in the room?”

Students assigned to the difficult search task were far less likely to report having picked up the aroma (25% of participants said they noticed a coffee smell) compared to the participants assigned to the easy task (60%-70% percent of participants). When the experimenters led the students back into the test room, all of them said they could smell it. Some of them even commented that the room smelled like a cafe.

Questions

1. What kind of study was this--experimental or correlational? How do you know? 
2. What was the independent variable? What was the dependent variable? 
3. Think about construct validity: What do you think of the way they measured their dependent variable? Is this a good measure?
4. Now think about statistical validity: How large does this effect seem to be? Take another look at the results and make a comment on the practical effect size.
5. What about external validity? To whom might these results generalize? Do you think the pattern for coffee and letter detection might generalize to other smells? To other tasks?
6. Now consider internal validity. The authors claim that it was concentration that caused people to not notice the smell. Can you think of any confounds in this design? 

 

You're not as smart as you think

College graduates were more likely than those who'd not been to college to report  they are "smarter than average." Is their perception overconfident, or not? Photo: PeopleImages/Getty Images

It seems to be conventional wisdom that people are overconfident in their own abilities. People tend to think they are nicer, smarter, and better looking than most other people. But what's the evidence? The scientist-authors of this Wall Street Journal summary  explain, 

The claim that "most people think they are smarter than average" is a cliche of popular psychology, but the scientific evidence for it is surprisingly thin. Most research in this area has been conducted using small samples of individuals or only with high school or college students. The most recent study that polled a representative sample of American adults on the topic was published way back in 1965.

The authors, Patrick Heck and Christopher Chabris, worked with a third colleague. 

..[W]e conducted two surveys: one using traditional telephone-polling methods, the other using internet research volunteers. Altogether we asked a combined representative sample of 2,821 Americans whether they agreed or disagreed with the simple statement "I am more intelligent than the average person." 

Here are some of the results:

We found that more than 50% of every subgroup of people -- young and old, white and nonwhite, male and female -- agreed that they are smarter than average. Perhaps unsurprisingly, more men exhibited overconfidence (71% said they were smarter than average) than women (only 59% agreed).

Perhaps "overconfidence" is really accuracy? Consider this pattern of results: 

In our study, confidence increased with education: 73% of people with a graduate degree agreed that they are smarter than average, compared with 71% of college graduates, 62% of people with "some college" experience and just 52% of people who never attended college.

The accessible Wall Street Journal summary is paywalled, but the original empirical publication is open-access in PLOS One.

Questions

a) What kind of study was this? Survey/poll? Correlational? Experimental? What are its key variables? 

b) The authors found that more than 50% of every subgroup of people considered themselves smarter than average. Why is this result a sign of overconfidence? 

c) The authors of this piece state that their combined sample was "representative". Re-read the section on how they got their sample and then make your own assessment--is the sample representative? (i.e., how is its external validity?). What population of interest do they intend to represent? 

d) Sketch a graph of this result: 

73% of people with a graduate degree agreed that they are smarter than average, compared with 71% of college graduates, 62% of people with "some college" experience and just 52% of people who never attended college.

e) In concluding their article, the authors wrote, "Our study shows that many people think they are smarter than they really are, but they may not be stupid to think so." What do you think? To what extent does this study's results support this conclusion? 

e) Ask a question about this study's construct, internal, external, and statistical validity. 

Replication Update: When do people cheat?

A blog post in this space (2012) once included a summary of a set of studies on people's cheating behavior. One of those studies found that recalling the Ten Commandments led people to do less cheating. That study has recently been revisited as a registered replication report. The new researchers were not able to replicate the Ten Commandments finding.  Here's a description of the replication study.

 

Do kids do better in private schools?

What evidence would it take to convince us that it's the schools, not other factors, that are responsible for the outcomes of private school students? Photo: Image Source / Alamy Stock Photo

A large study has compared the outcomes of children who've attended private schools to those who've attended public schools.  A journalist summarized the report in the Washington Post. The study provides a nice example of how multivariate regression can be used to test third variable hypotheses. 

When we look simply at the educational acheivement of students in private schools vs. public schools, private school students have higher achievement scores. However, all such studies are correlational, because the two variables--Type of School and Level of Achievement--are measured. 

Therefore, such studies how covariance, because the results depict a relationship. The study may even show temporal precedence, because attending school presumably precedes the measure of achievement. However, such studies are weak on internal validity. We can think of several alternative explanations for why children in private schools are scoring higher. 

One major alternative explanation is socioeconomic status. Children from wealthier families are more likely to afford private schools. And in general, children from wealthier families tend to score higher on achievement tests. 

The Washington Post journalist quoted one of study's authors, Robert Pianta, who summed up the study's results this way:

“You only need to control for family income and there’s no advantage,” Pianta said in an interview. “So when you first look, without controlling for anything, the kids who go to private schools are far and away outperforming the public school kids. And as soon as you control for family income and parents’ education level, that difference is eliminated completely.”

Questions

a) Draw little diagrams similar to those in Figure 8.15 (in the 3rd ed.) to depict the arguments being made in this study. What would A be? What about B? In the quote from Pianta, above, what would the C variable(s) be? 

b) The researchers used type of school (private vs. public), which is a categorical variable. But in some analyses, the researchers also used "number of years in private school" as an alternative version of this variable. Is "number of years in private school" categorical, ordinal, interval, or ratio data? 

c) Sketch a mock-up regression table with the criterion variable  at the top and predictors below (Use Table 9.1 as a model). Which variable do you think the researchers selected as the criterion (dependent) variable in their analyses? Which variable(s) would have been the predictors? 

d) Now that you know what the results were, think about how the beta associated with "number of years in private school"  would change when parental SES is added and removed from the regression analyses. 

Original journal article seems to be open-access. Give it a try!

Suggested answers

a) A and B would be Type of School and Level of Acheivement. It doesn't really  matter which one is A and which one is called B. C would be Family Income and/or Parental Education.

b) Ratio data (zero is meaningful in this scale because you could attend zero years of private school)

c) The criterion variable would be Achievement, and the predictors would be Number of Years of Private School, Family Income, and Parental Education.

d)When the Number of Years of Private School is on the table (in the analysis) by itself, its beta is likely to be positve and significant (more years of private school goes with higher achievement). When Family Income and Parental Education are added to the table, the beta for Number of Years of Private School should drop to zero.  This pattern of results is consistent with the argument that Family Income and Parental Education are the alternative explanation for the original relationship .