Everyday Research Methods

Time alone with your thoughts?

Credit: Ground Picture/Shutterstock

Imagine sitting alone with your thoughts for 20 minutes. You wouldn't be allowed to distract yourself with a book, phone, or even walking around. How much do you think you would you enjoy it?  

When this question was asked of participants in a series of studies, most reported that they'd not enjoy time alone very much.  But this study also tested how well their predictions matched their actual experience. The empirical research was published in the Journal of Experimental Psychology: General, and  summarized in a trade magazine for the American Psychological Association. According to the journalist, 

...study lead author Aya Hatano, PhD, of Kyoto University in Japan[, said] “Our research suggests that individuals have difficulty appreciating just how engaging thinking can be. That could explain why people prefer keeping themselves busy with devices and other distractions, rather than taking a moment for reflection and imagination in daily life.”

The research team conducted six studies. Here is an overview of some of them: 

In a series of six experiments with a total of 259 participants, the researchers compared people’s predictions of how much they would enjoy simply sitting and thinking with their actual experience of doing so. In the first experiment, they asked people to predict how much they would enjoy sitting alone with their thoughts for 20 minutes, without being allowed to do anything distracting such as reading, walking around or looking at a smartphone. Afterward, participants reported how much they had enjoyed it.

The researchers found that people enjoyed spending time with their thoughts significantly more than they had predicted. This held true across variations of the experiment in which participants sat in a bare conference room or in a small, dark tented area with no visual stimulation; variations in which the thinking period lasted for three minutes or for 20 minutes; and one variation in which the researchers asked people to report on their enjoyment midway through the task instead of after it was over. In every case, participants enjoyed thinking more than they had expected to.

a) The researchers conducted six different studies. Do these sound like direct replications? Replication-plus-extensions? or Conceptual replications? (you can skip this question if you haven't done Chapter 14 yet)

b) The journalist identifies these as "experiments," so let's assume that each study contains a manipulated variable. Take this study, as described by the journalist:

The researchers found that people enjoyed spending time with their thoughts significantly more than they had predicted [in a variation of] the experiment in which participants sat in a bare conference room or in a small, dark tented area with no visual stimulation. 

In this study, one of the independent variables was "predicted vs. actual".  What is the other main independent variable, or IV? What is the dependent variable, or DV? 

c) Do the same for this study:

variations in which the thinking period lasted for three minutes or for 20 minutes;

In this study, one of the independent variables was "predicted vs. actual". What is the other IV? What is the DV?

d) And for this one: 

In another experiment, the researchers compared one group of participants’ predictions of how much they would enjoy thinking with another group’s predictions of how much they would enjoy checking the news on the internet. Again, the researchers found that people underestimated their enjoyment of thinking. The thinking group expected to enjoy the task significantly less than the news-checking group, but afterward, the two groups reported similar enjoyment levels.

In this study, one of the independent variables was "predicted vs. actual". What is the other IV? What is the DV?

e) Challenge question: For the study above (comparing time alone to looking at news on the internet)  and graph the result. You can compare your graph to Figure 6 in the empirical article (available next to the APA journalist's story, here).

Here's a final caveat about the results. I appreciated this statement because it puts the statistical outcome in concrete and realistic terms: 

It is important to note that participants did not rate thinking as an extremely enjoyable task, but simply as more enjoyable than they thought it would be, according to Murayama. On average, participants’ enjoyment level was around 3 to 4 on a 7-point scale. Future research should delve into which types of thinking are most enjoyable and motivating, according to Murayama. “Not all thinking is intrinsically rewarding, and in fact some people are prone to vicious cycles of negative thinking,” he said.

 

 

What your taste in music says about your personality

According to the study, which personality trait is associated with upbeat, danceable music? Credit: Prostock-studio/Shutterstock

This splashy heading from Inc.com reads:

Here's what your taste in music says about your personality, according to a huge new study. whether you like Mozart or metal really does reveal something about your personality, new research finds

 

This was a correlational study, and provides a good chance to practice concepts from Chapter 3 and 8. Here is how the journalist summarized the methods and results:

...researchers from University of Cambridge and Israel's Bar-Ilan University used an online quiz to extract information on the musical preferences and personalities of more than 285,000 study subjects from 53 countries (you can try the quiz yourself here). They also set up a website where 71,000 visitors from 34 countries rated music clips and took a personality quiz. 

The team used well-known, scientifically validated frameworks for categorizing both music and personality. The aptly named "Music" framework categorizes music using five dimensions: mellow, unpretentious, sophisticated, intense, and contemporary. The familiar Big 5 personality framework categorizes people according to openness, conscientiousness, extroversion, neuroticism, and agreeableness. 

Did the research team find any correlations between the type of music someone enjoyed and their personality type? Yup, reported the researchers recently in the Journal of Personality and Social Psychology. 

a) You could think of this study as having two big variables or ten smaller variables. What would be the two "big" variables? What would be the ten smaller variables? 

b) What makes this a correlational study? 

c) Here is a description of one of their results:

Extroverts the world over tend to like upbeat, danceable contemporary music. 

Sketch a well-labelled scatterplot of this correlation. 

d) Here is another result: 

The study found fans of the Nirvana classic "Smells Like Teen Spirit" and other intense music are higher in neuroticism on average.

Sketch a well-labelled scatterplot of this correlation. 

e) Here is a third result:

Those high in conscientiousness tended to avoid more intense music...

Sketch a well-labelled scatterplot of this correlation. 

f) And here's one more:

Looking for an agreeable type? Then put on some Marvin Gaye and see how they react. The researchers found a strong correlation between liking mellow music and being high in agreeableness.

 

g) Now, here are two quotes from the journalist's article. For each quote decide which of the four big validities this point is addressing.

 

1. The team used well-known, scientifically validated frameworks for categorizing both music and personality. 

2. They ... set up a website where 71,000 visitors from 34 countries rated music clips and took a personality quiz. 

 

h) What is a question you could ask about this study's statistical validity?

 

i) Inc.com is a business magazine, so they framed the study in terms of job hiring:

 

Despite its sometimes dubious scientific basis, the personality testing industry rakes in around $2 billion annually telling companies their new hire is an INFJ or revealing her Hexaco type. [This] study suggests your business might be able to save some money and just ask candidates about their taste in music instead. 

What do you think? Does the research study support the idea that we could use music preferences in job interviews to select certain personality traits? Why or why not?  (to read the Inc.com take on this question, read the full article). 

 

 

Diet + Exercise: A factorial example

The study found that both exercise and diet are important. Is the effect additive or interactive? Photo credit:Dragana Gordic/Shutterstock

You might have heard someone claim that because they exercise regularly, they can eat whatever they want and still be healthy. When people say this, they seem to mean that either exercise OR diet can be sufficient to foster a healthy lifestyle. However, a recent study counters this "either/or" view. Like many other studies, this one found evidence that both diet and exercise contribute to health outcomes in adults.

In this blog post, we'll explore how the study can be viewed as a factorial design (Chapter 12), and we will discuss how its main finding can map on to different factorial outcomes. 

The empirical study appeared in the British Journal of Sports Medicine, which may be paywalled. I learned about the study from number of journalists who covered it, including this accessible summary in Healthline, which we'll use in this post.  

Here are the opening bullet points from the Healthline story: 

• A new study finds that in order to live longer, people need to eat healthily and exercise regularly.
• While a balanced diet and regular exercise independently have plenty of health benefits, the study notes the greatest reduction in mortality risk comes from combining diet and exercise together.

a) These bullet points mention the study's three main variables. One of the variables is mortality risk, or longevity (lifespan). What are the other two variables?

b) Chapter 12 concerns factorial designs, in which there are two (or more) independent variables (IVs) and one dependent variable (DV). Reflect back on the three variables you listed in a), above, and identify which are the IVs and which is the DV.  In addition, identify the levels of each IV. 

Here are some details about the study's method and results, as summarized by the journalist:

For the study, the researchers evaluated the health and exercise data of 346,627 individuals over the course of 11 years. They tracked how much exercise each participant completed in an average week along with how intense their physical activity was. The team also followed the participants’ eating habits. During the study window, 13,869 participants died — 2,650 from heart disease and 4,522 from adiposity-related cancers.

The research team found that any type of regular exercise was associated with a lower risk of mortality, and those who both ate well and exercised had the lowest risk of dying.

In addition, while exercise and diet are independently linked to a lower risk of mortality, high levels of exercise cannot entirely mitigate the harms of a poor diet.

I made up three patterns of data to illustrate possible outcomes of this study. You can see the graphs below, labelled Outcomes A, B, and C. Take a moment to orient yourself to the graphs, noticing how the DV is on the y-axis, one IV is on the x-axis, and the other IV is depicted using line color:

 

c) Which of these outcomes (A, B, or C, above) is consistent with the idea that EITHER exercise OR diet alone is sufficient to maintain a longer lifespan? Explain your reasoning.

d) The journalist wrote that "The research team found that any type of regular exercise was associated with a lower risk of mortality, and those who both ate well and exercised had the lowest risk of dying [i.e., the longest lifespan].”  Two of these outcomes (A, B, or C) are consistent with this statement. Which ones? 

e) In your own words, describe the difference between Outcome A and Outcome B.

f) Based on what the journalist wrote, can you tell which of these two (Outcome A or Outcome B) the study found? Specifically, which one could go with how the journalist described the study's outcome:

“In our study, those who ate a poor-quality diet and were active still had substantially reduced mortality risk than those who ate a poor-quality diet and were inactive,” Melody Ding, MPH, PhD, the lead author of the study and an associate professor of health and medicine in the Sydney School of Public Health at the University of Sydney, told Healthline. “It’s just that you had the maximum risk reduction when doing both things right.”

g) Just for more practice: Two of the outcomes above depict an interaction effect (an interaction between diet and exercise). Which 2 outcomes are they? Explain your reasoning--how do you know?

Suggested answers:

a) Besides longevity, a second variable is diet quality and a third variable is exercise habits.

b) Longevity is the DV. Diet quality is an IV (levels could be good vs. poor diet). Exercise habits are also an IV (levels good be regular exercise vs. no exercise)..

c) Outcome C. Outcome C shows that you have the same lifespan if you exercise or eat a good diet, and that doing both does not increase your lifespan any more than doing one or the other (all three conditions--exercise alone, diet alone, and both--show the same lifespan). Note that this is not actually the result that the study found.

d) Both Outcome A and Outcome B are consistent with the journalist's statement. 

e) The difference between Outcomes A and B can be summarized with the terms "additive" versus "interactive."
Outcome A shows an additive effect--A good diet adds (in this example) 3 years to your life; exercising adds 3 years, and if you do both, it adds 6 years. One is good, both is better.  
Outcome B shows an interactive, or multiplicative effect. It shows that if you do both diet AND exercise, you have an especially long lifespan. That is, in this example, a good diet adds 3 years to your life; exercising adds 3 years, and if you do both, it adds 11 years. One is good, both is amazing (kaboom!). In Chapter 12 you'll learn that this is an interaction--the combination of the two IVs results in something even bigger than simply adding the two together.

f) In my view, we can't tell which outcome the study found based on what the journalist wrote. However, since they used the terms "both" rather than "especially", it seems like it might be Outcome A. My reading of the empirical study backs this up--it seems to be an additive effect (not interactive). 

g) Outcome B and C both show interactions, but of different types. First, you can see the interactions in both because the lines are not parallel--there are differences of differences. That is, the effect of diet is different, depending on if we're talking about regular exercise or no exercise. 

In Outcome B, there is an interactive, or multiplicative effect--I sometimes call this an "especially" interaction. It shows that if you do both diet AND exercise, you have an especially long lifespan (compared to doing either one alone).

In Outcome C, there is a different kind of interaction--I might call it an "only if" or an "it depends" interaction. You could explain it by saying that lifespan is reduced only if you don't exercise AND don't eat well. If you exercise, or diet, or both, you'll live longer. 

Tuskegee Syphilis Study: An anniversary

Four months after Heller's story broke, the Tuskegee study was terminated with this letter.
Credit: U.S. National Archives

In this photo from the 1950s, a man participating in the study had his blood drawn.
Credit: U.S. National Archives

The year 2022 marks the 5oth anniversary of a major news event--journalist Jean Heller's story about the Tuskegee syphilis study. In Chapter 4 you can read about the study, in which poor Black men in Alabama were recruited and purposefully not treated for syphilis. The news story first appeared in the Associated Press (the AP). To honor its 50-year anniversary, the AP re-ran their original news report from July, 1972. 

Here is the lede (the opening passage) from Heller's original story:

For 40 years the U.S. Public Health Service has conducted a study in which human guinea pigs, denied proper medical treatment, have died of syphilis and its side effects.

The study was conducted to determine from autopsies what the disease does to the human body.

PHS officials responsible for initiating the experiment have long since retired. Current PHS officials, who say they have serious doubts about the morality of the study, also say that it is too late to treat syphilis in any of the study’s surviving participants.

In Chapter 4 (Ethics), you can read how the Tuskegee syphilis study led to the Belmont Report, which outlined major changes in how research is evaluated and conducted in the United States. The Tuskegee study violated three distinct ethical principles: Respect for Persons, Beneficence, and Justice. 

a) In your own words, define each of these principles. 

b) Read each of the quoted passages below, from Heller's original story. Decide what ethical problems each one seems to illustrate. Can you map each passage to one of the core ethical principles of the Belmont report (Respect for Persons, Beneficence, or Justice)?

b1) The experiment, called the Tuskegee Study began in 1932 with about 600 black men mostly poor and uneducated, from Tuskegee, Ala., an area that had the highest syphilis rate in the nation at the time.

b2) As incentives to enter the program, the men were promised free transportation to and from hospitals, free hot lunches, free medicine for any disease other than syphilis and free burial after autopsies were performed.

b3) The Tuskegee Study began 10 years before penicillin was discovered to be a cure for syphilis and 15 years before the drug became widely available. Yet, even after penicillin became common, and while its use probably could have helped or saved a number of the experiment subjects, the drug was denied them, Dr. J.D. Millar says.

Don Prince, another official in the venereal disease branch of CDC, said [...] “I don’t know why the decision was made in 1946 not to stop the program,” Prince said. “I was unpleasantly surprised when I first came here and found out about it. It really puzzles me.”

As part of the anniversary, the AP also ran a story-behind-the-story about how Heller researched and broke the story.  Here is Heller reflecting on how she felt as she was breaking the story: 

Normally, reporters celebrate these “Eureka” moments. But Heller felt no such elation.

“I knew that people had died, and I was about to tell the world who they were and what they had,” she says, her voice dropping. “And finding any joy in that ... would have been unseemly.”

Heller also said,

“As much injustice as there was for Black Americans back in 1932, when the study began, I could not BELIEVE that an agency of the federal government, as much of a mistake as it was initially, could let this continue for 40 years,” says Heller. “It just made me furious.”

What were some outcomes of Heller's reporting? 

Nearly four months after the story ran, the study was halted.

The government established the Tuskegee Health Benefit Program to begin treating the men, eventually expanding it to the participants’ wives, widows and children. A class-action lawsuit filed in 1973 resulted in a $10 million settlement.

c) As you reflect on this story, what are you most concerned about? What questions do you still have? 

 

Having more kids linked to lower cognitive function

The study found that parents with more than two children experienced greater cognitive decline as they aged, compared to parents with only two children--at least in Northern Europe.  Photo credit: fotoliza/Shutterstock

The website Study Finds produces  summaries of science research, publishing multiple summaries per day. This example concerns a study showing that how many kids people have had (in their younger years) predicted their later-life cognitive function. Somewhat confusingly, the print headline reads, "having more kids linked to weaker brain function," but the accompanying video segment includes the causal claim, "Kids can weaken your brain". 

Here's the opening segment from the journalist:

Everything in moderation — even kids? Researchers from Columbia University and Université Paris-Dauphine report having more than two kids may have a negative impact on late-life cognition. The study shows that older parents with just two children appeared sharper cognitively than those with three.

a) There are two main variables in this statement so far. What are they? Do you think these variables were manipulated or measured? 

b) Identify the verbs in the bolded statements above, and explain which ones are association claim verbs and which are causal claim verbs. 

c) On the face of it, the study found a simple correlation between having more than two kids (variable A) and cognitive function (variable B). You know that correlation is not causation, of course. What are some possibilities that explain this relationship? Consider temporal precedence first: Come up with a causal story in which A leads to B, as well as an alternative story in which B leads to A. Then come up with a C variable (a third variable, or internal validity problem) that might plausibly be related to both A and B. 

Now that you've assessed the basics of this bivariate correlational study,  here's more of what the journalist said about the study:

...study authors analyzed data provided by the Survey of Health, Aging and Retirement in Europe (SHARE). Featuring representative samples of the older populations across 20 European countries and Israel, SHARE was an ideal match for this project. Each individual was at least 65 years old and had at least two biological children.

Via a series of complex econometric methods, the research team was able to successfully disentangle various causal relationships from simple associations. In simpler terms, the analysis found that having three or more kids (as opposed to just two) is related to worse late-life cognition. This held up among both moms and dads.

[...]

“The negative effect of having three or more children on cognitive functioning is not negligible, it is equivalent to 6.2 years of aging,” Dr. Bonsang notes.

The journalist referred to "complex econometric methods" because the study itself used a creative strategy to try to test the relationship of having more than two kids on cognitive functioning, while ruling out potential alternative explanations. The researchers started with the assumption that parents who choose to have more than one kid often wish to have at least one boy and one girl. Indeed, in their large European data set, they found evidence that when people start out with two kids of the same sex (such as two boys or two girls), they are more likely to go for a third child, compared to people whose first two kids were a boy and a girl. Then, when the researchers statistical controlled for this variable (the sex composition of the first two kids), they observed a strong decline in cognitive function in later years. 

The upshot is that the researchers tried to take advantage of a natural experiment that goes something like this: When people's first two kids are the same sex, they're more likely to have a third child, which is likely to be followed by cognitive decline in later years. In contrast, when people's first two kids are different sexes, they're less likely to have a third child, which is followed by less cognitive decline. Because sex of one's first two children is more or less randomly assigned, it serves as an experimental variable. 

Importantly, they found the effect of children on cognitive decline only in the Northern European countries (not other regions such as Western, Eastern, or Southern). 

Here are some brief questions for you. 

Which of the four big validities (internal, external, construct, or statistical) is each of these statements addressing? Each quote comes from the journalist's coverage. 

d)  "...study authors analyzed data provided by the Survey of Health, Aging and Retirement in Europe (SHARE). Featuring representative samples of the older populations across 20 European countries and Israel" (Which validity is this?)

e) “'The negative effect of having three or more children on cognitive functioning is not negligible, it is equivalent to 6.2 years of aging,' Dr. Bonsang notes." (Which validity is this?)

f) Which two validities are left? Ask a question that would provide information about the remaining two validities.

g) Is it a problem that this effect was found only in the Northern European sample, and not the other three samples? Why or why not?

 

Life expectancy gap in different U.S. counties

An example of politically polarized health outcomes occurred during the pandemic.  Democratic-led areas were more likely to pass masking and lockdown mandates, and these areas later had lower COVID death rates compared to Republican-led areas.
Photo credit: MikeEdwards/Depositphotos

The American political system is polarized--increasingly so in the last few decades. Such polarization has gridlocked national politics in the past few years. In contrast, U.S. states and counties are passing legislation, and increasingly, this legislation is reflecting polarized local politics.

A new study decided to look at whether these local political differences were linked to actual health outcomes. The researchers showed that county by county, the average life expectancy in a particular U.S. county is associated with that county's presidential voting patterns. Specifically, the study found that 

how a state votes in presidential elections helps predict life expectancy for people living in that [county].

The study was summarized by journalists at NPR, and the story takes the form of a conversation between two journalists. It reported:

...political affiliation is one factor shaping the life span of people around the country according to this new study. Researchers went back about 20 years. They analyzed a CDC database that collects death data on, essentially, every individual in the U.S. They linked this data with federal election data, looking specifically at how counties voted in presidential elections and also in state governor's races.

a) So far, the journalist has mentioned two major variables in this study. What are they?

b) Were these two variables measured or manipulated? 

Now here is some more detail: 

Now, overall, mortality rates in the U.S. had been declining in the early 2000s. People were generally living longer. But the researchers found some of those gains have faded away. And when they looked at areas losing ground, they found Republican counties were losing more. Bottom line, people in Republican-leaning counties appear to be more likely to die prematurely. 

In many correlational studies in psychology, the person is the unit of analysis (the unit of focus). But in this study, the county is the unit of analysis. So for each county, the researchers recorded the death rates and the voting patterns. 

c) Take a moment to sketch a scatterplot of the relationship described above. Put "average age at death" on one axis and "percentage of votes for Republicans" on the other. Should your scatterplot show a positive or a negative relationship? 

d) What does one dot in your scatterplot represent? 

The researchers used regression to rule out several possible alternative explanations for this relationship, as you'll read about in this next passage:

Now, it's not something as simple as Republican counties being older. The researchers adjusted for age. Nor is it just an urban-rural divide in the country. They found the mortality gap held up in suburban and urban counties that vote Republican, too.

Note that you can see the results of these analyses in the empirical journal article. 

e) What were the variables they controlled for in their analysis?

The rest of the journalist's discussion is about mediators--they discuss WHY there might be a link between Republican voting and death rates. Here's one example: 

[researcher Dr. Steven Woolf argues that] if you look at policies such as the expansion of Medicaid, access to health care ... tobacco control, gun legislation, drug addiction - a whole range of policies have an impact on health and mortality rates. Democratic states have supported more of these. But Republican states have gone the other direction.

Researcher Woolf mentioned the pandemic, as well:

States with Republican leaders were pushing back on COVID-19 vaccination and enforcement of public health policies. And what we basically had was a controlled experiment where, you know, some states adopted this proactive approach, other states didn't. And we had an outcome that could be measured within weeks. And we saw massive differences in death rates.

f) After reading the passage above, you should be able to sketch the different mediators they are discussing, using the following template:

   Republican voting pattern. ---> __________ --> Higher death rate

g) What other potential mediators could you propose for this link?

h) Are there potential third variables that you think the researchers might wish to control for in their regression analysis? 

Amazon's Alexa and baby-naming trends

Amazon's choice to use the wake word "Alexa" seems to have led to a decrease in that name's popularity for babies. Photo: vectorstockcompany/Depositphotos

Here's a report from the Washington Post, entitled "Amazon, can we have our name back?". The journalist interviewed several women named Alexa, reporting about their experiences related to having the same name as the popular digital assistant. For example, 

Alexa Morales wore her name proudly. But after Amazon launched its voice service, also called Alexa, in November 2014, people began speaking to Morales differently. She said they made jokes about her name, giving her commands or asking her questions in a robotic tone. “When I hear my name now, it’s not good thoughts, it’s like, tensing,” said Morales, 28, a pharmacy technician and student in Bridgeport, Conn.

Much of the article is dedicated to the negative experiences (including harassment) that people with the name Alexa have had. Several Alexas have chosen to adopt nicknames. One psychologist was quoted as saying,

“They’re sort of growing up with the communication that their name is associated with a servant role,” Christopher Kearney, a professor of psychology at the University of Nevada at Las Vegas, said in an interview. “If a teacher is saying something derogatory, even if it’s just as a joke, or it’s to change their name, or … allows other kids to participate in that, it can create a lot of psychological damage.”

Aside from these detailed experiences, one part of this article can be used as an example of an interrupted time-series design. 

Nearly 130,000 people in the United States have the name Alexa. It gained popularity after singer Billy Joel and model Christie Brinkley named their daughter Alexa in 1985. In 2015, more than 6,000 baby girls in the United States were named Alexa, according to a Washington Post analysis of Social Security Administration data.

After Amazon chose Alexa as the wake word of its voice service, the name’s popularity plummeted. In 2020, only about 1,300 babies were given the name. 

It's much easier to think about this result if you see the graph of it, so I urge you to visit the Washington Post story and scroll down to the line graph before reading on. 

a) In this interrupted time series design, what is the quasi-independent variable? What is the dependent variable?

b) What do you think--to what extent can the baby name data in the article support the causal claim that "Amazon's Echo devices caused a decrease in people naming their babies Alexa"?

Specifically, the graph shows covariance and temporal precedence--Amazon's Alexa device is associated with a subsequent, dramatic drop in the number of babies named Alexa. But what about internal validity--are there alternative explanations we can consider for this drop? 

c) What are your thoughts after reading the article? Do you that Amazon should change the Echo's default "wake word?"

 

 

 

 

Music festivals "make people more generous, socially connected"

Credit: halfpoint/Deposit Photos

Summer is the time for music festivals like Burning Man, Burning Nest, Firefly, and Bonnaroo. If you are planning to attend one this year, this post might interest you.

The science journalism clearinghouse called Study Finds recently reported on a study conducted at Yale University. The journalists' report included the causal headline that Music festivals like Burning Man make people more generous, socially connected to others". 

a) What are the variables in this causal headline, and what makes it a causal claim?

b) In general, researchers need to conduct an experiment in order to support a causal claim. Do you see any barriers to conducting an experiment on the causal effects of music festivals?

It will be interesting to read about whether the researchers' study can feasibly support the causal claim that the journalist attached to it. Here's how Study Finds described the study's method: 

The Yale researchers studied the behavior and experience of over 1,200 people attending festivals such as Burning Man, Burning Nest, Lightning in a Bottle, Dirty Bird, and Latitude. All of the festivals took place in the United States or the United Kingdom and emphasized art, music, and self-expression. The team set up booths at all the events, which they called “Play Games for Science.”

People ... self-reported their experiences at the event and whether they were willing to share their resources with friends and strangers. For example, week-long festivals such as Burning Man do not sell food, and people who find themselves short must rely on the generosity of others.

[...] Overall, 63.2 percent of people said they had a transformative experience and felt themselves changing. ... One of the takeaways from the transformative experience was feeling more connected to other humans, which made them more likely to share food and resources with other attendees.

c) So far, do the method and results of this study (as reported by the journalist) support the conclusion that attending the festivals caused a transformative experience? Why or why not? (Hint: Think about the need for a comparison group.)

d) Based on what you've read, was the variable of "transformative experience" operationalized as a physiological, observational, or self-report measure?

In this case, the journalist's coverage of the study doesn't do the original study justice, as I discovered when I read the original empirical article, which appeared in the journal Nature Communication (you can access it here).There are several important omissions in the journalist's report. One is that the researchers themselves did not use causal language in their work. Instead, their article is titled "Prosocial correlates of transformative experiences at secular multi-day mass gatherings" (my emphasis).

The Study Finds journalist also omitted key methods and variables. For example, one important variable in the original study was that the researchers graphed people's self-reports of having a transformative experience on each day of a 7-day festival (see Figure 2,  here). Using this method, the researchers found that the later the day (going from Day 1 to Day 7), the more likely respondents were to report having a transformative experience. 

e) Why is the above method and result important, in terms of supporting the claim that the music festivals can increase people's tendency to have a transformative experience? (Hint: think about temporal precedence.)

Another important detail omitted by the Study Finds story, is that in addition to asking people to self-report their transformative experiences, researchers also asked participants to imagine playing a "dictator game". In this game, participants are given 10 valuable tickets. Participants can then donate some number of their tickets to randomly selected stranger. The dictator game is an observational measure of generosity to strangers. The researchers found that festival participants chose to donate between 50 and 75% of their tickets to the stranger. This was much higher than the level of donations typically found, as reported in a meta-analysis of all previous studies using the dictator game (which found an average of 26%). 

f)  This dictator game result is a type of quasi-experiment--we might call it a non-equivalent control group, posttest-only design. What is the quasi-independent variable--that is, what are the two levels of the non-equivalent control group (and why are they "non-equivalent")? What is the dependent variable? 

g) The result suggests that attending the music festival might cause people to be more generous in the dictator game, but there might be a selection effect. Why? 

 

Where do you get more creative ideas?

The research tested whether people came up with more, and more creative, ideas when working face-to-face compared to Zoom.
Photo credit: MBI/Alamy Stock Photo

Apparently, researchers have found that Zoom can stifle pairs' creativity. Here's how an empirical study on this topic was introduced by journalists writing for The Guardian:

While the benefits of Zoom and other videoconferencing tools made them indispensable in the pandemic, the research suggests that heavy reliance on the technology comes at a cost to creative thinking.

a) Before moving on, how might you design an experiment to test which type of meeting would be better for creativity? What would your study's independent variable (IV) be? What would the dependent variable (DV) be? 

Let's see how the researchers did it. According to the journalist,  

... the researchers recruited more than 600 volunteers who were paired up to tackle a creativity task either together in the same room, or virtually over Zoom.

The pairs had five minutes to come up with creative uses for a Frisbee or bubble wrap and a minute to select their best idea. Independent judges ruled that turning a Frisbee into a plate was less creative than using it to knock fruit from a tree, while using bubble wrap to send morse code messages was more creative than using it to protect a baby. Overall, those who worked over Zoom had 20% fewer ideas than those who met face to face.

b) In the study above, what was the IV and what were its levels?

c) Was the IV manipulated as between groups or within-groups?

d) What was the DV and what were its possible levels?

e) What simple experiment is this: Posttest-only? Pretest-posttest? Repeated measures? Concurrent measures? 

Now here's a second experiment: 

The same effect was apparent in the real world. In a field study, the researchers analysed ideas for new products generated by 1,490 engineers for a multinational company. The engineers, who were in Finland, Hungary, India, Israel and Portugal, were randomly paired up and given an hour or so to brainstorm products either in person or over Webex videoconferencing. They then selected their best idea.

...the researchers report that the engineers produced more ideas, and more innovative ideas, when working face to face. “They are not only generating a larger number of creative ideas, but their best idea is better,” 

f) This experiment was billed as a "field study." What does that mean? 

g) In the study above, what was the IV and what were its levels?

h) Was the IV manipulated as between groups or within-groups?

i) What were the two DVs in this study?

j) What simple experiment is this: Posttest-only? Pretest-posttest? Repeated measures? Concurrent measures? 

k) The second study is a replication of the first one. What role does replication play in the scientific process? If you've studied Chapter 14, you could identify whether this was a direct replication, a replication-plus-extension, or a conceptual replication.

Interestingly, the journalist notes, "it is unclear whether the impact on creativity holds for larger teams." This statement is a good example of external validity as it applies to situations (rather than the most common focus of external validity, which is on a study's generalizability to participants).

l) What other situations might we wonder about as we consider the external validity of this research? 

The original empirical study was published in Nature. The original study contains a number of interesting additional DVs, including potential mediators for the overall effect on creativity. 

How much weekly exercise can prevent depression?

The weekly recommendation for exercise is 150 minutes.
Photo credit: blasbike/Deposit Photos

A journalist wrote about a recent meta-analysis in JAMA Psychiatry which evaluated the connection between weekly time spent exercising and levels of depression. The headline of the Washington Post summary read, "Exercising even half the recommended amount can help prevent depression". Let's delve into this research, as covered by the journalist in their brief report. You can access the original empirical work here. 

Here's the opening message:

Already known to help ease depression, regular exercise may also help prevent it, with people who exercised just half the recommended weekly amount lowering their risk for depression by 18 percent.... However, those who were more active, meeting at least the minimum recommended physical activity level, reduced their risk for depression by 25 percent, compared with inactive people.

The empirical research was a meta analysis, and it seems that all of the studies involved longitudinal designs: 

The findings stem from the analysis of data from 15 studies, involving 191,130 adults who were tracked for at least three years.

a) What are the steps you'd typically follow if you were conducting a meta-analysis? 

According to the journalist, the recommended amount of weekly exercise is 150 minutes. Therefore, it seems that working out only 75 minute per week (by walking, for example) does predict lower rates of depression over time. 

Let's apply the four big validities to this study. 

b) Construct validity: We'd need to address the construct validity of each major variable--exercise and depression. It's likely you can measure exercise in multiple ways (self-report, observational, physiological). Can you tell from the journalist's report how the studies measured exercise? 

What about depression--can you tell from the report how the studies measured depression?

I would assume that across the 15 studies, there were a variety of ways that depression and exercise were measured. Is this variability of operationalizations a strength of the meta-analysis, or a weakness? 

c) Statistical validity. The journalist merely reports a single estimate:  "people who exercised just half the recommended weekly amount lowering their risk for depression by 18 percent." But the empirical article gives us a 95% CI for this estimate: [13%, 23%].  In addition the journalist writes, "meeting at least the minimum recommended physical activity level, reduced their risk for depression by 25 percent, compared with inactive people." Again the empirical article gives the 95% CI as [18%, 32%%]. 

In your own words, explain what these two 95% CIs mean.

d) External validity. As we consider external validity, you might be tempted to focus on the total number of participants in the meta-analysis (more than 191,000 over 15 studies). Why would it be incorrect to use sample size to make decisions about external validity? 

e) Internal validity. All 15 studies in this analysis were prospective, or longitudinal. What does that mean? Use the term "temporal precedence" in your response.

Now to focus on internal validity specifically: Can you think a possible third variable that might be associated with both exercise and depression over time? (It would be potentially important for the researchers to control for this variable if they hoped to support a causal claim.)