Everyday Research Methods

It matters how professors talk to students....

Students imagined asking a question in class, and getting one of three responses from the professor. Photo credit: Ground Picture/Shutterstock

This example gives you practice extending a simple experiment (Chapter 10) into a factorial design (Chapter 12).

Think back to the last time you asked a question in your research methods class--or any other class. What was the professor's reaction to your question? Maybe they said, "That's a great question!" or maybe they said, "Didn't we just go over this?" As you probably know, it matters how professors respond--you might feel validated, encouraged, rejected, or ignored, all of these, depending on the professor's tone. 

A group of researchers studied how a professor's responses can make students feel, especially in STEM (science, technology, engineering, and math) courses. They used a scenario method in which students imagined asking a question in class. The researchers summarized their research for a general audience on the SPSP news website.  Here's why they were interested in this topic: 

Students' confidence and belonging may influence their interest and motivation in STEM. Confident people believe that they can succeed. Belonging refers to how much people feel they fit in and are accepted. We expected that when students imagined their instructors responding to their questions in a positive way, students would feel more confident and accepted, which would boost their motivation and interest in STEM.

Study 1 (Chapter 10)

The researchers conducted five studies. Let's talk about Study 1: 

College students imagined receiving a response from an instructor to a question they asked.  The response was either positive ("That's a great question, I'm glad you brought that up"), negative ( "I'm not sure why you're asking this question, we went over this already") or neutral ("We're actually out of time today, so please hold your question till next time"). They then reported their feelings of confidence, belonging, interest in joining the research lab, and intentions to recruit others to the lab.

When students imagined receiving a positive rather than a negative or neutral response from their STEM instructor after asking a question, they felt more confidence and belonging in STEM.

a) Classify the variables in this study: (Hint: There's one IV and four DVs)

Variable name	What are the variable's possible levels?	Is this manipulated or measured?	Is this an IV or a DV?	For IVs: is it within groups or independent groups?

b) Which kind of experiment is Study 1: Posttest only? Pretest/posttest? Concurrent measures? Repeated measures? How do you know?

c) Sketch a bar graph or line graph of the results they described (select one of the four DVs). 

Study 2 (Chapter 12)

In another study, the researchers compared whether the feedback's effects depended on the type of student. Here's the theoretical reasoning: 

Some students may be more vulnerable to these effects because they doubt their belonging or ability in certain areas. For example, being 'bad at math' is a common stereotype that negatively affects women's experiences in STEM. Women often feel uncertain about their belonging in STEM and perform worse when reminded of negative gender stereotypes about women's math abilities. Based on these ideas, we reasoned that women, compared to men, would react more strongly to imagining receiving certain types of instructor responses to their questions in STEM contexts.

Here's the study's results:

Indeed, compared to men, women reported greater confidence, belonging, and intentions to join a STEM lab after imagining their instructor responding to their question with, "That's a great question, I'm glad you brought that up." Women also reported lower confidence, belonging, and intentions to join a STEM lab after imagining a negative instructor response or even a neutral instructor response of, "We're actually out of time today, so please hold your question till next time."

d) Classify the variables in this study: (Hint: The IV and three DVs are the same as before, but now there is a participant variable (PV)/independent variable :

Variable name	What are the variable's possible levels?	Is this manipulated or measured?	Is this an IV, PV or a DV?	For IVs and PVs: is it within groups or independent groups?

e) Which kind of experiment is Study 2: It's a factorial design, so state it like this ___ X ___ and state whether it's an independent groups factorial, within groups factorial, or mixed factorial design. 

f) Sketch a bar graph or line graph of the results they described (select the confidence DV). I recommend putting the "feedback condition" variable on the X axis and the Gender PV as a line or bar color. 

g) Can you detect an interaction in the data? How about a main effect for type of feedback? 

The Gender Shades project

The design was a 2x2 factorial. What were the IVs?
Photo credit: Djomas/Shutterstock

Researcher Joy Buolamwini of MIT has conducted a fascinating study of how accurately image-processing algorithms detect and classify the faces of people from different racial and gender backgrounds. She coined the term "the coded gaze" for a bias she found in these algorithms. Her research was inspired by a personal experience.  Joy is a Black woman, and she noticed that computers were able to correctly recognize the faces of her light-skinned friends. However, when she gave the facial recognition computers an image of herself, they did not accurately detect it was a face at all. Other times, it detected her face, but misgendered it as male. The tools she tried included commercially-available image processors from Microsoft, Kairos, IBM, and Face++. 

Joy's research is an example of how artificial intelligence (AI) tools can recreate biases in the world. Specifically, because AI is trained on a corpus of images and text that contains gender and racial biases, its algorithms will tend to reproduce those biases. 

To get started with the questions, watch this 5-minute video in which she describes the overall patterns. 

The research Dr. Buolamwini designed can be described as a factorial design. Let's walk through it. In addition to watching the video, you could also explore the website, www.gendershades.org. 

Buolamwini's project systematically tested how accurate software from three companies (IBM, Microsoft, and Face ++)  was, across a large sample of faces that varied in terms of skin type and gender. Buolamwini used publicly-available photos from members of parliament from 8 countries. Because she used known people, she knew ahead of time which faces were men and which were women. 

In her analysis, the dependent variable was accuracy, and its levels were yes (the computer correctly identified the target's gender) and no (the computer did not)

Here are the accuracy results from the IBM tool, based on data presented in the video (minute 2:42-3:00)

Darker subjects accuracy: 77.6%

Lighter subjects accuracy: 96.8%

However, these two percentages could be further broken down by target sex:

Darker male accuracy: 88%

Darker female accuracy: 65.3%

Lighter male accuracy: 99.7%

Lighter female accuracy:  92.9%

As you can see, the results mean that IBM's product failed to identify darker skinned women up to 1/3 of the time! The other two products were just a little bit better. 

Buolamwini points to the fact that although we need to know more about why this bias exists, it can almost certainly traced back to the fact that the companies didn't train their AI tools on enough dark-skinned male and female faces. 

Questions

a) If you consider only the data from IBM, what were the two independent variables in this analysis? What were each IV's levels? 

b) Convert the data above (The four categories) to a carefully labelled 2x2 grid, as you see throughout Chapter 12 of your text.

c) Use your grid (and your calculator) to estimate the marginal means for each of the IVs. (Hint: You can see two of the marginal means above!)

d) Now make a bar graph or a line graph of the four data points. Label your axes mindfully.

e) Next, assuming that differences larger than 2% are statistically significant, write sentences for each main effect and interaction. Use these sentence templates:

There is/is not a main effect for _[IV #1]_, such that ____ faces are recognized more accurately than ____ faces.

There is/is not a main effect for _[IV #2]_, such that ____ faces are recognized more accurately than ____ faces.

There is/is not an interaction between  _[IV #1] and [IV #2]_, such that for darker skinned faces, ____ are recognized more accurately than ____, but for lighter skinned faces, ____ are recognized more accurately than_____.

f) Buolamwini's research illustrates that machines are not neutral. What is one real-world application of this research that you can anticipate? 

 

 

 

 

Culture and the "marshmallow test"

Cultural practices around both snacks and gift-giving were proposed as one of the mechanisms behind this study's results. Photo credit: Prostock-studio/Shutterstock

In a recent study, researcher Yuko Munakata tested the self-regulation and self-control abilities of children in two cultures: The U.S. and Japan. She published the research in the empirical journal Psychological Science, but also wrote an accessible description of it for the popular online magazine, Scientific American. The heading wrote, A classic test of self-control can carry complex cultural biases.

In case you are unfamiliar with the marshmallow test, here is Dr. Munakata's description of it:

The classic marshmallow test, invented by the late psychologist Walter Mischel, involves presenting a child with one marshmallow and explaining that they can have it now or they can have two later if they wait until the marshmallow giver returns. Children sit alone in a room with the tempting treat. Psychologists have generally viewed the length of time kids resist the sweet as a measure of their self-control: how effectively they can inhibit impulsive behaviors and work toward longer-term goals.

In their study, the research team compared marshmallow waiting time among four- and five-year-old children in two cultures. Here's a description of part of the study:

We worked with 144 children in both the U.S. and Japan. We first ensured that all of these four- and five-year-olds had eaten marshmallows before. We asked parents how often their children waited to eat until others were served and how well their children could suppress impulsive behaviors.

Then the kids faced the classic test: Did they want one marshmallow now or two later? Most children in the U.S. waited less than four minutes before tasting the one marshmallow. Most children in Japan waited for two marshmallows for the maximum possible time—15 minutes!

But the study did not stop there:  

If we had stopped there, we might have simply concluded that Japanese kids have better self-control. But we conducted another test. We presented children with a wrapped gift and told them that they could open it now or they could have two gifts if they waited. The pattern flipped. Most children in Japan waited less than five minutes before unwrapping the gift, while most children in the U.S. waited the maximum 15 minutes or close to it.

In a moment, we'll go over the researchers' reasoning about why they found this cultural difference. But for now, see if you can identify the features of this factorial design. 

a) Take a moment to classify the variables. There are three of them!

Variable name	What are the variable's levels?	Is this manipulated or measured?	Is this an IV, PV*, or a DV?	For IVs and PVs: is it within groups or independent groups?

*PV means participant variable--it is treated as an IV, but it is not actually manipulated. Examples of PVs are demographic variables such as gender, ethnicity, or class year. 

b) This was a factorial design. What kind? Your answer should read ___ x ___, and mention whether it was an independent groups factorial, a within-groups factorial, or a mixed factorial design (hint: to decide, look back at how you coded the IVs and PVs on the table. 

c) See if you can sketch the results the author described. You can choose a bar graph or a line graph. Remember to put the DV on the y-axis, one IV/PV on the x-axis, and the other IV/PV as a line or bar color. 

d) Do you think this factorial design's results found an interaction? If so, describe it in a sentence or two. 

Now let's read the researcher's explanation for why they that cultural differences in self-control depended on the task used:

What was going on? It seems that with repeated experiences delaying gratification, children can develop habits that make it easier to wait in the future. According to the reports from their parents, the Japanese children had stronger habits of waiting to eat than the U.S. children. The stronger these habits, the longer the children waited for two marshmallows. But when it comes to opening gifts, children in the U.S. likely experience waiting more consistently. Birthday presents can sit on a table, unopened, until the end of a party. Christmas presents may sit under a tree for days before they can be unwrapped on December 25. In Japan, people give gifts year-round on simple occasions that do not involve traditions of waiting.

This result was fascinating to me as a cultural psychologist! Turns out there was no "culture main effect"--there's no overall difference in self control between U.S. and Japanese children. Instead, culture interacts with task, so kids do better at the self-control task they practice more in their culture.

 

In U.S. football, Whiter-looking men more likely to be head coaches

Football coaching seems to have a stereotype problem
Photo credit: dotshock/Shutterstock

Here's a story from the website Psypost. The headline reads, "Study of U.S. football coaches finds the more White you look, the more likely you are to be head coach".

This report is a study based on a well-known disparity in U.S. football (both college and pro): Even though 57% of college players are Black, Whites are more likely to be in football coach positions. Among college teams, for example, position coaches are around 31% Black; only 9% of head coaches are Black. The higher you look in the coaching leadership hierarchy, the more likely you are to see White men (no women are head coaches of college teams). 

A research team got together to document this disparity and also to study whether men who looked especially "White" were even more likely to be head coaches. In their study, the research team obtained coaches' photos and titles from team publicity programs; all coaches were in the college Football Bowl Subdivision (FBS). Over 100 Black and White coaches were in the sample. In addition to recording the coaches' roles (such as position coach, defensive or offensive coordinator, or head coach) the team also coded how "stereotypically White" or "stereotypically Black" each coach appeared. "Stereotypicality" was coded by showing the photographs to a new sample of Americans, who were asked to rate each coach on a 5-point scale according to how stereotypically White (or, if they were Black, how stereotypically Black) they looked. 

Here's one of the researchers, Melissa J. Williams, explaining to the Psypost journalist some of the background for the study:

“Race and ethnicity have both internal aspects (such as the importance a person places on their cultural heritage) and external aspects (such as how other people perceive a person’s physical appearance). Each of these aspects has a different effect on the role that race plays in a person’s life.  In terms of the specific setting we explored in this paper, football leadership represents an interesting collision of historical stereotypes about Black and White individuals. Black individuals, especially Black men, are stereotyped as good at sports and as physically imposing – characteristics that go with being good at football. Whereas White individuals, especially White men, are stereotyped as confident and charismatic – characteristics that go with being good at leadership. That combination of stereotypes got us interested in looking at who makes it to the top in football coaching, and how their appearance plays into their success.”

Here's how the journalist summarized the study's results:

The findings revealed that White coaches were more likely than Black coaches to occupy leadership roles. Moreover, among White coaches, the more stereotypically White their appearance, the greater the likelihood they occupied a leadership role. When it came to specific roles, appearing stereotypically White increased the likelihood of being a head coach among White coaches. By contrast, appearing stereotypically Black decreased the likelihood of being a head coach among Black coaches.

a) This was a correlational study. What were the measured variables? (There are three variables mentioned in the paragraph above).

b) This result presents a moderator. One way to draw it is to create two matching scatterplots: One for White coaches and one for Black coaches. What would go on the x and y axes of the two scatterplots? What would the slope of each scatterplot be?

c) Write a moderator sentence of this form: _________ moderates the relationship between ________ and __________, such that ________________.

After documenting the correlation between appearing White and being a head coach, the team conducted a second study, an experiment: 

In a second study, Williams and team had a sample of 375 football fans view photos of real college football players. Through a series of trials, participants were shown side-by-side images of two football coaches [both White or both Black], with one being high and the other low in racial stereotypicality. Participants were asked to select which coach they felt would be promoted to head coach in the future.

The results revealed that when the two players were White, participants chose the more stereotypical photo as more likely to become head coach in 70% of trials. But when the two players were Black, participants chose the more stereotypical photo only 52% of the time.Thus, stereotypicality had a stronger effect on being perceived as a leader for White compared to Black football players.

c) What are the independent variables in this experiment? What is the dependent variable?

d) To study the "stereotypicality" variable, this second study used a concurrent measures design. Can you explain why?

e) The first study was correlational and the second was experimental. Why is it helpful to have one of each study type in this investigation? 

At the end of the Psypost article, the researchers were quoted about their study: 

Overall, the researchers said their findings, “provide insight into the insidious nature of racial stereotypes, the role of physical appearance in perpetuating them, and how leadership attainment may be constrained for those who do not look the part.”

Here's a link to the empirical journal article, which includes photos of coaches who were coded as higher or lower on the stereotypicality scale: Personality and Social Psychology Bulletin. 

 

 

 

 

 

 

Time alone with your thoughts?

Credit: Ground Picture/Shutterstock

Imagine sitting alone with your thoughts for 20 minutes. You wouldn't be allowed to distract yourself with a book, phone, or even walking around. How much do you think you would you enjoy it?  

When this question was asked of participants in a series of studies, most reported that they'd not enjoy time alone very much.  But this study also tested how well their predictions matched their actual experience. The empirical research was published in the Journal of Experimental Psychology: General, and  summarized in a trade magazine for the American Psychological Association. According to the journalist, 

...study lead author Aya Hatano, PhD, of Kyoto University in Japan[, said] “Our research suggests that individuals have difficulty appreciating just how engaging thinking can be. That could explain why people prefer keeping themselves busy with devices and other distractions, rather than taking a moment for reflection and imagination in daily life.”

The research team conducted six studies. Here is an overview of some of them: 

In a series of six experiments with a total of 259 participants, the researchers compared people’s predictions of how much they would enjoy simply sitting and thinking with their actual experience of doing so. In the first experiment, they asked people to predict how much they would enjoy sitting alone with their thoughts for 20 minutes, without being allowed to do anything distracting such as reading, walking around or looking at a smartphone. Afterward, participants reported how much they had enjoyed it.

The researchers found that people enjoyed spending time with their thoughts significantly more than they had predicted. This held true across variations of the experiment in which participants sat in a bare conference room or in a small, dark tented area with no visual stimulation; variations in which the thinking period lasted for three minutes or for 20 minutes; and one variation in which the researchers asked people to report on their enjoyment midway through the task instead of after it was over. In every case, participants enjoyed thinking more than they had expected to.

a) The researchers conducted six different studies. Do these sound like direct replications? Replication-plus-extensions? or Conceptual replications? (you can skip this question if you haven't done Chapter 14 yet)

b) The journalist identifies these as "experiments," so let's assume that each study contains a manipulated variable. Take this study, as described by the journalist:

The researchers found that people enjoyed spending time with their thoughts significantly more than they had predicted [in a variation of] the experiment in which participants sat in a bare conference room or in a small, dark tented area with no visual stimulation. 

In this study, one of the independent variables was "predicted vs. actual".  What is the other main independent variable, or IV? What is the dependent variable, or DV? 

c) Do the same for this study:

variations in which the thinking period lasted for three minutes or for 20 minutes;

In this study, one of the independent variables was "predicted vs. actual". What is the other IV? What is the DV?

d) And for this one: 

In another experiment, the researchers compared one group of participants’ predictions of how much they would enjoy thinking with another group’s predictions of how much they would enjoy checking the news on the internet. Again, the researchers found that people underestimated their enjoyment of thinking. The thinking group expected to enjoy the task significantly less than the news-checking group, but afterward, the two groups reported similar enjoyment levels.

In this study, one of the independent variables was "predicted vs. actual". What is the other IV? What is the DV?

e) Challenge question: For the study above (comparing time alone to looking at news on the internet)  and graph the result. You can compare your graph to Figure 6 in the empirical article (available next to the APA journalist's story, here).

Here's a final caveat about the results. I appreciated this statement because it puts the statistical outcome in concrete and realistic terms: 

It is important to note that participants did not rate thinking as an extremely enjoyable task, but simply as more enjoyable than they thought it would be, according to Murayama. On average, participants’ enjoyment level was around 3 to 4 on a 7-point scale. Future research should delve into which types of thinking are most enjoyable and motivating, according to Murayama. “Not all thinking is intrinsically rewarding, and in fact some people are prone to vicious cycles of negative thinking,” he said.

 

 

Diet + Exercise: A factorial example

The study found that both exercise and diet are important. Is the effect additive or interactive? Photo credit:Dragana Gordic/Shutterstock

You might have heard someone claim that because they exercise regularly, they can eat whatever they want and still be healthy. When people say this, they seem to mean that either exercise OR diet can be sufficient to foster a healthy lifestyle. However, a recent study counters this "either/or" view. Like many other studies, this one found evidence that both diet and exercise contribute to health outcomes in adults.

In this blog post, we'll explore how the study can be viewed as a factorial design (Chapter 12), and we will discuss how its main finding can map on to different factorial outcomes. 

The empirical study appeared in the British Journal of Sports Medicine, which may be paywalled. I learned about the study from number of journalists who covered it, including this accessible summary in Healthline, which we'll use in this post.  

Here are the opening bullet points from the Healthline story: 

• A new study finds that in order to live longer, people need to eat healthily and exercise regularly.
• While a balanced diet and regular exercise independently have plenty of health benefits, the study notes the greatest reduction in mortality risk comes from combining diet and exercise together.

a) These bullet points mention the study's three main variables. One of the variables is mortality risk, or longevity (lifespan). What are the other two variables?

b) Chapter 12 concerns factorial designs, in which there are two (or more) independent variables (IVs) and one dependent variable (DV). Reflect back on the three variables you listed in a), above, and identify which are the IVs and which is the DV.  In addition, identify the levels of each IV. 

Here are some details about the study's method and results, as summarized by the journalist:

For the study, the researchers evaluated the health and exercise data of 346,627 individuals over the course of 11 years. They tracked how much exercise each participant completed in an average week along with how intense their physical activity was. The team also followed the participants’ eating habits. During the study window, 13,869 participants died — 2,650 from heart disease and 4,522 from adiposity-related cancers.

The research team found that any type of regular exercise was associated with a lower risk of mortality, and those who both ate well and exercised had the lowest risk of dying.

In addition, while exercise and diet are independently linked to a lower risk of mortality, high levels of exercise cannot entirely mitigate the harms of a poor diet.

I made up three patterns of data to illustrate possible outcomes of this study. You can see the graphs below, labelled Outcomes A, B, and C. Take a moment to orient yourself to the graphs, noticing how the DV is on the y-axis, one IV is on the x-axis, and the other IV is depicted using line color:

 

c) Which of these outcomes (A, B, or C, above) is consistent with the idea that EITHER exercise OR diet alone is sufficient to maintain a longer lifespan? Explain your reasoning.

d) The journalist wrote that "The research team found that any type of regular exercise was associated with a lower risk of mortality, and those who both ate well and exercised had the lowest risk of dying [i.e., the longest lifespan].”  Two of these outcomes (A, B, or C) are consistent with this statement. Which ones? 

e) In your own words, describe the difference between Outcome A and Outcome B.

f) Based on what the journalist wrote, can you tell which of these two (Outcome A or Outcome B) the study found? Specifically, which one could go with how the journalist described the study's outcome:

“In our study, those who ate a poor-quality diet and were active still had substantially reduced mortality risk than those who ate a poor-quality diet and were inactive,” Melody Ding, MPH, PhD, the lead author of the study and an associate professor of health and medicine in the Sydney School of Public Health at the University of Sydney, told Healthline. “It’s just that you had the maximum risk reduction when doing both things right.”

g) Just for more practice: Two of the outcomes above depict an interaction effect (an interaction between diet and exercise). Which 2 outcomes are they? Explain your reasoning--how do you know?

Suggested answers:

a) Besides longevity, a second variable is diet quality and a third variable is exercise habits.

b) Longevity is the DV. Diet quality is an IV (levels could be good vs. poor diet). Exercise habits are also an IV (levels good be regular exercise vs. no exercise)..

c) Outcome C. Outcome C shows that you have the same lifespan if you exercise or eat a good diet, and that doing both does not increase your lifespan any more than doing one or the other (all three conditions--exercise alone, diet alone, and both--show the same lifespan). Note that this is not actually the result that the study found.

d) Both Outcome A and Outcome B are consistent with the journalist's statement. 

e) The difference between Outcomes A and B can be summarized with the terms "additive" versus "interactive."
Outcome A shows an additive effect--A good diet adds (in this example) 3 years to your life; exercising adds 3 years, and if you do both, it adds 6 years. One is good, both is better.  
Outcome B shows an interactive, or multiplicative effect. It shows that if you do both diet AND exercise, you have an especially long lifespan. That is, in this example, a good diet adds 3 years to your life; exercising adds 3 years, and if you do both, it adds 11 years. One is good, both is amazing (kaboom!). In Chapter 12 you'll learn that this is an interaction--the combination of the two IVs results in something even bigger than simply adding the two together.

f) In my view, we can't tell which outcome the study found based on what the journalist wrote. However, since they used the terms "both" rather than "especially", it seems like it might be Outcome A. My reading of the empirical study backs this up--it seems to be an additive effect (not interactive). 

g) Outcome B and C both show interactions, but of different types. First, you can see the interactions in both because the lines are not parallel--there are differences of differences. That is, the effect of diet is different, depending on if we're talking about regular exercise or no exercise. 

In Outcome B, there is an interactive, or multiplicative effect--I sometimes call this an "especially" interaction. It shows that if you do both diet AND exercise, you have an especially long lifespan (compared to doing either one alone).

In Outcome C, there is a different kind of interaction--I might call it an "only if" or an "it depends" interaction. You could explain it by saying that lifespan is reduced only if you don't exercise AND don't eat well. If you exercise, or diet, or both, you'll live longer. 

That baby smell...

The study found an interaction effect: Gender of participant interacted with the smell condition they were in. Credit: HayDmitriy/Deposit Photos

If you’ve ever smelled a baby’s head (I know....it can be irresistible!), you probably inhaled a variety of chemicals, including one called HEX.

A group of researchers recently published a study in which they exposed men and women to this particular chemical. They suggested that, given human evolutionary theory and emerging research on how different smells affect us, the smell of HEX might impact people's behavior. In the study, they focused on the behavior of aggression, using a laboratory game. The research was summarized by a journalist for the Voice of America.

The researchers’ hypotheses were derived from human evolutionary theory related to aggression:

…lead author Eva Mishor, who was studying signals of aggression for her doctorate at the Weizmann Institute, noted that in animals, female aggression is usually aimed at defending their young, while male aggression is often directed at the offspring themselves.

Specifically, the authors demonstrated that in a childrearing context, smelling HEX (the baby-head chemical) led to different responses in males than females.

The study tested people's responses to HEX using rigged games designed to aggravate the player. In one game, when the aggravated player is allowed to win, he or she gets to blast the opponent with a loud noise. The louder the noise, the higher the scientists rated the player's aggression level.

[Among players who] sniffed HEX before playing [compared to players who had sniffed a control chemical], women's blasts were louder and men's were quieter.

In case you didn't notice, this study used a 2x2 factorial design with an IV and a PV. Let’s walk through it.

a. What is the first factor (the independent variable?) What are its 2 levels? Was it manipulated as independent-groups or within groups?

b. What is the second factor (the participant variable?) What are its 2 levels? Was this variable independent groups or within groups?

c. What was the dependent variable?

d. Read the description of the results in the last quoted paragraph above. Then, sketch a little graph of the results they are describing (please assume that, in the non-HEX control condition, men are a bit more aggressive than women).

e. The results of the study show an interaction. Describe the interaction in your own words. 

f. The journalist describes the results like so:

The effect was somewhat subtle. On a six-point scale, the differences were, on average, roughly between half a point and less than a point in either direction.

Here the journalist seems to be characterizing the effect size, and seems to be referring to the 95%CI of the difference. This is unique; most journalists do not mention the size of such effects or their CI.

Suggested answers:

a. The first factor was exposure to HEX (or not). This variable was manipulated as a between-groups variable—people played the aggression-inducing game either after smelling HEX OR after smelling a control (non-HEX) scent (i.e., there were different people in each condition). 

b. The second factor was the participant variable of participant gender. It’s a participant variable because there are two levels (male and female) that cannot be truly manipulated. This is an independent-groups variable because people were either male or female.

c. The dependent variable was level of aggression, as operationalized as noise blasts.

d. Here's a sketch I made, based only on the journalist's description. You should look at the actual results in Figure 2 of the open-access article in Science Advances.  Because the researchers also coded noise blast intensity over time, their results figure includes that factor as well. 

 

 

 

Does mindfulness make people selfish? It depends....

Mindfulness instructions appeared to interact with people's independence/interdependence in this set of factorial studies.  Photo: Rob Wilkinson/Alamy Stock Photo

A new set of studies on mindfulness and selfishness provides an opportunity to practice skills related to factorial designs (Chapter 12). Let's get started.

 

Here's the theoretical background of the study, as described by a BBC journalist. He explains that mindfulness can 

sharpen our concentration, extend our working memory and boost our mental flexibility. With practice, we should become less emotionally reactive – allowing us to deal with our problems more calmly. 

One ‘benefit’ that you might not expect to gain, however, is heightened egotism. Yet a recent study suggests that, in some contexts, practicing mindfulness really can exaggerate some people’s selfish tendencies. With their increased inward focus, they seem to forget about others, and are less willing to help those in need. 

The journalist then describes two studies run by psychologist Michael Poulin. The first compared the effects of a mindfulness meditation session among people from two different cultures. Notice the term "depends" in this description, which is a signal that the study was investigating an interaction. The pair of studies comes....

from Michael Poulin, an associate professor in psychology at the State University of New York at Buffalo, who wanted to investigate whether the effects of mindfulness might depend on its cultural context and the existing values of the people who are practicing it.

He was particularly interested in the ways people think about themselves – their “self-construal”. Some people take a more independent viewpoint, focused on personal characteristics. If they are asked to describe themselves, they might emphasise their intelligence or their sense of humour. People with an interdependent view, on the other hand, tend to think of themselves in terms of their relations to others. 

Read this description of the study, and identify the first independent variable (IV, or factor), the second  IV (factor), and the dependent variable:

... Poulin invited 366 college students into the lab and first gave them a questionnaire measuring their independence or interdependence. 

Half were then asked to perform a meditation focused on the sensation of breathing. The control group were given a “sham” meditation that involved sitting and letting their mind wander for 15 minutes. The exercise may have been relaxing, but it wasn’t designed to increase their mindfulness. 

Next came a test of pro-social behaviour, in which the students were told about a new project to help fund a charity for the homeless. They were then given the opportunity to stuff envelopes ...which would be sent to the university’s alumni – but they were told there was no obligation to do so, if they wished to leave early.

Now for the results:

Sure enough, Poulin found that the effects of the meditation depended on people’s existing attitudes. If they were already interdependent, then the people who took the mindfulness exercise were willing to spend much more time on the charitable task; overall, they stuffed about 17% more envelopes than the control group. If they were independent-minded, however, the exact opposite occurred – the mindfulness had made them even more self-centered, so they were less willing to help the homeless. Overall, they stuffed around 15% fewer envelopes than the control group. 

Questions about Study 1:

a) What was the study's first factor (Hint: It was an IV)? Was it independent groups or within groups?

b) What was the second factor? (hint: it was a participant variable, or PV). Was it independent groups or within groups?

c) What was the DV?

d) State the design of this study using this template: "__x__ , [independent-groups/within-groups/mixed] factorial"

e) Sketch a bar or line graph of the results, as described in the paragraph above. use Figures 12.16 and 12.17 as models.  Do you see an interaction here? 

Now here is a second study, as described by the same journalist:

To be sure the finding was robust, Poulin’s team conducted a second experiment, in which the participants were first given a short text written either in the first-person singular (I), or first-person plural (we). As they read the text, they had to click on all the pronouns – a simple task known to prime either independent or interdependent thinking. They then completed the meditation tasks and, to test their pro-sociality, were asked whether they wanted to devote time to chat online with potential donors for the homelessness charity. 

Once again, the mindfulness exercise exaggerated the effects of their self-perception, driving increased altruism among the interdependent-minded, and decreased altruism among the more independent-minded. 

Questions about Study 2:

f) What was Study 2's first factor (Hint: It was an IV)? Was this one independent groups or within groups?

g) What was the second factor? (This was also an IV). Was it independent groups or within groups?

h) What was the DV in this study?

i) State the design of this study using this template: "__x__ , [independent-groups/within-groups/mixed] factorial"

j) Sketch a bar or line graph of the results, as described in the paragraph above. use Figure 12.11 as a model. Do you see an interaction? (You should, because this study was described with the phrase, "it depends"!)

k) There's causal language in the journalist's summary (e.g., " the mindfulness exercise ...was driving increased altruism" and " the mindfulness had made them even more self-centered"  Is causal language warranted here? How come? 

Note:Poulin's study has been peer reviewed and will be appearing in the journal Psychological Science, in the future. You can read an early version of the paper here. 

 

Saturated fat can impair concentration

Breakfast of (concentration) champions? Maybe not....Photo credit: Kyndall Ramirez/Unsplash

A headline at Psych Central reads, "High fat meal may impair concentration." Does the study behind the headline support that causal claim?

At first, you might assume that this came from a correlational study on diet and concentration. However, it turns out the research summarized in the article was experimental: It manipulated the saturated fat content of one specific meal. In the study, researchers created two versions of a breakfast--one made with saturated fat and the other made with sunflower oil, which is also high in dietary fat but is not the saturated type. 

The journalist describes the method of the study this way: 

Women in the study [n = 52] completed a baseline assessment of their attention during a morning visit to the lab. The tool, called a continuous performance test, is a measure of sustained attention, concentration and reaction time based on 10 minutes of computer-based activities.

The high-fat meal followed: eggs, biscuits, turkey sausage and gravy containing 60 grams of fat, either a palmitic acid-based oil high in saturated fat or the lower-saturated-fat sunflower oil. Both meals totaled 930 calories and were designed to mimic the contents of various fast-food meals such as a Burger King double whopper with cheese or a McDonald’s Big Mac and medium fries.

Five hours later, the women took the continuous performance test again.

Between one and four weeks later, they repeated these steps, eating the opposite meal of what they had eaten on the first visit.

After eating the meal high in saturated fat, all of the participating women were, on average, 11 percent less able to detect target stimuli in the attention assessment.

(The original empirical journal article is here; it might be paywalled)

Questions:

a) What was the independent variable (IV) in this study and what were its levels?

b) What was the dependent variable?

c) Was the IV manipulated as independent groups or within-groups?

d) This design is difficult to classify because it has a pretest and posttest, but the women also experienced both levels of the IV. Therefore it shares features with two of the four designs in Chapter 10. What are those two designs it shares features with? (Note to students: As you can see in this example, the four designs you learned about in Chapter 10 are not the only types of experiments there are, and this study provides an example of alternative designs.) 

e) Sketch the results of this study--using "reduction in concentration" (i.e., a difference score) as the dependent variable. 

Next let's work through the four big validities. 

f) Here's some text related to external validity of this study:

For this work, Madison [the lead author] conducted a secondary analysis of data from Kiecolt-Glaser’s study assessing whether high-fat meals increased fatigue and inflammation among cancer survivors.

and

[the experimental meals]  were designed to mimic the contents of various fast-food meals such as a Burger King double whopper with cheese or a McDonald’s Big Mac and medium fries.

Given the information above, to what kinds of people and what kinds of situations might this research be generalized? How do you know? 

g) The researchers found an 11 percent drop in concentration after the meal high in saturated-fat. What further information would you like to know in order to evaluate the study's statistical conclusions? 

h) Can this study support the causal claim that "A high-fat meal may impair concentration"? Apply the three criteria to the study: Covariance (what did the results show?), temporal precedence (did the method ensure that the meal came before the drop in concentration?), and internal validity (What potential confounds did they control for in the study?) 

i) Here's some information about the study's construct validity: 

Women in the study completed a baseline assessment of their attention during a morning visit to the lab. The tool, called a continuous performance test, is a measure of sustained attention, concentration and reaction time based on 10 minutes of computer-based activities.

What questions would you need to ask to determine if this test is a good measure of people's ability to concentrate? 

Instructors: If you teach Chapter 12, you can also use this example as a mixed factorial design, because the study also included a participant variable of endotoxemia, a condition called "leaky gut." From the journalist's summary: 

After eating the meal high in saturated fat, all of the participating women were, on average, 11 percent less able to detect target stimuli in the attention assessment. Concentration lapses were also apparent in the women with signs of leaky gut: Their response times were more erratic and they were less able to sustain their attention during the 10-minute test.

“If the women had high levels of endotoxemia, it also wiped out the between-meal differences. They were performing poorly no matter what type of fat they ate,” Madison said.

Students could identify the IV, PV, and DV and practice graphing the results described above. 

Thanks to my colleague Stephen Chew, who shared this example.

Research on Native American mascots

The College of William and Mary has phased out its Indian-themed mascot over the years. In 2010, the College replaced its former mascot with "Griffin", above. Photo: Mitchell Layton/Getty Images

Since 2005, the NCAA has required colleges and universities to reflect upon, and if necessary, change sports team mascots that feature depictions of Native-Americans.  Now a team of psychology researchers have published an investigation about support for such mascots among different groups of people. The set of studies  includes both correlational and experimental designs, including one testing whether the mascots can affect people's donations to the universities. 

The lead researcher, Dr. Michael Kraus, told a journalist what led to his interest in the research question. 

“I was at the University of Illinois at the time, and despite their Native American mascot being officially discontinued in 2007, it was surprising to see the mascot on campus in 2013. The continued presence said something about the norms on campus, and what kinds of students feel safe and welcome there,” Kraus said.

The journalist summarized two studies conducted by Kraus and his team. Here's the first:

In a survey of 201 students, the researchers found that those with lower levels of explicit prejudice toward Native Americans reported lower levels of belonging at the University of Illinois.

In other words, students who disagreed with statements such as “It is now unnecessary for the U.S. government to honor their treaty obligations to Native tribes” also tended to disagree with statements such as “I identify with the university” and “I show school spirit regularly.”

In addition, students with higher levels of explicit prejudice, as well as white students, were more likely to have positive views of the retired “Chief” mascot.

Questions

a) This is a bivariate correlational study. What are its two main variables?

b) The journalist gives some clues about how each of these two variables was measured. What kinds of measurements were used--self-report? physiological? observational? What do you think of the items? 

c) Sketch a scatterplot of the results that are described. Label your axes carefully.

d) Can this study support the causal claim? Why or why not? (Apply the three causal criteria; when you come to internal validity, specify at least one third variable that might be associated with both measured variables.)

The empirical article contains several studies on this topic. One of them tested the impact of  the University of Illinois' retired mascot, Fighting Illini, on donations to the school. 

The experiments showed that stereotypic Native American imagery “reduced actual donations to the institution by 5.5%, and was particularly likely to decrease (increase) donations among those low (high) in prejudice,” the researchers wrote.

In the study, researchers had online participants read about different universities, one of which was the University of Illinois. Some participants  were randomly assigned to see the Native American mascot, Chief Illini, next to the school name, while some did not.  Then participants had the chance to distribute $2 in donations to the four universities. When people had been exposed to the  Native American mascot, donations to the University of Illinois decreased. 

But like all research, the study includes some caveats. “Probably the biggest one is that we really only studied these dynamics at the University of Illinois, but we expect these norms to operate similarly everywhere. Someone, maybe us, needs to do that additional work,” Kraus said.

More Questions

e) Is the donation study an experiment or a correlational study? What are its two main variables?

f) When Kraus states that "we expect these norms to operate similarly everywhere," to which of the four big validities is he referring?

You can access the empirical research here.

Instructors: The complete package of studies is a complete set with a frequency claim (via an observational study of mascot use on campus) , a correlational study, and an experimental study.

 

 

 

 

Interested students might learn from Dr. Stephanie Fryberg's work, showing how American Indian high school students respond to being reminded of professional sports mascots that feature American Indians.