Everyday Research Methods

Culture and the "marshmallow test"

Cultural practices around both snacks and gift-giving were proposed as one of the mechanisms behind this study's results. Photo credit: Prostock-studio/Shutterstock

In a recent study, researcher Yuko Munakata tested the self-regulation and self-control abilities of children in two cultures: The U.S. and Japan. She published the research in the empirical journal Psychological Science, but also wrote an accessible description of it for the popular online magazine, Scientific American. The heading wrote, A classic test of self-control can carry complex cultural biases.

In case you are unfamiliar with the marshmallow test, here is Dr. Munakata's description of it:

The classic marshmallow test, invented by the late psychologist Walter Mischel, involves presenting a child with one marshmallow and explaining that they can have it now or they can have two later if they wait until the marshmallow giver returns. Children sit alone in a room with the tempting treat. Psychologists have generally viewed the length of time kids resist the sweet as a measure of their self-control: how effectively they can inhibit impulsive behaviors and work toward longer-term goals.

In their study, the research team compared marshmallow waiting time among four- and five-year-old children in two cultures. Here's a description of part of the study:

We worked with 144 children in both the U.S. and Japan. We first ensured that all of these four- and five-year-olds had eaten marshmallows before. We asked parents how often their children waited to eat until others were served and how well their children could suppress impulsive behaviors.

Then the kids faced the classic test: Did they want one marshmallow now or two later? Most children in the U.S. waited less than four minutes before tasting the one marshmallow. Most children in Japan waited for two marshmallows for the maximum possible time—15 minutes!

But the study did not stop there:  

If we had stopped there, we might have simply concluded that Japanese kids have better self-control. But we conducted another test. We presented children with a wrapped gift and told them that they could open it now or they could have two gifts if they waited. The pattern flipped. Most children in Japan waited less than five minutes before unwrapping the gift, while most children in the U.S. waited the maximum 15 minutes or close to it.

In a moment, we'll go over the researchers' reasoning about why they found this cultural difference. But for now, see if you can identify the features of this factorial design. 

a) Take a moment to classify the variables. There are three of them!

Variable name	What are the variable's levels?	Is this manipulated or measured?	Is this an IV, PV*, or a DV?	For IVs and PVs: is it within groups or independent groups?

*PV means participant variable--it is treated as an IV, but it is not actually manipulated. Examples of PVs are demographic variables such as gender, ethnicity, or class year. 

b) This was a factorial design. What kind? Your answer should read ___ x ___, and mention whether it was an independent groups factorial, a within-groups factorial, or a mixed factorial design (hint: to decide, look back at how you coded the IVs and PVs on the table. 

c) See if you can sketch the results the author described. You can choose a bar graph or a line graph. Remember to put the DV on the y-axis, one IV/PV on the x-axis, and the other IV/PV as a line or bar color. 

d) Do you think this factorial design's results found an interaction? If so, describe it in a sentence or two. 

Now let's read the researcher's explanation for why they that cultural differences in self-control depended on the task used:

What was going on? It seems that with repeated experiences delaying gratification, children can develop habits that make it easier to wait in the future. According to the reports from their parents, the Japanese children had stronger habits of waiting to eat than the U.S. children. The stronger these habits, the longer the children waited for two marshmallows. But when it comes to opening gifts, children in the U.S. likely experience waiting more consistently. Birthday presents can sit on a table, unopened, until the end of a party. Christmas presents may sit under a tree for days before they can be unwrapped on December 25. In Japan, people give gifts year-round on simple occasions that do not involve traditions of waiting.

This result was fascinating to me as a cultural psychologist! Turns out there was no "culture main effect"--there's no overall difference in self control between U.S. and Japanese children. Instead, culture interacts with task, so kids do better at the self-control task they practice more in their culture.

 

How we spot leadership potential in men and women

Which one of these people will benefit most from sharing good ideas with the group? Credit: Dean Drobot/Shutterstock

Imagine you're working at a company and you'd like people to view you as a potential leader. How might you go about it? Perhaps you'd share your good ideas at meetings, hoping they'll notice your talent. Perhaps you'd pursue higher level training to improve your credentials. Maybe you'd point out ways your group's processes could improve. Would these strategies work? The answer may depend on your gender. 

The trouble is that people's prototype of a "leader" is a man. As this journalist describes, when people are asked to draw a picture of a leader, they almost always draw a male figure (by the way, that's a frequency claim). How might people's default assumption--that leaders are male--affect their ability to spot leadership potential in their colleagues? 

According to the journalist,

...getting noticed as a leader in the workplace is more difficult for women than for men. Even when a man and a woman were reading the same words off a script, only the man’s leadership potential was recognized.

Here is a study described in the piece. It's a factorial design (that is, it has two independent variables). As you read, decide what the two IVs are and what their levels are: 

In one experiment, participants were asked to call into a monthly sales team meeting of a fictional insurance company, during which they would hear from either an Eric or an Erica. Later they were asked to rate the speaker on the degree to which he or she had “exhibited leadership,” “influenced the team” or “assumed a leadership role.”

The Erics who spoke up with change-oriented ideas were far more likely to be identified as leaders [rated higher on the leadership questions] than Erics who simply critiqued their team’s performance. But Ericas did not receive a boost in status from sharing ideas even though they were exactly the same as the Erics’.

a) What is the first IV? How many levels does it have and what are they? Is it manipulated or measured? 

b) What is the second IV? How many levels does it have and what are they? Is it manipulated or measured? 

c) What are the three DVs? 

d) Sketch a well-labeled graph of the results of this study, using the DV of "exhibited leadership". Remember to put one of the IVs on the x-axis and the other IV as a line color or bar color. 

Here's another Factorial design described in the journalist's article. 

...in a second experiment, the researchers attempted to investigate the benefits of speaking up in the “wild.” The researchers asked participants in a 36-team skills competition at the U.S. Military Academy at West Point to fill out a series of questionnaires before, during and after their competition.

The authors acknowledged that there were limitations in studying this atypically male-dominated place, but the nature of the competition allowed them to control for athletic ability, cognitive ability and other factors.

Still, after the competition, when participants ranked who they wanted to be the team leader, only the men who had spoken up with ideas received a boost.

“It didn’t matter whether women spoke up 1) almost never, 2) rarely, 3) sometimes, 4) often, or 5) almost always,” Kyle Emich, a professor at Alfred Lerner College of Business and Economics at the University of Delaware, and one of the authors, wrote in an email. “Women did not gain status for speaking up, and subsequently were less likely (much less) to be considered leaders.”

e) One of the IVs in this study has FIVE levels--that's unusual, but it can happen!  What is this IV and what are its five levels?  Is this IV manipulated or measured? 

f) What is the second IV? How many levels does it have and what are they? Is it manipulated or measured? 

g) The DV is not explicitly stated in this summary. What do you think the DV was, based on this description? 

h) Sketch a well-labeled graph of the results of this study, using the DV of "being nominated by peers as being the leader". I suggest that you put the 5-level IVs  on the x-axis and the other IV as a line color or bar color. 

i) The first example (With Eric and Erica) was a 2x2 factorial design. How would you state the design of this West Point example? 

 

 

Who benefits from putting the phone away?

Should she have put her phone away in the next room? That depends.
Photo: Suwat Sirivutcharungchit/Shutterstock

Students, if you're not familiar with the study tips on The Learning Scientists website, you should be. This page in particular sums up six evidence-based things you should be doing while you study (spoiler alert: The list does not include highlighting!).

The Learning Scientists' latest blog post sums up the results of an experimental study on where your phone should be while you engage in cognitive tasks. It's titled, "Separation from your cellphone boosts your cognitive capacity." Take a look at the description to get an overview of the research design:

They invited students to participate in an experiment where students were randomly assigned into one of three conditions.

• In the "other room" condition, students were asked to leave their belongings (including their cellphones) in the lobby before coming into the room where the experiment would take place.
• In other two conditions, students were asked to take their belongings with them to the experiment room, and were either told to leave the cellphone out of sight, e.g., in their bags or pockets (bag/pocket condition) or place it face down on the desk within sight (desk condition).

Then, participants worked on two cognitive tasks: One working memory task – called Automated Operation Span task (OSpan) – where people are asked to actively process information while holding other information in mind....For the other task – the Raven’s Standard Progressive Matrices (RSPM) – participants had to identify the missing piece in a matrix pattern. This test is used to assess fluid intelligence and your performance depends to a large extent on the available attentional capacities to identify the underlying rule of the pattern matrix.

a) Based on the description, what kind of experiment was this: Concurrent measures? Repeated measures? Posttest-only? or pretest/posttest? 

b) What is the independent variable here? There are two dependent variables in this design. What are they? (Note: You might recognize the OSpan task from Chapter 8; it was used in a correlational study about ability to multitask.)

c) What results would you predict from this study? Take a moment to sketch your prediction in graph form. Then click over to the blog post and scroll to the graphs they've made of the results.Do they match your own prediction? 

You can stop working here if you're studying Chapter 10. But if you're studying Chapter 12, keep reading, because there's more!  The second part of the blog post is headed "Cellphone dependence as moderator". Get ready for a factorial design.

The researchers separated people into two new participant variable (PV) groups: Those who reported feeling dependent on their cellphone throughout the day, and those who did not. They then used this PV in combination with the IV of the original design.

d) Given the description, how would you state this design? (Use the form: __ X __ factorial.) 

Here are the results:

• For people who reported a strong dependence, putting the cellphone in the bag or leaving it in another room made a tremendous difference for their cognitive capacity: They performed much better in these two conditions compared to the one where the phone was on the desk.
• For people who reported a weaker dependence, it made no difference where the phone was. Thus, their performance was not affected by the location of the phone.

e) Sketch a bar graph or line graph of the  factorial results described above. You can do it either way, but I'd recommend putting the "cell phone condition" IV on the x-axis. 

f) Do you see an interaction in the results? (You should, because the term "moderator" is a sign of an interaction)

g) Let's return to the headline, "Separation from your cellphone boosts your cognitive capacity." Does the headline seem appropriate for this study? Why or why not? 

Good news: The published article on which this blog post was based is open source! You can view it here. 

Subtle social class cues on your resume

 

What resume cues did the researchers use to manipulate gender? Social class? Photo: Andrey Popov/Shutterstock

The Harvard Business Review described a study that tested biases among people who read internship applications. Not just any internships, either: These are internships at prestigious law firms. The HBR reports that these internships..

... open doors to even more lucrative employment in the private sector as well as prestigious judiciary and government roles. For these reasons, employment in top law firms has been called the legal profession’s 1%.

Here's a preview of the design of the study: 

Now imagine four applicants, all of whom attend the same, selective second-tier law school. They all have phenomenal grade point averages, are on law review, and have identical, highly relevant work experiences. The only differences are whether they are male or female and if their extracurricular activities suggest they come from a higher-class or lower-class background. Who gets invited to interview?

a) Given this brief description, identify the design of the study. What are the independent variables? What are the levels of each IV?  What is the dependent variable? Do you think the IVs were manipulated as independent groups or within groups? 

Now, here's some more detail: 

We uncovered this through a field experiment with the country’s largest law firms. Specifically, we used a technique — known as the resume audit method — that is widely seen as the gold standard for measuring employment discrimination. This method involves randomly assigning different items to the resumes and sending applications to real employers to see how they affect the probability of being called back for a job interview. All in all, we sent fictitious resumes to 316 offices of 147 top law firms in 14 cities, from candidates who were supposedly trying to land a summer internship position. All applicants were in the top 1% of their class and were on law review, but came from second-tier law schools. 

We signaled gender by varying the applicant’s first name (James or Julia).

...to capture the economic component of class, our lower-class applicants received an award for student-athletes on financial aid. To incorporate its educational competent, they listed being a peer tutor for fellow first-generation college students. By contrast, our higher class candidate pursued traditionally upper-class hobbies and sports, such sailing, polo, and classical music, while the lower-class candidate participated in activities with lower financial barriers to entry (e.g., pick-up soccer, track and field team) and those distinctly rejected by higher-class individuals (e.g., country music). But crucially, all educational, academic, and work-related achievements were identical between our four fictitious candidates.

Now here are the results: 

Even though all educational and work-related histories were the same, employers overwhelmingly favored the higher-class man. He had a callback rate more than four times of other applicants and received more invitations to interview than all other applicants in our study combined. But most strikingly, he did significantly better than the higher-class woman, whose resume was identical to his, other than the first name.

Take a look at the bar graph of the results on the HBR story site here. 

Additional questions

b) How were the IVs and DVs operationalized?

c) Interrogate the construct validity of the operationalization of social class. Do you think this manipulation was accurate and valid?

d) Using the three criteria for causation, interrogate the internal validity of the study. What controls do you notice that reduce design confounds. Is there any concern about selection effects?   Why or why not?

e) What interaction in the results shows a moderator?

f) Using the data provided in the story, do you think that the study showed a main effect for gender? A main effect for social class? How about an interaction? Describe each of these effects. 

g) Finally, the researchers dug deeper by conducting a follow-up study to investigate why the law firms were biased. Read this section of the article and decide, What mediating theory does the follow-up study propose?

Why did the higher-class man do so much better than the higher-class woman? To further explore this issue, we conducted a follow-up experiment with a sample of 210 practicing attorneys from around the country. We asked each attorney to evaluate one of the same resumes we used in our field experiment and...rate their candidate on factors proven to influence how favorably people view job candidates but that vary between men and women. These included perceptions of the candidate’s competence, likability, fit with an organization’s culture and clientele, and career commitment.

Just like the employers in our audit study, the attorneys we surveyed favored interviewing the higher-class man above all applicants, including the higher-class woman. This time, though, we were able to understand why. Attorneys viewed higher-class candidates of either gender as being better fits with the culture and clientele of large law firms; lower-class candidates were seen as misfits and rejected. In fact, some attorneys even steered the lower-class candidates to less prestigious and lucrative sectors of legal practice, such as government and nonprofit roles, positions that tend to be more socioeconomically diverse than jobs at top law firms.

Thanks to Dr. Kathleen Lewis of Point Park University for sending this example my way and for writing the questions! 

Motivating students to work harder through teacher support

When teachers wrote encouraging notes (rather than neutral notes) on returned papers, students were motivated to revise them. Photo: Fuse/Getty Images

How do we get students to work harder?  Standards in schools may be rising, which means that students are more likely to encounter failures along the road to mastery of a subject. What motivates them to keep getting up, after academically falling down?

This piece in the Atlantic describes the impact of different motivational interventions on students from different backgrounds.  In the study, students wrote an essay for their teachers, and the teachers graded their essays like they normally would, adding comments to the essay about what the students need to revise.  But the researchers intervened before the essays got handed back. This passage from the journalist's story describes a 2x2 factorial design. See if you can identify the IV, PV, and DV in this example. 

... the researchers randomly attached one of two sticky notes to each essay. None of the students were aware that they were part of a study and thought their teachers had written the notes. Half of them received a bland message saying, "I'm giving you these comments so that you'll have feedback on your paper." The other half received a note saying, “I’m giving you these comments because I have very high expectations and I know you can reach them”—a comment intended to signal teachers' investment in their students' success.

Then teachers offered the students an opportunity to revise their essays.

The results were striking. Among white students, 87 percent of those who received the encouraging teacher message turned in new essays, compared to 62 percent of those who got the bland note. Among African American students, the effect was even greater, with 72 percent in the encouraged group doing the revision, compared to only 17 percent of those randomly chosen to get the bland message. And the revised essays received higher scores from both the students' teachers and outside graders hired for the study.

a) List this study's IV, PV, and DV.
b) For each IV and PV, indicate: Is it within groups or independent groups? Is it manipulated or measured?
c) Using the results reported in the quoted passage, create a graph of the results.
d) Using marginal means or using your visual inspection of the graph you made in (c), estimate and describe this study's main effects and interactions.

e) Challenge question: Like most experimental studies, this one used additional dependent variables other than the primary one described above. In the study described, the researchers also measured students' trust in their teachers. What results do you think they may have found? Sketch a graph showing the results you'd hypothesize for this DV. 

Going on an interview? Avoid the "vocal fry..."

You might not know the name "vocal fry," but if you've been on a college campus  lately, you've definitely heard it: Vocal fry is a creaky, low sound that up to 2/3 of college women put at the end of sentences. A friend once called it the "sticky voice." 

The Atlantic just covered a research study on how people perceive others who use vocal fry in a job interview.  The story includes little audio clips of men and women speaking in regular voices and with vocal-fried voices (I recommend listening to them!)

Here's how the study was described: 

Researchers at the University of Miami and Duke University asked seven male and seven female young people to say the phrase “Thank you for considering me for this opportunity” in both a normal tone and in vocal fry. Then, 800 men and women of a variety of ages were invited through an online survey to listen to the samples and [to rate each] speaker (normal or fry) [on how] educated, competent, trustworthy, attractive, and appealing [they were] as a job candidate.

a) This is a factorial design. What are the independent variables?  For each independent variable, indicate if it is independent groups or within groups. Then state the design as precisely as you can.

b) Note that this study has more than one dependent variable. Name at least three dependent variables,

Here's a summary of the results:

For each trait, the listeners preferred the normal voice to the fry voice for both the male and female speakers. They were less likely to say they'd want to hire the person with the fry voice, [and] they found them to be less trustworthy. When making hiring judgments, people preferred a normal voice 86 percent of the time for female speakers and 83 percent of the time for male speakers. Women using fry were viewed more negatively than men doing so, and the negative perceptions were stronger when the listener was also a woman. 

c) Graph the results for the dependent variable, "rated negativity," described in the last sentence above.

d) In the graph that you drew, is there an interaction? (There'd better be!)

e) What about main effects? Based on your inspection of the graph you made, decide whether there will be main effects for the two independent variables in this study. 

 f) given the results and method of the study, did the study design and results allow the journalist to say that "Vocal fry may hurt women's job prospects?"  How might you improve this headline to make it more faithful to the actual study? 

 

 

Eating more from big bowls....

It's not every day you get to see a three-way interaction described in the news! But here's one, a study covered by the Huffington Post.

The headline was as follows:

Extroverted Children More Likely To Be Swayed By Environmental Cues: Study

Read this study description and identify the three independent variables in the design. Identify the dependent variable, too.

The study...included 18 kids ages 6 to 10, whose levels of introversion and extroversion were rated on a scale by teachers and counselors.

On one day, the kids were served breakfast by adults; they were given a large [or a small] bowl, and then told the adults how much cereal and milk they wanted to have. On another day, the kids served themselves breakfast [after being given a large or small bowl]. The amount of food served -- whether by the adults, or the kids themselves -- was secretly weighed by scales hidden in the tables.

a) What are the three IV's and the DV in this study?

b) Try graphing the result. Here's your hint: you'll have two graphs: one for kids being served by adults, and the other for kids serving themselves. Each graph will have the other two IV's on it. Use this description of results to estimate the pattern:

Extroverted kids served themselves 33.1 percent more breakfast when they had the larger bowl, compared with introverted kids, who only served themselves 5.6 percent more when they had a larger bowl.

When the adults were serving for them, both extroverted and introverted kids asked for more than 50 percent more when they had a bigger bowl.

 

 

 

Suggested answers

a) The three IVs are:

Personality type: Extroversion/introversion (this is actually a participant variable, not a true IV). This was between subjects.
Bowl size: Large or small, between subjects.

Server:  Self or adult, between subjects.

b) You can check your graph against the 2x2x2 bar graph in the original paper. Click here, and scroll to Figure 1.

Bacon makes (almost) everything taste better...here's the evidence

Wired magazine presents a set of datagraphics depicting their ingenious analysis of the corpus of recipes on the website foodnetwork.com. I bet you'll have fun looking through the graphics.   They correlated the ingredients list in each recipe with the average number of five-star rating points the recipe has received from readers.

For example, one graph shows that the more ingredients a recipe has, the higher it gets rated (Do more ingredients taste better? Or do people just have to justify all that prep time?)

The lead graphic shows the five-star ratings of recipes that either contain bacon or not. You can check it out here. Across almost all categories (sandwiches, asparagus recipes, kale recipes, and spinach salad recipes), those containing bacon are rated higher than those that aren't.

But there are two exceptions to this pattern: bacon as an ingredient is associated with lower ratings of pasta and dessert recipes.  Though bacon generally is associated with higher recipe ratings, this doesn't apply to pasta and dessert, apparently. (Just say no to bacon ice cream.)

In these data, recipe, not person, is the unit of analysis. But your research methods lessons still come in handy. For example:

a) Can we conclude from Wired's data that adding bacon will cause a sandwich to be rated more positively? Why or why not?

b) Do you see what I see--that there's an interaction between the bacon variable (bacon or no bacon) and the type of recipe variable? Whether bacon recipes get higher ratings depends on what kind of recipe it is.

c) How might you extend these correlational data into an experiment? Which variable would you manipulate? Which one would you measure?

 

 

Targets of prejudice manage impressions

How would you respond if you knew that people might stereotype you negatively? That's what one study recently asked. They wondered how people would approach a social situation if they were worried about the stereotypes the other person might have about them.

The studies in question can provide you with some handy practice on factorial designs. 

Scientific American wrote about the study by Rebecca Neel and her fellow researchers. They explain that they recruited 75 college students, some of whom were overweight and some of whom were not.

In the first study, the participants answered questions about obese people and other stereotyped groups.  Scientific American continues:

The students also were asked to envision meeting someone new and then to choose how they'd make a good impression from options such as arriving on time, wearing clean clothes, smiling and looking relaxed. Some students answered the group [stereotype] questions first so they'd have group-related stereotypes in mind when they got to the first-impressions' questions. Others completed the study the other way around.

The results showed that thinking about stereotyping changed people's behavior. Overweight students who'd first answered questions about  obese people were more likely than other participants to rank "wearing clean clothes" as a very important way to make a good first impression. Normal-weight students and overweight students who hadn't been primed to think of stereotypes were more likely to prioritize arriving on time.

 a) For the study above, the dependent variable could be described as the degree to which participants prioritized wearing clean clothes (over arriving on time). In addition to this DV, there is an independent variable and a participant variable in this factorial design. What are they?

b) How many levels are there in each of the IV's/PV's?  Are these variables independent groups or within-groups?  What kind of design is it (use this format: ___ x ___)?

c) Sketch a graph of what the results are, as they are described in the paragraph above. What main effects and interactions are present in the graph that you sketched? 

 

 

The second study reported is also a factorial design. It is more difficult to find the IV's in this description, but give it a shot:

 

In a second study, researchers repeated the test with overweight men and black men. When prompted to think of stereotypes, overweight men ranked wearing clean clothes as the most important step toward making a good first impression. Black men, who are often stereotyped as violent and anti-social, prioritized smiling.

d) What are the IV's/PV's in this study? What is the dependent variable? (There seem to be two key dependent variables in this version of the study.)

e) Make two graphs of the 2x2 design above. One graph is for the first DV, and another graph should have the same IV's/PV's as the first graph, but will have the second DV on its y-axis.

 

 

A blog about interactions

You don't see people blogging about interaction effects, at least not every day. I just ran into this 2011 Ben Goldacre column about interactions, which he describes in my favorite way, as a "difference in differences." Take a look here. Instructors, this might be a handy example for teaching Chapter 11.