Everyday Research Methods

The most fulfilling jobs in America

The story reported that members of the clergy are most likely to score high on all four questions.
Photo: Diego Cervo/Shutterstock

Before you read on, make a prediction: Which U.S. jobs would you predict would be the most fulfilling?  In this story from the Washington Post column,"Department of Data" you can explore some fascinating data on U.S. job satisfaction.

In a fun twist, the article also talks about how the construct of "fulfilling work" is measured. (It's not everyday that we get to celebrate construct validity: hooray!)

The data come from the CEV--the Civic Engagement and Volunteering Supplement, which is part of the Census Bureau’s Current Population Survey.

Here's how the journalist introduces the data:

In 2021 and in 2023, the researchers behind the CEV asked whether you agree or disagree with these four statements:

• I am proud to be working for my employer.
• My main satisfaction in life comes from work.
• My workplace contributes to the community.
• I contribute to the community through my work.

The two years of surveys give us enough responses to start doing some serious analysis — or, in the grand tradition of this column, some less-than-serious analysis.

The Post column presents data (click here and scroll down) showing that agreement with these four items increases with age and with education. 

In general, the figures and charts of this story are fun to explore--but they're under copyright, so I can't paste them here. Here are some of the journalist's summary points. You'll see that which profession comes out on top depends on the question. 

The workers most likely to say they are proud to be working for their employer and that they gain satisfaction from work are — surprise! — the self-employed.The self-employed who are incorporated — a group that often includes small-business owners — are almost twice as likely as private-sector, for-profit workers to strongly profess pride in their employer.

Government and nonprofit workers fall somewhere in the middle on those [pride and satisfaction] questions. But they rank at the very top on “My workplace contributes to the community” and “I contribute to the community through my work.” Local government workers, who include teachers, take the top spot for strong agreement on both, followed by nonprofit workers. Private-sector, for-profit workers again lag behind.

The jobs that do worse on these measures [of contributing to the community] tend to be in manufacturing or other blue-collar production and extraction jobs, or at the lower-paid end of the service sector. Folks in food services (e.g., bartenders and food prep), janitorial roles and landscaping, and personal services (e.g., barbershops, laundry and hotels) all struggle to find greater meaning in their work. Though some in better-paid service jobs also struggle by some measures — think sales, engineering or software development.

The story indicates that members of the clergy report the most positive answers across all four of the satisfaction items.

a) Do the four items used in this story (included above in this blog post) have face validity as a measure of job satisfaction? Why or why not?

b) Click to the story and scroll to find the line graph that shows agreement with the four items over the lifespan. What do you notice? What do you wonder?

c) One of the concepts in Chapter 5 is internal reliability, or internal consistency, which is relevant when researchers use multiple items to measure the same thing. In this story, the journalist kept the four satisfaction items separate. But based on what you see in the two line graphs, do you think the four items would have internal reliability? That is, would they likely correlate with each other and have a strong Cronbach's alpha?

d) Students often mistrust self-report questions like these. However, in my opinion, the demographic and job category data provide criterion validity evidence that these four questions are construct valid--that is, that people are not just lying or making things up. Specifically, I think the data shows known groups criterion validity for these four items. Can you explain why? Review what known groups criterion validity is, and apply that definition to this new example. 

e) A friendly reminder: Frequency claims (such as "local government workers are most likely to think their workplace contributes to their community" need not just construct validity, but also external validity. How would you ask about that for this data set? Do you think this data set is externally valid--why or why not? 

For even more fun, the column also includes some charts from a different set of data, showing the happiest, most meaningful, and most stressful daily activities (including spiritual pursuits, volunteering, exercising, and working), as well as charts indicating the happiest places (such as places of worship, one's own home, others' homes, and so 0n). Data-loving psychologists (and research methods students) will enjoy exploring these figures. 

 

 

Psychological research: How to talk with family you disagree with

The journalist provides a mixture of empirical evidence and authority to support advice for how to talk to family you disagree with.
  Photo: PeopleImages.com - Yuri A/Shutterstock

During the holidays, we may find ourselves interacting with family members who don't agree with us--especially on politics.

Here's a story from NPR news with advice on how to navigate these situations. It provides practice in identifying variables, classifying claims, and evaluating whether a claim is supported by empirical evidence or not. 

Political conversations with family members might fill us with dread, especially in light of this claim reported in the story:

Polling data from SNF Agora Institute at Johns Hopkins University shows that almost half of the U.S. electorate thinks members of the opposing political party are "downright evil." In a 2022 Pew Research Center study, growing numbers of Americans said members of the other party are dishonest, immoral and closed-minded.

a) In the quoted text above, what type of claim is being made--frequency, association, or cause?

b) In the quoted text above, what is/are the variable(s) in the claim?

c) In the quoted text above, what kind of evidence does the journalist provide--experience, authority, or empirical research? (You can review these terms in Chapter 2.)

 

Here's another excerpt from the story:

Neuroscience has shown that when two people agree, their brain activity is more synchronized than when they disagree.

Yale School of Medicine neuroscientist Joy Hirsch led a 2021 study that found that people's brains lit up in similar ways when they agreed. She thinks this means those people share more information and that the two people are more in consensus. Versus when people disagree: Their brains act like a cacophony instead of a harmonious duet.

d) In the text from the quote above ("Neuroscience has shown..."), what type of claim is being made--frequency, association, or cause?

e) In the text above, what are the variables in the claim?

f) In the quoted text above, what kind of evidence does the journalist provide--experience, authority, or empirical research? 

 

Here's another section: 

Tool 1: Focus on your breathing 

When we face potential conflict, our body's automatic response may not help us regulate our emotions. [...] "We get angry because we hear someone say something that we feel is just so wrong and we have to tell them that it's wrong," says Ken Barish, a psychologist at Weill Cornell Medical College who wrote the forthcoming book Bridging Our Political Divide. "So we have this mixture of anxiety and anger, and over time that becomes resentment. And contempt — and contempt is a very destructive interpersonal process."

But Briscoe-Smith says we can begin to move through these emotions by slowing down and refocusing on breathing. This can help combat the body's automatic response to conflict, helping us think more clearly and move to the next step.

g) Advice is usually a causal claim, because it suggests that if you do ____, it will cause ___.   In the text above, what are the variables in the causal claim? 

h) In the quoted text above, what kind of evidence does the journalist provide--experience, authority, or empirical research? 

 

And here's one more element you can analyze:

Tool 3: Empathy 

Humanize the other person you're talking to by asking about their lives, their families, their hobbies — not just their opinion on a single topic. That can help create more common ground outside of the conversation at hand.

i) As before, advice is usually a causal claim, because it suggests that if you do ____, it will cause ___.   In the text above, what are the variables in the causal claim? 

j) In the quoted text above, what kind of evidence does the journalist provide--experience, authority, or empirical research? 

 

Reflection

k) Looking back on questions h) and j), What do you think about this journalist's use of authority over empirical evidence? And, what kind of empirical evidence might you like to see to support their advice? 

 

Is your dog judging you?

Prostock-studio/Shutterstock

Dog research has gotten much more popular in the last 10 years, as researchers study what cognitive and social abilities dogs have. Here's a study that tested whether dogs could tell the difference between a competent and an incompetent human. Here's how the study was summarized by by a dog blogger. 

So, the gist of the experiment is that randomly-selected pet dogs watched two people manipulate a box. One person, the “competent” person, opened it easily. The other person, the “incompetent” person, did not.

Then, the researchers put a treat in the boxes.

The male dogs in the study asked the competent and incompetent people for the treats pretty equally [I imagine it's about 50-50].

The female dogs?

A whopping 83 percent of the time, the females asked the person they witnessed to be competent.

a) In the passage above, identify the manipulated IV and its levels.

b) How was this IV manipulated--as independent groups or within groups?

c) What was the DV in this study? 

d) Dog's gender is another variable in this study--it's a PV, or participant variable. Is this an independent groups or within groups variable?

Because this study has an IV with two levels and a PV with two levels, it is a 2x2 factorial design.

e) Fill in this 2x2 table with the data mentioned above:

	Male dog	Female dog
Competent
Incompetent

f) Compute the marginal means for the IV and the PV, and estimate any main effects.
Describe main effects with this format:
"There is/is not a main effect for _[IV name]_ such that _[IV level]_ is higher on [DV] than _[IV level]_" 

"There is/is not a main effect for _[PV name]_ such that _[PV level]_ is higher on [DV] than _[PV level]_" 

g) Sketch a small line or bar graph of the results. Remember to put the DV on the y axis. 

h) There is an interaction in these data. See if you can describe it!

 

Instructors: This would be a good example with which to have students compare popular press coverage to the original article. The empirical article is published in an Elsevier journal, which makes it harder to obtain for most people, but ILL can help. The blogger and Yahoo! newsboth make errors in reporting the study--for example, the dogs were not randomly selected, and there were about 60 dogs in the study, not 30 (they left out one of the conditions). 

Do "hands free" cell phone laws make drivers less distracted?

To measure distracted driving, the study recorded how often people's cell phones moved while they were driving.
Photo: Ekaterina Pokrovsky/Shutterstock

In the summer of 2023, 28 U.S. states, including Alabama, started banning handheld cell phones while driving. This news story from the Birmingham News (AL) shows data from an interrupted time series design that tracked people's distracted phone use through automatically collected phone movement data. They plot out the results of four states (OH, AL, MI, and MO) and report values for 3 months before these hands-free laws began and for 10 months afterwards. Take a look at the figure here: https://tinyurl.com/23tt7xmq

a) Why might this design be called an "interrupted time series"? (Chapter 13 will have definitions to help you.)

b) Even though there are four states depicted on the main figure, this is probably not a "non-equivalent control group interrupted time series design." That's because there are no examples of states that did not implement a hands-free ban that summer. When we are trying to decide whether the hands-free cell phone ban reduced distracted driving, it would be helpful to have a set of states with no ban. Why? (You might use the term "history threat" in your answer).

c) Based on the figure, what do you think of the effect size in each state?

d) This study's interrupted time series has only about three months of data from before the time when the bans went into effect. Why might we want more than that as a baseline?

You might wish to look through an official report with bar-graph versions of the data, which can be viewed here: https://m.cmtelematics.com/hubfs/A%20Road%20Map%20for%20Safer%20Roads.pdf 

I'm grateful to Stephen Chew, who shared this story with me.

Body Scan Meditation for Food Cravings

 

A body-scan meditation might prevent mindless trips to the fridge.
Photo: F8 studio/Shutterstock

Do you consider yourself an emotional eater? If you'd like to reduce your susceptibility to emotional eating, here's something to try. It's also a good chance to practice one of the simple experiments. The research was summarized by a journalist for PsyPost.  

Recently, scientists have begun investigating the effects mindfulness-based interventions may have on emotional eating.[...]  One such practice is body scan meditation, which involves slowly focusing attention on different parts of the body to promote relaxation and awareness of physical sensations.

The study participants were 81 adults classified as emotional eaters based on the results of the Dutch Eating Behavior Questionnaire. Seventy-four percent of them were female. Participants’ average age was 27, and 45% identified as European or North American.

The researchers randomly divided participants into two groups. One group underwent a 16-minute body scan meditation, while the other group listened to a recording of an equally long description of the human musculoskeletal system. Neither the participants nor the researcher working with them was informed of which condition was experimental and which was the control.

Before the intervention, participants completed an interview in which researchers asked them to recall a recent episode of emotional eating. This was intended to induce a negative emotional state in participants.

Participants also completed several assessments, including "negative emotion (using the Positive and Negative Affect Scale)" and "food cravings intensity (using the Desire subscale of the Food Cravings Questionnaire – state version)", as well as other variables. 

Results showed that participants who completed the body scan meditation session experienced a greater reduction in negative affect and cravings from before to after the treatment compared to the group subjected to the listening task. 

a) Reread the description above and identify the study’s variables.

Variable name	What were this variable’s levels?	Was this manipulated or measured?	Was this variable an IV? Or a DV?	For the IV: was it independent groups or within groups?

b) What kind of experiment was this: Posttest only? Pretest/posttest? Repeated measures? Or concurrent measures?

c) Sketch a graph of the results. Don't forget to include pretest and posttest data. 

d) Here is a list of internal validity threats in Chapter 11. Check the definition of each term in Table 11.1.  Which of these threats is likely ruled out by the study's design, and why? 

Selection effect
Order effect
Design confound
Maturation
History
Instrumentation
Testing
Attrition
Regression to the mean
Placebo effects
Demand characteristics
Observer effects

 

Cooling down with slushies

Those tiny pieces of ice help cool you down. Photo: DelightPhotos/Shutterstock

Summer’s almost over, but you should save this idea for next time you’re feeling too hot. It turns out that the best way to cool yourself down is by eating small pieces of ice—like those found in your favorite slushy.

You can see a short video about this study on the Weather Channel, and you can read a bit more about it here, at the Washington Post.

The Weather Channel’s video says:

Here’s what multiple studies found: Drinking a slushy can drop your internal body temperature by 1 degree in 20 minutes

and

The slushy method has been tested and works for firefighters, cyclists, and triathletes.

The video explains that when you ingest tiny pieces of ice, your body sends energy to your belly to melt them. This draws energy (and heat) away from the rest of your body, cooling you off.

a) Before reading on, how might you conduct an experimental study to test the slushy method of cooling? What would your IV need to be? What would your DV be?

Now read these lines from the Washington Post journalist to see how close these studies came to your answer in a).

[A] study on Australian firefighters found that drinking a big slushy was just as effective at lowering core body temperature as sitting in a tub of cool water up to your bellybutton. 

b) According to this summary, there were at least two groups in this Australian firefighters study. The journalist does not mention a control group, but do you think they would need one? Why or why not?

The Post article included a subsection asking, How big does the slushy need to be?

The amount of slushy you need to cool down depends on your weight.

Both Maley’s 2020 study and the earlier 2014 study used a similar formula that translates to about one fluid ounce per eight pounds of body mass. For the firefighters in the 2014 study, who weighed around 185 pounds, that would be 23 ounces, which is roughly one large Slurpee from 7-Eleven.

c) Even though Maley’s two studies followed a 1 fluid ounce per 8 pounds rule, that doesn’t necessarily mean the 1-to-8 ratio is ideal. How could you design a study that would test the relative efficacy of different ratios of slushy-to-body weight? Specifically, what would the IV need to be to test this question? What levels might you test? What would the DV be?

The Post article also asked about timing:

It doesn’t matter how fast you drink it, as long as you don’t let it melt. The firefighters in the 2014 study slurped their slushies in 15 minutes after exercising, while in the 2020 study, they sipped them over the course of half an hour before exercising. The results were the same: Their internal body temperatures dropped by about 1 degree Fahrenheit.

d) Some researchers would prefer to test the timing question in a single experiment. How could you design a study that would test the relative efficacy of different timings of slushy consumption on body temperature? What would your IV and its levels be? What would the DV be? 

Clowns help kids get out of the hospital faster

 

We can apply the four big validities to this study on medical clowns. Photo: Halfpoint/Shutterstock

Here’s a study about a behavioral intervention that apparently helps hospitalized children recover faster. The study tested the effect of medical clowns—a specially trained person who can distract hospitalized kids, guide them through relaxation exercises, and help them feel more relaxed during a stay in the hospital.

 

Some researchers experimentally tested the use of medical clowns. Their recently-published empirical study was covered by the journalistic source, Study Finds.

These aren’t your average birthday party clowns. Medical clowns are specially trained professionals who use humor, music, and playfulness to help patients cope with the stress of being in the hospital. […]

So, how did the researchers figure out that clowns could help kids with pneumonia? They looked at 51 children between the ages of two and 18 who were in the hospital with pneumonia. The kids were split into two groups. One group received standard medical care, while the other got the same care plus two 15-minute visits from a medical clown each day for the first two days of their hospital stay.

The clowns, who were part of a group called The Dream Doctors Project, used various techniques to help the kids relax. They played music, sang songs, and even used something called “guided imagination”

…The [results showed that] kids who got to hang out with the clowns got better faster! On average, they were able to leave the hospital after about 43.5 hours, compared to 70 hours for the kids who didn’t see the clowns. That’s more than a full day difference!

a) Reread the description above and identify the study’s variables.

Variable name	What were this variable’s levels?	Was this manipulated or measured?	Was this variable an IV? Or a DV?	For the IV: was it independent groups or within groups?

b) What kind of experiment was this: Posttest only? Pretest/posttest? Repeated measures? Or concurrent measures?

c) Sketch a bar or line graph of the study’s results. Remember to put the dependent variable on the y-axis.

Now let’s apply the four big validities to this design.

d) Internal validity: The journalist says that “The kids were split into two groups”. But it’s not specified that the children were randomly assigned to the two groups. Why would random assignment be important here?

Imagine that instead of random assignment, the researchers had asked parents which group they’d like their hospitalized child to be in (the clown group or the standard care group). Why would that be an internal validity problem?

e) Statistical validity: After describing the study’s results, the journalist reports: “That’s more than a full day difference!” Which statistical validity concept is the journalist addressing here?

f) External validity: What questions might you ask to determine this study’s ability to generalize?

g) Construct validity: The main measured variable in this study was how many hours children were kept in the hospital. Do you foresee any potential issues with the measurement of this variable?

Suggested answers:

a) Reread the description above and identify the study’s variables.

Variable name	What were this variable’s levels?	Was this manipulated or measured?	Was this variable an IV? Or a DV?	For the IV: was it independent groups or within groups?
Clown condition	Medical clown or Standard care only	Manipulated	IV	Independent groups
Hours in the hospital	0 up to many (100 perhaps?)	Measured	DV	n/a

B) This was a posttest only design because the IV was manipulated as independent groups and the participants were measured on the DV only once.

Now let’s apply the four big validities to this design.

c) Random assignment would be important here to ensure that kids in the two groups were approximately equal in how sick they were, how old they were, how much they like clowns, how good their doctors were—in short, equal on almost any relevant individual differences that might provide an alternative explanation for the “clown” group getting out of the hospital sooner.

If, instead of random assignment, the researchers asked parents which group they’d like their hospitalized child to be in (the clown group or the standard care group), that would  introduce an internal validity problem—a confound. Specifically, it would introduce a selection effect in which there were systematically different types of participants in the two groups. The researchers couldn’t be sure if the medical clown kids had shorter hospitalizations because of the clowns, or because the kids’ parents were more motivated to provide them with extra services and care.

d) The journalist is addressing the effect size of the study—how strong or how large the difference was between the two conditions. Effect size is one of the standard things to ask about when it comes to Statistical Validity (you’d also want to ask about the precision of the estimate [the 95% CI] and about replication—what other studies have found).

e) You might ask how they got this sample—was it a random sample of children from this hospital? (if so, you could at least generalize to that hospital’s population.) You might also ask if the medical clown results would generalize to kids who are hospitalized for other conditions, such as cancer or other illnesses. (This would require conducting additional studies using samples from these populations.)

f) You might disagree, but to me, it seems that “time spent in the hospital” is a very reliably measured variable. It is probably included clearly on each kid’s medical records and is less susceptible to problems such as response biases.

Emotional and behavioral outcomes of bed-sharing with infants

Is this non-bed-sharing baby more or less likely to show emotional or behavioral symptoms? Photo: morrowlight/Shutterstock

Bed sharing, in which parents sleep in the same bed with their infants, is a fairly common practice in North America and the United Kingdom. Parents who bed-share sometimes do so to facilitate breastfeeding, or because they believe it helps babies feel safe or that it will lead to closer emotional relationships. 

However, many people believe that bed-sharing might be bad for children's development. And, the American Association of Pediatrics officially recommends against adults sharing a sleeping surface with infants under the age of six months because of the potential risk of sudden infant death.  

A large study aimed to see if there were relationships between bed sharing and emotional and behavioral outcomes. The study was summarized by PsyPost here. The study illustrates the use of regression (Chapter 9) and is an example of a null effect (Chapter 11). 

The researchers aimed to clarify whether bed-sharing during infancy has any lasting influence on a child’s development, particularly concerning emotional stability and behavioral tendencies.

The study utilized data from the Millennium Cohort Study, a large, nationally representative longitudinal study from the United Kingdom. This cohort followed over 18,000 infants born in the early 2000s, tracking their development through various stages of childhood. For this particular research, the focus was on children who were bed-sharing at 9 months of age and their subsequent emotional and behavioral development up until they were 11 years old.

a) Given the information above, what can you infer about the external validity of this study--can it generalize to the population of children in the UK?

To assess emotional and behavioral outcomes, the researchers used the Strengths and Difficulties Questionnaire, a widely respected tool for measuring psychological adjustment in children. This questionnaire, completed by parents when the children were 3, 5, 7, and 11 years old, allowed the researchers to track patterns of internalizing symptoms, such as anxiety and depression, and externalizing symptoms, like aggression and hyperactivity, over time.

b) The passage above addresses construct validity. It tells you how emotional and behavioral outcomes were measured, but it doesn't say much about whether this measure is reliable and valid. What might you want to know about this measure to decide if it is construct valid? 

The study also accounted for several other factors that could influence a child’s development, including gender, socio-economic status, night-waking frequency, breastfeeding practices, maternal psychological distress, and parenting beliefs. By considering these variables, the researchers aimed to isolate the specific impact of bed-sharing on child development.

Interestingly, while bed-sharing at 9 months was more common among children in the groups with elevated symptoms [of emotional and behavioral symptoms], the researchers found no direct evidence linking bed-sharing to an increased risk of these symptoms once other factors were taken into account. In other words, after considering variables like parenting beliefs and maternal distress, bed-sharing itself did not predict whether a child would belong to one of the higher-risk groups.

c) Read the passage above and restate the simple bivariate relationship between bed-sharing and emotional/behavioral symptoms. 

d) Now consider the regression analysis. What is the criterion variable? 

e) What are the predictor variables? (Hint--there are seven of them)

f) Given the results described above, what might the beta have been for the bed-sharing predictor variable? 

To summarize the results so far: There was a bivariate relationship between bed-sharing and symptoms. However, it seems that this bivariate relationship was attributable to some third variable like parenting beliefs or maternal distress because when these variables were controlled for, the relationship between bed-sharing and emotional/behavioral symptoms went away. 

g) In Chapter 11's section on null effects, you learn that we can be more confident that null effects are real in the population when the study has more power and precision. What do you think--to what extent does this study help support that there is no relationship between bed-sharing and emotional/behavioral symptoms? 

 

Suggested Answers to Selected Questions

b) we might want to know why the Strengths and Difficulties Questionnaire is described as "well-respected." Do parent's reports on this scale also go with other evidence of children's difficulties or psychologists' ratings?

c) The bivariate relationship is that kids who bed-shared at age 9 months had more emotional/behavioral symptoms at age 10-11.

d) The criterion variable was emotional/behavioral symptoms

e) The predictor variables were bed-sharing, gender, socio-economic status, night-waking frequency, breastfeeding practices, maternal psychological distress, and parenting beliefs.

f) The beta for bed-sharing would have been zero, or close to zero, because the text says, "after considering variables like parenting beliefs and maternal distress, bed-sharing itself did not predict whether a child would belong to one of the higher-risk groups."

g) This study's strength is its large sample of 16,000 children. Such large samples lead to more narrow 95% CIs and more precision in our estimates. If we can't find an effect of bed-sharing with this large of a sample, we probably won't find it. In other words, it would take a lot of exceptional cases to budge our "zero" conclusion. I think we can trust it. 

Can stress turn your hair gray?

Correlational and experimental research seems to converge on the conclusion that stress causes gray hair. Photo: Kunlathida6242/Shutterstock

Can stress turn your hair gray? Here's a story on gray hair research, written by a New York Times science journalist. In our culture, we certainly believe that gray hair can be caused (in some cases at least) by stress. There is clear evidence that gray hair is inherited genetically, apparently stress plays a role, too 

Here's how the journalist introduces the first study:

In past studies, researchers have asked participants to fill out questionnaires about their hair color and stress levels, and then the scientists would see if they could link them.

In one study published in 2016, for instance, scientists surveyed more than 1,100 young Turkish adults and found that the 315 who reported prematurely graying hair had higher stress levels than those who didn’t. 

a) Identify the two variables in this study.

b) Were the variables manipulated or measured?

c) Was this a correlational or experimental study? 

d) Can the results support a claim that stress causes premature gray hair? Why or why not? Apply the three criteria for causation: Covariance, Temporal Precedence, and Internal Validity.

For question d above, you should have responded no; although this study did show a relationship between premature graying and perceived stress, the study doesn't establish temporal precedence since both variables were measured at the same time. And, it's possible some third variable is responsible--for example, maybe older participants have both more stress and more gray hair. (Note that even though genetics is the biggest predictor of premature gray hair, genetics isn't a confound here because the genes for gray hair don't also predict perceived stress.)

To show causation, it helps to have an experiment. Here's an example: 

But a mouse study published in 2020 took the research a step forward. In it, researchers stressed mice in various ways, including by injecting them with a chili-pepper-like chemical that induced a “fight-or-flight” response. This caused them to release the stress hormone norepinephrine, which, in turn, depleted their hair follicles of the stem cells involved with adding pigment to mouse fur. The hair then grew in gray.

But studies on this topic are challenging to perform on people, because researchers can’t ethically induce artificially high stress responses in humans like they can in animals or cells, Dr. Hsu said.

e) What is the independent variable in this study?

f) What were the dependent variables (there were two)? 

g) If you've read Chapter 9, you'll recognize this as a mediator argument: "[The chili-pepper chemical] caused them to release the stress hormone norepinephrine, which, in turn, depleted their hair follicles of the stem cells involved with adding pigment to mouse fur. The hair then grew in gray." Can you draw this mediator argument using boxes for each variable? 

Here's one more study on humans that was better able to manage temporal precedence: 

One small human study published in 2021 still advanced the narrative: Researchers plucked various strands of hair from 14 volunteers who had at least some graying. Several of the strands were fully gray, some were partially gray and some hadn’t grayed at all. The scientists then created high-resolution digital images of the hairs and calculated when each strand went gray using estimates of how quickly hair grows.

They also asked the participants to plot out stressful experiences from the past year on a timeline, and rank them from least to most distressing. The researchers found that when a strand turned gray frequently corresponded with the most stressful moment of that volunteer’s previous year.

h) There seem to be two variables in this study. Can you name them and their levels?

i) Was this a correlational or experimental study? 

Taken together, the three studies here can be seen as an example of "pattern and parsimony." Some of the studies are correlational, others are experimental. Some are on humans and some are on animals. But they all seem to parsimoniously support an argument that stress precedes gray hair. What do you think?

 

Burrito bowl variability

But how big will the next burrito bowl be?
Generic burrito bowl photo: B Young/Shutterstock

An important concept in both statistics and methods is variability. And this analysis conducted by Wells Fargo consultants does a nice job illustrating the concept of variability. They'd been seeing social media complaints about a popular restaurant chain wasn't always providing a full burrito bowl, and decided to look into it. 

Chipotle recently got a wave of social media attention from users claiming the fast-and-casual chain was holding back. Analysts at Wells Fargo sought to put the debate to rest and ordered 75 identical burrito bowl orders across eight locations in New York City.

You need to see the results of the burrito bowl analysis, so click here for a figure from Sherwood.com  (a business news site).

a) Orient yourself to this figure. What do you see on the y-axis? What's on the x-axis? What does one bar represent? 

b) Find the bar on the graph representing the largest/fullest burrito bowl. Find the bar on the graph representing the lightest one. 

c) If you were this fast food chain, what would be the shape of the graph that you'd most like to see? What actions would you  take in your stores and franchises to achieve this shape?

d) The phenomenon depicted in the graph originally reminded me of inter-rater reliability, but it's really not the same. Why not? 

e) How do you think they sampled the restaurants for this analysis? Would it be important to use a random sample, or not? 

Selected answers

a) Here, one bar represents a single burrito bowl; its height corresponds to how heavy it was. 

b) The lightest bowl is all the way on the right.

c) The restaurant chain probably wants to see a rectangular distribution, in which all of the bars are the same height--this would indicate that every burrito bowl weighed the same.

To get closer to this shape, you'd probably need to do some training of your chefs, setting clearer standards for how many ingredients to use in each burrito bowl. 

d) In inter-rater reliability, you're looking to see if two or more raters make consistent observations of a set of people or objects. In other words, you want to make sure that the behavior of a set of targets (say, schoolchildren) is rated the same no matter who rates them. (In this example, Wells Fargo might have wanted to make sure that two analysts who weighed a set of burrito bowls, would come up with exactly the same weights.) In contrast, if you are a restaurant owner, you'd be looking for something we might call "inter-burrito bowl reliability," in which burrito bowls weigh the same no matter who makes them. 

That said, if people have poor inter-rater reliability, you'd see too much variability between them. And if burrito chefs have poor inter-bowl reliability, you'd see a graph that looks a lot like the one we're talking about. 

e) Sure, random sampling of restaurants would support external validity--it would mean that we could generalize from this data--we'd know that the median and the standard deviation in this sample could be generalized to the population restaurants. However, we probably don't need a random sample to support a proof of concept--that is, it's probably a problem that in this sample of 75 burrito bowls, there's a great deal of variability, even if those 75 bowls and restaurants weren't sampled randomly. 

Thanks to Jess Hartnett and Stephen Chew, who shared this one on twitter.