Everyday Research Methods

Running on cannabis

How did the researchers manipulate the IV in this study?
Photo credit: mansong suttakarn/Shutterstock

Cannabis is considered a performance-enhancing drug in the Olympic Games and other competitions, but it's also (stereotypically) associated with being too chill to work out. In the past, psychologists could not easily test the relationship of marijuana and exercise because the drug was illegal. Now, however, since recreational marijuana is legal in several U.S. states, researchers can ethically test its effects on humans. 

One such study investigated the effect of cannabis on how runners experience their workouts. Here's a summary of the research by the online source, ScienceAlert. 

In a recent experiment, 42 healthy adult runners from Colorado who identified as regular cannabis users ran on a treadmill at a moderate pace for 30 minutes. Before, during, and after the run, scientists monitored their physical and mental state. On another occasion, participants were given the choice to use either THC or CBD products before a similar 30-minute run.

THC, or tetrahydrocannabinol, is the compound in cannabis largely responsible for the plant's psychoactive effects. Whereas CBD, or cannabidiol, is a compound in the plant that offers muscle-relaxing, anti-inflammatory effects without producing the trippy mind-altering effects traditionally associated with marijuana.

"The bottom-line finding is that cannabis before exercise seems to increase positive mood and enjoyment during exercise, whether you use THC or CBD. But THC products specifically may make exercise feel more effortful," says psychologist Laurel Gibson from UCB.

[...]Compared to running sober, running high was more enjoyable for participants, even if the inclusion of THC did make the exercise feel slightly harder.

a) Here's a table on which you can classify the variables in this study. Here's a hint: there were two conceptual dependent variables. 

Name of the variable (Conceptual level)	What are the variable's possible levels?	Is this manipulated or measured?	Is this an IV or DV?	For IVs: Is it manipulated as independent groups or within-groups?

 

b) This was an experiment. Why? (Use the term, "manipulated" in your response). 

c) What type of experiment is this? Posttest only? Pretest/posttest? Repeated measures? Concurrent measures?

d) Sketch a graph of the "enjoyable" result that is described in the journalist's coverage above. 

Now let's consider the internal validity of the study. Read this detail: 

Because of ethical considerations, the study was not double-blinded or randomized, and dosage among participants wasn't regulated.

e) The study was "not double-blinded." What are the names and definitions of the internal validity threats (see Table 11.1 in Chapter 11) that would be impacted by this decision? 

The journalist says that the study was "not randomized." What do you think they are criticizing? Is this critique relevant for a within-groups design? Why or why not? 

And, if dosage among participants wasn't regulated, is that a serious problem for the design, or not? Think through how that might have affected this design. 

Here's another detail from the journalist's coverage: 

Furthermore, the group of participants was limited to include regular cannabis users who ran a lot, which means enrollment could very well be biased towards those with positive cannabis experiences while exercising.

f) Which of the four big validities is this comment addressing? 

Really? Prenatal cannabis exposure and early language development

Photo: Viacheslav Lopatin/Shutterstock

A story in PsyPost was headlined with, Prenatal cannabinoid exposure appears to have a strange impact on early language development. 

You might have noticed that the verb phrase, "has a strange impact on" is a causal one. We are being asked to believe that prenatal exposure to cannabis causes something in children's early language development.  I hope your causality suspicions are activated here, since it's almost certain that both cannabis exposure and language development are measured variables. This was probably a correlational study. 

Let's read on to find out what the mysterious "strange impact" is.

The journalist reported on a study in the open-source journal, Frontiers in Pediatrics. The PsyPost journalist drops the "strange impact" in the first paragraph:

...infants exposed to cannabis in the womb tend to exhibit improved language development by their first birthday. This surprising finding sheds light on the impacts of prenatal cannabis exposure on one aspect of early childhood neurodevelopment. However, experts advise that women avoid cannabis use during pregnancy due to known risks and unknown long-term effects.

Given past research, it's a bit surprising to find a positive relationship between cannabis use and child development, so that was the "strange impact" PsyPost mentions in their headline.

Here's how the journalist summarized the study's methods:

The researchers recruited 207 pregnant individuals and their 12-month-old infants who were part of The Safe Passage Study. This was a large prospective study that followed participants from 2007 to 2015.

Talavera-Barber and colleagues assessed the infants according to the Mullen Scale of Early Learning, a tool that assesses cognitive development in children from 2 days old to 68 months. 

[...]Prenatal cannabis exposure was collected from the mothers through self-report. Participants were categorized based on whether the exposure to cannabis occurred early (first trimester only; 51 participants) or late (second or third trimester; 18 participants), and they were randomly matched with unexposed participants for comparison (138 participants).

Don't be misled by the term "randomly matched with unexposed participants." A peek at the full empirical article shows that these were simply people in the sample who did not use cannabis. They were randomly selected from a larger set of participants in the study--but that doesn't mean they were "randomly matched." (In fact, I'm not sure I know what the journalist meant by "randomly matched" here.)

Questions

a) Based on what you've read so far above, complete this table of the variables in the study:

Variable name (stated at the conceptual level)	How was this variable operationalized?	Is this a measured or a manipulated variable?

Some of the results were summarized this way by PsyPost:

Unexpectedly, infants who were exposed to cannabis later in pregnancy scored higher in both expressive and receptive language areas than those who were not exposed.

Additionally, infants exposed early in pregnancy showed better gross motor skills, although there was no difference in fine motor skills and visual reception skills.

b) Sketch a bar graph of the relationship between exposure to cannabis later in pregnancy and expressive language scores.

c) Now apply the three criteria for causation. Sketch three diagrams following the model in Chapter 3, assigning one variable (cannabis exposure) to A and another (expressive language) to B. 

d) For the internal validity step, think of a specific third variable, C, that goes both with high levels of cananbis exposure and with higher levels of language development. 

e) Notably, if you read the original empirical article, you can see that the researchers used multiple regression to control for some "C" variables (i.e., third variables). They controlled for age, race, education, household income, prenatal alcohol and tobacco use, parent's anxiety, and infant sex. That means if you used one of these variables as your "C" variable in question d, we can probably rule it out. Can you think of an additional "C" variable--one that they did not control for and that is plausibly associated with both cannabis use and higher language development? 

When questionable survey methods reach the Supreme Court

According to the report, the Court may have been misled by the quality and true findings of survey on the habits of gun owners. 
Photo: B Christopher/Alamy Stock Photo

Polling research can play a critical role in shaping public opinion and public policy, so it's crucial that it be conducted well. Polling (survey) research is increasingly used in court cases, too. Because survey research can have an important impact, it's important to scrutinize a survey's research methods.

An example from the United States is a landmark gun-rights case (known as the Bruen case) from 2022, one of many pro-gun rulings in recent years.  In the Bruen case, the Court ruled that New York State could not outlaw concealed-carry handguns. In their written opinions, several of the justices cited polling research that supposedly found that  guns are "commonly" used for self-defense. 

The full story can be found here, and it was covered in the podcast, The Daily, on June 20, 2024 (available here).

According to reporting by New York Times journalists, the U.S. Supreme Court's decision in the Bruen case depended, in part, on the results of expert research: a survey of gun owners conducted by Georgetown business professor William English. English's survey data was used to argue that many gun owners use guns for self-defense. The Court accepted the research as expert testimony and allowed it to influence their reasoning.  However, when journalists started looking into the survey and the researcher who conducted it, they found instances of questionable scientific practice.

First, the researcher who conducted the survey has not made himself available to journalists to answer questions about his research and he also did not respond to requests that he testify in court about his work on the survey. This reticence to answer questions about one's research goes against scientific norms of transparency, or as Merton called it, communality. As the textbook says, "Scientists should transparently and freely share the results of their work with other scientists and the public", p. 15. From the journalist:

Government lawyers tried to talk with [Dr.English] in December, after his research surfaced in a Firearms Policy Coalition lawsuit seeking to overturn Washington State’s ban on assault weapons. But Dr. English did not respond to emails, phone calls or a certified letter, and a process server bearing a subpoena could not find him at the university or his home, where someone inside “looked at the door, then turned away,” according to court records.

Faced with efforts to compel his testimony, the pro-gun plaintiffs in [a different] Washington case chose to drop all references to his research.

Second, the Dr. English misled people about his impartiality as a scientist. Specifially, the journalists learned that the surveys he conducted were funded by pro-gun legal groups connected to the National Rifle Association. Failing to disclose one's funding is a potential conflict of interest and violates Merton's norm of disinterestedness, which says "scientists strive to discover the truth, whatever it is; they are not swayed by conviction, idealism, politics, or profit," (p. 15). From the story:

Dr. English seems at first glance to be an impartial researcher interested in data-driven insights. He has said his “scholarly arc” focuses on good public policy, and his lack of apparent ties to the gun lobby has lent credibility to his work.

But Dr. English’s interest in firearms is more than academic: He has received tens of thousands of dollars as a paid expert for gun rights advocates, and his survey work, which he says was part of a book project, originated as research for a National Rifle Association-backed lawsuit....

At least one of Dr. English's papers on the gun survey in question (which is published here) does not disclose this funding. 

Third, the journalists reported that Dr English's research methods probably led to misleading results and interpretations of the data. Specifically, the journalists reported:

... the wording of some questions could elicit answers overstating defensive gun use, and that he cherry-picked pro-gun responses.

An example comes from the self-defense question on the survey. In reporting his results, English asserted that 31% of his survey respondents said "yes" when asked, "Have you ever defended yourself or your property with a firearm, even if it was not fired or displayed?" But in that presentation, English left out the full text of the question. The Times report highlighted in blue the omitted portions:

Many policymakers recognize that a large number of people participate in shooting sports but question how often guns are used for self-defense. Have you ever defended yourself or your property with a firearm, even if it was not fired or displayed? Please do not include military service, police work, or work as a security guard.

In Chapter 6, you can read that this is a potential example of a leading question. Gun owners might be more motivated to answer "yes" to it, given its lead-in.

Here's another example of leading question wording (with a leading portion highlighted in blue), coupled with English's misleading presentation of results:

Some have argued that few gun owners actually want or use guns that are commonly classified as ‘assault weapons.’ Have you ever owned an AR-15 or similarly styled rifle? You can include any rifles of this style that have been modified or moved to be compliant with local law. Answering this will help us establish how popular these types of firearms are.

Elsewhere in his findings, Dr. English quotes about 30 gun owners attesting to the usefulness of large-capacity magazines for self-defense — but only two reported actually firing a gun, both at animals. He did not quote any of the hundreds of negative responses from gun owners, including: “There is no reason to have a magazine like that unless you’re fighting in a war.”

 

The journalist concludes that while English's research was only one piece of the full argument in Bruen and other pro-gun rulings, "...his credentials and findings carried weight. At least five briefs cited his research."

Reflection Questions:

a) What's the difference between writing a leading survey question and writing a misleading summary of one's research? 

b) What do you think might have happened in the court cases if Dr. English had been more up-front about who funded his research? 

c) What do you think might have happened in the court cases if briefs about Dr. English's presentation of his research had included the full texts of the questions he used?  

 

 

Loneliness and age: A curvilinear relationship

Loneliness is associated with poorer health. Photo: JCVStock/Shutterstock

Loneliness has been called an "epidemic" by the U.S. Surgeon General. People who report feeling lonely are also likely to suffer from poor physical health. In a recent international study on adults, researchers detected a curvilinear relationship between age and loneliness.  The study was summarized by the press office at Northwestern University and picked up by the APS News website.

The article summarized the study's key result like this:

Loneliness in adulthood follows a U-shaped pattern: It’s higher in younger and older adulthood, and lowest during middle adulthood, according to new research that examined nine longitudinal studies from around the world.  

a) Sketch a graph of this relationship. Which variable should go on which axis?  

Here are some more details: 

The study—published in Psychological Science—also identified several risk factors for heightened loneliness across the whole lifespan, including social isolation, sex, education, and physical impairment. 

b) The quote above describes some bivariate relationships between loneliness and other variables, but does not indicate which direction the relationships go. Make some predictions: Which gender do you predict is at higher risk for loneliness, and why? What direction do you predict the relationship between education and loneliness will be--will higher or lower educated people be more lonely? What about physical impairment?

After you make your predictions, you can check the results by scrolling to Figure 2 of the empirical article. 

Now read about the international nature of this study:

The study replicated this U-shaped pattern across nine datasets from studies conducted in the U.K., Germany, Sweden, the Netherlands, Australia, Israel, and more. Only one of the datasets was from the U.S., which Graham said points to how widespread the loneliness epidemic is globally.  

“Our study is unique because it harnessed the power of all these datasets to answer the same question—‘How does loneliness change across the lifespan, and what factors contribute to becoming more or less lonely over time?’”  

c) Which of the four big validities does this study's international nature address? (Internal, construct, external, or statistical?)

 

In the full empirical article in Psychological Science  you can see a figure depicting the curved functions for each country, as well as forest plots showing the effect in each of the seven countries. 

Playing outside and myopia

In one study, the more time kids spent outdoors, the less likely they were to develop myopia. Photo: Evgeny Atamanenko/Shutterstock

It's summertime--when kids head outdoors for beach time, pool parties, and picnics. Time spent outdoors is not only fun; it's also associated with lower rates of near-sightedness (myopia).  A journalist from NPR covered this topic in a recent story, titled, Want to protect your kids' eyes from myopia? Get them to play outside. 

Myopia has been increasing around the world:

In the U.S., 42% of people are now myopic – up from 25% back in the 1970s. In some East Asian countries, as many as 90% of people are myopic by the time they're young adults.

Although there are treatments for myopia, researchers are also interested in prevention. One Australian scientist, Ian Morgan, decided to test light exposure. According to NPR:

A couple of decades ago, he noticed that the rates of myopia in East Asia were much higher than they were in Sydney. [...] He knew from animal studies that light stimulates the eye to release the neurotransmitter dopamine, which can slow the eyeball from stretching. "Australians are famous for their outdoor-oriented lifestyle," he thought. "Maybe there's a link between getting outside a lot and preventing the development of myopia."

To test that theory, he and his colleagues designed a two-year study involving more than 4,000 6 and 12-year-olds in Sydney. Turns out, the researchers were right.

"The children who reported spending more time outdoors were less likely to be myopic and, we showed later on, less likely to become myopic," Morgan says of the finding, which was published in 2008.

a) In the Morgan team's study, what are the two key variables?  Were they measured or manipulated?

  In the empirical article in which the Morgan team's study was published, the conclusion reads, 

"Higher levels of total time spent outdoors ...were associated with less myopia ... after adjusting for near work, parental myopia, and ethnicity"

b) The language above is a sign that the researchers used multiple regression to control for potential confounds. What would be the criterion variable in this analysis? What were the four predictor variables?  

c) This regression result means we can rule out internal validity threats such as amount of near work, parent's levels of myopia, and ethnicity. What is another variable that you would like to see controlled for in this study? You'll need to select a third variable (C) and explain how it is associated with both myopia (A) and time spent outdoors (B). (C-->A and C-->B)

The NPR journalist reported on two, quasi-experimental designs as well:

Morgan's research caught the attention of Dr. Pei-Chang Wu, an ophthalmologist in Taiwan....

At the time, Wu's young son was starting first grade, and he worried about Taiwan's sky-high rates of myopia. Around 90 percent of teens there have it by the end of high school. Wu says the academic culture in Taiwan's primary schools didn't allow for much outdoor recess. "Many teachers want students to practice their homework during recess," he says.

But Wu convinced his son's elementary school to increase outdoor time. He also recruited a control school. A year later, his son's school had half as many new myopia cases as the other school. "We saw the results – they were very successful," Wu says.

d) What are the two variables in Dr. Wu's study?

e) What kind of quasi-experimental design is this? Pick one of the four from Chapter 13 (non-equivalent control group posttest-only, non-equivalent control group pretest-posttest, interrupted time series, non-equivalent control group interrupted time series).

Here is a second quasi-experiment:

He did more research, at more schools, and eventually convinced Taiwan's Ministry of Education to encourage all primary schools to send students out doors for at least 2 hours a day, every day. The program launched in September 2010. And after decades of trending upward, the rate of myopia among Taiwan's elementary school students began falling – from an all-time high of 50% in 2011 down to 45.1% by 2015. 

f) What kind of quasi-experimental design is this? To help you answer, take a peek at the figures in the Wu team's empirical article, which is open access here. Pick one of the four from Chapter 13 (non-equivalent control group posttest-only, non-equivalent control group pretest-posttest, interrupted time series, non-equivalent control group interrupted time series).

g) What do you think of the size of the effect here? 

 

Patients of female doctors live longer, research shows

The study found lower mortality among patients of female physicians. Photo: PeopleImages.com - Yuri A/Shutterstock

Here's a large study that tested whether the life expectancies of Medicare patients differed, depending on whether their primary physician was a woman or a man.

The website StudyFinds reports: 

The study, published in the journal Annals of Internal Medicine, analyzed Medicare claims data from 2016 to 2019 for over 700,000 patients. They found that the mortality rate for female patients was 8.15 percent when receiving care from female physicians, compared to 8.38 percent when the physician was male. While this difference may seem small, it’s considered clinically significant by researchers.

Male patients also fared better under the care of female doctors, though the difference was less pronounced. Their mortality rate was 10.15 percent with female physicians, compared to 10.23 percent with male physicians.

A similar pattern emerged for hospital readmission rates. Patients were less likely to return to the hospital within 30 days of discharge if a female doctor saw them during their initial visit. 

a) Read the above passage carefully and identify the variables the researchers studied. Classify them on this table: 

Name of the variable (Conceptual level)	What are the variable's possible levels?	Is this manipulated or measured?

b) We might consider this a correlational study. Why?  

c) If you've read Chapter 13, you might also consider this a quasi-experiment. Why?

d) The journalist reported "They found that the mortality rate for female patients was 8.15 percent when receiving care from female physicians, compared to 8.38 percent when the physician was male. While this difference may seem small, it’s considered clinically significant by researchers."  This is a good example of how context matters when it comes to effect sizes. In your own words, explain why the difference might seem small in terms of effect size. 

e) Now, in your own words, explain why this tiny difference was considered clinically significant.

To help you make this argument, consider these values. There were 458,108 female patients in this study. The difference in 30-day mortality between male and female and male physicians was 0.24%.  Multiply this percent difference (i.e., .0024) by the number of female patients to see how many additional patient deaths there were for male physicians compared to female physicians. 
 

The empirical article, which may be behind a paywall, can be found at the Annals of Internal Medicine. 

Kids' brains while reading: Screens vs. paper

The study found that children's brains were differently active when they read on screens vs. on paper. Dasha Trofimova/Shutterstock

As a reader, you've certainly had experience reading text on both screens and on paper. You probably have a preference for one over the other. Have you ever wondered whether your brain reacts differently to these different reading formats? 

A journalist from PsyPost has reported on a study that tested this question in young readers, ages 6 to 8. The study used EEG (measured by placing a net of electrodes on the head) while children read on screens and on paper. 

Here's how the journalist described the study's method: 

Each child participated in a reading session where they were exposed to two different conditions: reading from printed paper and reading from a screen. Each condition involved reading age-appropriate expository texts followed by comprehension questions to assess their understanding. The comprehension questions served both to engage the children actively with the texts and to provide a measure of how well they understood the material under each medium.

The researchers were interested in a specific brain wave pattern called the theta/beta ratio. In lots of past research, a high theta/beta ratio has been associated with worse concentration, less attention, and more mind-wandering. The researchers wondered if the theta/beta ratios of each child would differ across the two conditions. 

a) Based on what you've read so far, what are the variables in this experiment?

Name of the variable (Conceptual level)	What are the variable's possible levels?	Is this manipulated or measured?	Is this an IV or DV?	For IVs: Is it manipulated as independent groups or within-groups?

b) What type of experiment is this? Posttest only? Pretest/posttest? Repeated measures? Concurrent measures? 

Here's the journalist's description of the study's results:

... a higher theta/beta ratio was observed during screen reading compared to paper reading. This ratio is commonly interpreted as a marker of cognitive load, suggesting that children may experience more difficulty in maintaining focused attention and processing information when reading from screens.

Together, the findings provide neurobiological evidence to support the idea that reading from a screen may impose a greater cognitive burden on children, affecting their ability to concentrate and process information as effectively as when reading from paper.

c) Sketch a bar graph of this study's results.

d)  In a design like this, it would be important to employ counterbalancing. Why? Which of the four big validities is counterbalancing addressing? 

e) In this study, children in the study read two different, age-appropriate reading passages, and they also read the texts under two different conditions. How might you arrange counterbalancing for this set of circumstances? 

The empirical article is open access, on PLOSOne. 

 

Suggested answers:

a) Based on what you've read so far, what are the variables in this experiment?

Name of the variable (Conceptual level)	What are the variable's possible levels?	Is this manipulated or measured?	Is this an IV or DV?	For IVs: Is it manipulated as independent groups or within-groups?
Reading condition	On screen, on paper	manipulated	IV	Within-groups
EEG pattern: Theta/beta ratio	Higher ratio to lower ratio	measured	DV	n/a

b) Repeated measures (it's a within-groups design, and participants' brain activity was measured as/after they read first one type of text and then another type).

c) Theta/beta ratio should be on the y-axis. The two reading conditions are on the x-axis. The bar for "on screen" reading should show a higher theta/beta ratio than the bar for "on paper" reading.

d) Counterbalancing addresses internal validity; specifically, an order effect, which is an internal validity threat that is applicable only to within-groups designs. 

e) Some children would need to read Passage A on screen and Passage B on paper, and other children would need to read Passage B on screen and Passage A on paper. Similarly, some children would need to read on screens first, followed by paper; other children would need to read on paper first, followed by screens.  In the empirical article, it is not very clear how well they counterbalanced these factors. 

 

Exercise and health: Practice with mediators

Exercise is good for us. Brain-based mediators help explain why. Photo: muse studio/Shutterstock

This summary of exercise research, written by a journalist for the New York Times, is a good opportunity to practice diagramming mediators: How exercise strengthens your brain. 

Here is one passage from the article:

Exercise offers short-term boosts in cognition. Studies show that immediately after a bout of physical activity, people perform better on tests of working memory [...]. This may be in part because movement increases the release of neurotransmitters in the brain, most notably epinephrine and norepinephrine.

“These kinds of molecules are needed for paying attention to information,” said Marc Roig, an associate professor in the School of Physical and Occupational Therapy at McGill University.

 

A mediator is an explanation for why two variables might be associated. In this example, the two original variables are exercise and working memory. You can think of the author as saying this:

 

 

But then the author explains why these two variables are associated. They explain that the reason--the mediator--is the release of epinephrine and norepinephrine.  We can diagram this particular mediator argument like this:

 

 

The argument suggests that exercises leads to the release of epinephrine and norepinephrine, which leads to improvements in working memory. 

 

a). Now try it yourself with another passage. First, read to identify the two core variables. Draw them in a diagram like the first one, above. Next, redraw the diagram by adding the mediator to it. To give you a hint, I've bolded the key variables in the quoted text. (You can check your answer to this one by scrolling down).

The neurotransmitters dopamine and serotonin are also released with exercise, which is thought to be a main reason people often feel so good after going for a run or a long bike ride.

 

b) Here's another example. Read this passage to identify the two core variables (you'll notice they're similar to previous examples). Next, redraw the diagram adding the mediator to it. To give you a hint, I've bolded the key variables in the quoted text: 

It starts with the muscles. When we work out, they release molecules that travel through the blood up to the brain. Some, like a hormone called irisin, have “neuroprotective” qualities and have been shown to be linked to the cognitive health benefits of exercise, said Christiane Wrann, an associate professor of medicine at Massachusetts General Hospital and Harvard Medical School who studies irisin. 

 

c) And here's one more to try: 

The hippocampus, an area important for memory and mood, is particularly affected. “We know that it shrinks with age,” Dr. Roig said. “And we know that if we exercise regularly, we can prevent this decline.”

Exercise’s effect on the hippocampus may be one way it helps protect against Alzheimer’s disease, which is associated with significant changes to that part of the brain. 

 

Not everything is a mediator, of course. The Times article also includes this passage, which depicts a moderator:

Once [key signaling molecules] are in the brain, other chemicals are released locally. The star of the show is a hormone called brain-derived neurotrophic factor, or B.D.N.F., that is essential for neuron health and creating new connections — called synapses — between neurons. “It’s like a fertilizer for brain cells to recover from damage,” Dr. Voss said. “And also for synapses on nerve cells to connect with each other and sustain those connections.”

A greater number of blood vessels and connections between neurons [caused by exercise] can actually increase the size of different brain areas. This effect is especially noticeable in older adults because it can offset the loss of brain volume that happens with age. 

For your general enlightenment, here's a diagram of this moderator, using the diagramming conventions in Chapter 9:

 

 

Suggested answers

 

Diagrams for question a)

 

 

 

 

 

Really? Wearing high heels can improve your walking

Should we trust a study that tested only 8 people? Photo: sirtravelalot/Shutterstock

Yahoo news picked up a story from the Washington Post , which, in turn, had covered an empirical journal article about high heels and walking (did you follow that?). The Yahoo headline read, "Can wearing high heels actually improve your walking?"  In this post, we'll check out the weaknesses of this study--and think about why these two news sources thought the study would be worthy of coverage. 

Yahoo's story started out like this: 
Can wearing high heels actually be good for you? A new study published in the Journal of Applied Physiology says that wearing high heels can help you improve your walking overall. This could come as a surprise since there are many warnings against wearing high heels constantly.

Then the Yahoo journalist says:

Researchers wanted to find out what changes occur in the human leg when someone starts wearing heels for the first time. Researchers had participants wear custom heels that were, at most, 3 inches for 14 weeks, per The Washington Post.

The participants that consistently wore the heels over the 14 weeks had become efficient walkers, specifically using less energy to walk than before they started wearing the heels,

a) Based on what you've read so far, what is the independent variable in this study? What is the dependent variable?

Now, read a few more details about the study from the Washington Post article (linked here). 

For the new study, published this month in the Journal of Applied Physiology, Beck and his colleagues looked for healthy young men, as well as women, who had never or almost never worn high heels. By recruiting heel virgins, the researchers hoped to be able to track changes inside people’s legs associated with starting to wear the higher footwear.

They wound up with five male and three female volunteers.

The scientists then created high-heeled shoes acceptable to all recruits by attaching foam wedges to the soles of flat Chuck Taylor All-Star Low Top sneakers. Each pair was custom fitted so that a participant’s foot would be flexed downward at a 14-degree angle. In practice, this meant the shoes sported about a 2½- to 3-inch wedge heel.

This summary above should alert you to one of the potential problems with this study: its small sample size.

b) Why might it be a problem to have a sample of only eight people in any study? Your answer should talk not about generalization, but about how a small sample might result in results that are unusual and cannot be replicated. 

The study was a pretest-posttest design. The Post article explains that the researchers tested each of the participants on their walking efficiency (as a pretest), and then gave them their high heeled Chuck Taylors to wear for 14 weeks. After the 14 weeks, everybody's walking efficiency was tested again. 

As you know from Chapter 10, in a pretest-posttest design,  participants are tested on the dependent variable (here, walking efficiency) twice, once before and once after exposure to one level of the independent variable. So let's read more about the independent variable. Some people wore high heels and some did not, but....

[The researchers] asked them to go forth into the world and wear their new heels every day for 14 weeks. Not everyone did. Some of the volunteers “were kind of embarrassed” by the makeshift stacked sneakers, Beck said. Others complained of squished toes or other discomforts, quibbles familiar to any flat-shoe habitue who dons heels on occasion.

But more than half of the volunteers, male and female, wore the shoes most of the time. After 14 weeks, everyone returned to the lab and repeated the earlier tests. Those volunteers who had mostly given up early on wearing heels showed no changes in their legs or walking. But those who wore the heels fairly consistently tended to have shorter calf muscles and stiffer Achilles tendons than before.

c) Check that paragraph again, and ask yourself, Did the researchers randomly assign people to the "high heels" and "no high heels" conditions? Why or why not? 

d) Looking back at c), think about how this poses an internal validity problem for the study. Which of the internal validity threats (from Chapter 10 or 11) does it seem to be? 

e) Why do you think this particular study (which had a tiny sample, put only 5 people in one group and 3 in the other, not randomly assigned) was picked up by the Washington Post and by Yahoo news?  Do you think the study should have been covered by the press? 

 

You can find the original empirical journal article here. 

 

Really? "Cutting out meat might help prevent snoring"

Does the study support the causal claim the journalist attached to it? Photo: Dusan Petkovic/Shutterstock

Guest post by Carli Fine, University of Delaware

A recent news article for the general public was headlined, “Cutting Out Meat Might Help Prevent
Snoring: Study”. The journalistic source was Health Day: News for Healthier Living, which makes recommendations for the general public and has news for medical professionals. The original empirical journal article  (open access) was entitled “Plant-based and vegetarian diets are associated with reduced obstructive sleep apnoea risk”.

This study offers a great opportunity to discuss how association claims differ from causal claims and how empirical journals differ from popular journalism.

In the empirical study (Melaku et al., 2024), researchers investigated the variables of adherence level to a general plant-based diet (PDI) and risk level for obstructive sleep apnoea (OSA). PDI was measured with a 24-hour recall method, as participants reported the foods that they had
consumed in the past day.  OSA was measured using a tool called the STOP-BANG (which combines data about people's Snoring, Tiredness, Observed snorting, Pressure (blood pressure), Body mass index (BMI), Age, Neck size, and Gender). The researchers found a negative relationship between PDI adherence and OSA risk (STOP-BANG scores), such that higher PDI adherence went with lower OSA risk.

a) The journalist's headline reads, "Cutting Out Meat Might Help Prevent
 Snoring". What's the verb in this headline? To what category of claim does this verb belong: Frequency, association, or cause?  (Use Table 3.4 below for a hint.)

Table 3.4

b) What were the study's variables?  

Name of the variable (Conceptual level)	What are the variable's possible levels?	Is this manipulated or measured?

c) How do you know this was a correlational study (and not an experiment)?

d)  Sketch a well-labeled graph of the result they describe. It’s okay if you don’t know the exact
data for each group. Before you start: Will your graph be a bar graph or a scatterplot? 

 

e) Does the study support a causal claim/headline? Name, and then apply, each of the three causal criteria to this situation. When you get to the third one, you should specify a plausible third variable that could explain the relationship.

Name of causal criterion	Application to this example

f) At the end of the popular article you'll see a “What This Means For You” box, copied below. Comment on this takeaway: Does the study support the claim in the box? 

g) Scholarly sources are sometimes referred to as primary source literature, while journalistic sources may be referred to as secondary source literature. Fill out the table below identifying who the authors are for each type of article. Which type of source should you use for a literature review?  You'll find Table 2.3 helpful. 

Type of Source	“Degree” of Source	Authors of Source	Intended Audience of Source	Examples of this Type of Source
Scholarly	Primary
Journalistic	Secondary

h) Journalistic sources tend to analyze, interpret, or summarize the content of primary source literature. Besides the headline, here are a few more examples of quotes that differ between the two articles. What are some of the differences in meaning? Why do you think that the journalist made these choices?

Journalistic Source	Scholarly Source
“People with diets highest in plant-based foods were 19% less likely to have sleep apnea compared to those with the lowest amounts of plant-based foods in their diet.”	“Participants in the fifth (highest) quintile of PDI had 19% lower odds (OR=0.81; 95% CI: 0.66–1.00) of sleep apnea risk compared to those in the first quintile (p-value for trend=0.008).”
“A plant-based diet worked more powerfully to lower sleep apnea risk in men”	“Overall PDI is primarily associated with reduced odds of OSA risk in males, not females.”

 

 

 

 

Suggested Answers:

a) The popular article title is “Cutting Out Meat Might Help Prevent Snoring: Study”. The verb is “prevent”, which is a causal claim verb. The empirical article title is “Plant-based and vegetarian diets are associated with reduced obstructive sleep apnoea risk”. The verb is “associated with” which is an association claim verb.

b)

Name of the variable (Conceptual level)	What are the possible levels of this variable?	Is this variable measured or manipulated?
Level of Adherence to a General/Healthy Plant-Based Diet (PDI)	Low to High	Measured
Level of Risk for Sleep Apnoea (OSA)	Low to High	Measured

c) This was a correlational study because all the variables were measured. In an experiment, at least one variable is manipulated.

d)

e)

Name of causal criterion	Application to this example
Covariance	The results show that level of adherence to a general/healthy plant-based diet was associated with level of sleep apnoea. Yes, covariance is achieved!
Temporal Precedence	This one is tricky! The researchers asked about both variables in the same questionnaire, at the same time. However, PDI was assessed based on foods consumed in the past 24 hours, which therefore took place before the STOP-BANG tool was administered. Yes, temporal precedence is achieved!
Internal Validity	This is a correlational study, so we can’t rule out any third variables that might be responsible for the underlying relationship. For example, people who adhere more to a plant-based diet may also choose to engage more in exercise and healthy behaviors that reduce risk for sleep apnoea. Alternatively, people who have a higher socioeconomic status (SES) might have better access to and means to afford fresh fruits and vegetables and also have better access to healthcare services leading to better overall sleep hygiene. No, internal validity Is not achieved!

f) The key takeaway uses the verb “reduce” which is a causal claim. The causal claim made by the popular article cannot be supported because the study did not manipulate a variable to establish internal validity. In fact, the researchers even write, “The cross-sectional nature of the study prevents us from inferring causality between dietary indices and OSA risk.”

g) 

Type of Source	“Degree” of Source	Authors of Source	Intended Audience of Source	Examples of this Type of Source
Scholarly	Primary	Researchers who conducted a scientific study	Other scientists, researchers, readers of this blog and students who have taken research methods!	European Respiratory Society, Nature, Science, Cell, Journal of the American Medical Association
Journalistic	Secondary	Journalists or reporters, typically employed by a news organization	General population	Health Day: News for Healthier Living, The New York Times, The Washington Post, BBC News, CNN

g) You should only use empirical/scholarly/primary sources for a literature review.

h) Journalists write for a general audience that might not have specialized knowledge in a particular field. They might oversimplify the research by avoiding technical jargon and statistical terms to make it less precise but more accessible. Journalists typically want their stories to be impactful for people’s daily lives. They might make causal interpretations of correlational studies to be more actionable.