Everyday Research Methods

Do "hands free" cell phone laws make drivers less distracted?

To measure distracted driving, the study recorded how often people's cell phones moved while they were driving.
Photo: Ekaterina Pokrovsky/Shutterstock

In the summer of 2023, 28 U.S. states, including Alabama, started banning handheld cell phones while driving. This news story from the Birmingham News (AL) shows data from an interrupted time series design that tracked people's distracted phone use through automatically collected phone movement data. They plot out the results of four states (OH, AL, MI, and MO) and report values for 3 months before these hands-free laws began and for 10 months afterwards. Take a look at the figure here: https://tinyurl.com/23tt7xmq

a) Why might this design be called an "interrupted time series"? (Chapter 13 will have definitions to help you.)

b) Even though there are four states depicted on the main figure, this is probably not a "non-equivalent control group interrupted time series design." That's because there are no examples of states that did not implement a hands-free ban that summer. When we are trying to decide whether the hands-free cell phone ban reduced distracted driving, it would be helpful to have a set of states with no ban. Why? (You might use the term "history threat" in your answer).

c) Based on the figure, what do you think of the effect size in each state?

d) This study's interrupted time series has only about three months of data from before the time when the bans went into effect. Why might we want more than that as a baseline?

You might wish to look through an official report with bar-graph versions of the data, which can be viewed here: https://m.cmtelematics.com/hubfs/A%20Road%20Map%20for%20Safer%20Roads.pdf 

I'm grateful to Stephen Chew, who shared this story with me.

Body Scan Meditation for Food Cravings

 

A body-scan meditation might prevent mindless trips to the fridge.
Photo: F8 studio/Shutterstock

Do you consider yourself an emotional eater? If you'd like to reduce your susceptibility to emotional eating, here's something to try. It's also a good chance to practice one of the simple experiments. The research was summarized by a journalist for PsyPost.  

Recently, scientists have begun investigating the effects mindfulness-based interventions may have on emotional eating.[...]  One such practice is body scan meditation, which involves slowly focusing attention on different parts of the body to promote relaxation and awareness of physical sensations.

The study participants were 81 adults classified as emotional eaters based on the results of the Dutch Eating Behavior Questionnaire. Seventy-four percent of them were female. Participants’ average age was 27, and 45% identified as European or North American.

The researchers randomly divided participants into two groups. One group underwent a 16-minute body scan meditation, while the other group listened to a recording of an equally long description of the human musculoskeletal system. Neither the participants nor the researcher working with them was informed of which condition was experimental and which was the control.

Before the intervention, participants completed an interview in which researchers asked them to recall a recent episode of emotional eating. This was intended to induce a negative emotional state in participants.

Participants also completed several assessments, including "negative emotion (using the Positive and Negative Affect Scale)" and "food cravings intensity (using the Desire subscale of the Food Cravings Questionnaire – state version)", as well as other variables. 

Results showed that participants who completed the body scan meditation session experienced a greater reduction in negative affect and cravings from before to after the treatment compared to the group subjected to the listening task. 

a) Reread the description above and identify the study’s variables.

Variable name	What were this variable’s levels?	Was this manipulated or measured?	Was this variable an IV? Or a DV?	For the IV: was it independent groups or within groups?

b) What kind of experiment was this: Posttest only? Pretest/posttest? Repeated measures? Or concurrent measures?

c) Sketch a graph of the results. Don't forget to include pretest and posttest data. 

d) Here is a list of internal validity threats in Chapter 11. Check the definition of each term in Table 11.1.  Which of these threats is likely ruled out by the study's design, and why? 

Selection effect
Order effect
Design confound
Maturation
History
Instrumentation
Testing
Attrition
Regression to the mean
Placebo effects
Demand characteristics
Observer effects

 

Cooling down with slushies

Those tiny pieces of ice help cool you down. Photo: DelightPhotos/Shutterstock

Summer’s almost over, but you should save this idea for next time you’re feeling too hot. It turns out that the best way to cool yourself down is by eating small pieces of ice—like those found in your favorite slushy.

You can see a short video about this study on the Weather Channel, and you can read a bit more about it here, at the Washington Post.

The Weather Channel’s video says:

Here’s what multiple studies found: Drinking a slushy can drop your internal body temperature by 1 degree in 20 minutes

and

The slushy method has been tested and works for firefighters, cyclists, and triathletes.

The video explains that when you ingest tiny pieces of ice, your body sends energy to your belly to melt them. This draws energy (and heat) away from the rest of your body, cooling you off.

a) Before reading on, how might you conduct an experimental study to test the slushy method of cooling? What would your IV need to be? What would your DV be?

Now read these lines from the Washington Post journalist to see how close these studies came to your answer in a).

[A] study on Australian firefighters found that drinking a big slushy was just as effective at lowering core body temperature as sitting in a tub of cool water up to your bellybutton. 

b) According to this summary, there were at least two groups in this Australian firefighters study. The journalist does not mention a control group, but do you think they would need one? Why or why not?

The Post article included a subsection asking, How big does the slushy need to be?

The amount of slushy you need to cool down depends on your weight.

Both Maley’s 2020 study and the earlier 2014 study used a similar formula that translates to about one fluid ounce per eight pounds of body mass. For the firefighters in the 2014 study, who weighed around 185 pounds, that would be 23 ounces, which is roughly one large Slurpee from 7-Eleven.

c) Even though Maley’s two studies followed a 1 fluid ounce per 8 pounds rule, that doesn’t necessarily mean the 1-to-8 ratio is ideal. How could you design a study that would test the relative efficacy of different ratios of slushy-to-body weight? Specifically, what would the IV need to be to test this question? What levels might you test? What would the DV be?

The Post article also asked about timing:

It doesn’t matter how fast you drink it, as long as you don’t let it melt. The firefighters in the 2014 study slurped their slushies in 15 minutes after exercising, while in the 2020 study, they sipped them over the course of half an hour before exercising. The results were the same: Their internal body temperatures dropped by about 1 degree Fahrenheit.

d) Some researchers would prefer to test the timing question in a single experiment. How could you design a study that would test the relative efficacy of different timings of slushy consumption on body temperature? What would your IV and its levels be? What would the DV be? 

Clowns help kids get out of the hospital faster

 

We can apply the four big validities to this study on medical clowns. Photo: Halfpoint/Shutterstock

Here’s a study about a behavioral intervention that apparently helps hospitalized children recover faster. The study tested the effect of medical clowns—a specially trained person who can distract hospitalized kids, guide them through relaxation exercises, and help them feel more relaxed during a stay in the hospital.

 

Some researchers experimentally tested the use of medical clowns. Their recently-published empirical study was covered by the journalistic source, Study Finds.

These aren’t your average birthday party clowns. Medical clowns are specially trained professionals who use humor, music, and playfulness to help patients cope with the stress of being in the hospital. […]

So, how did the researchers figure out that clowns could help kids with pneumonia? They looked at 51 children between the ages of two and 18 who were in the hospital with pneumonia. The kids were split into two groups. One group received standard medical care, while the other got the same care plus two 15-minute visits from a medical clown each day for the first two days of their hospital stay.

The clowns, who were part of a group called The Dream Doctors Project, used various techniques to help the kids relax. They played music, sang songs, and even used something called “guided imagination”

…The [results showed that] kids who got to hang out with the clowns got better faster! On average, they were able to leave the hospital after about 43.5 hours, compared to 70 hours for the kids who didn’t see the clowns. That’s more than a full day difference!

a) Reread the description above and identify the study’s variables.

Variable name	What were this variable’s levels?	Was this manipulated or measured?	Was this variable an IV? Or a DV?	For the IV: was it independent groups or within groups?

b) What kind of experiment was this: Posttest only? Pretest/posttest? Repeated measures? Or concurrent measures?

c) Sketch a bar or line graph of the study’s results. Remember to put the dependent variable on the y-axis.

Now let’s apply the four big validities to this design.

d) Internal validity: The journalist says that “The kids were split into two groups”. But it’s not specified that the children were randomly assigned to the two groups. Why would random assignment be important here?

Imagine that instead of random assignment, the researchers had asked parents which group they’d like their hospitalized child to be in (the clown group or the standard care group). Why would that be an internal validity problem?

e) Statistical validity: After describing the study’s results, the journalist reports: “That’s more than a full day difference!” Which statistical validity concept is the journalist addressing here?

f) External validity: What questions might you ask to determine this study’s ability to generalize?

g) Construct validity: The main measured variable in this study was how many hours children were kept in the hospital. Do you foresee any potential issues with the measurement of this variable?

Suggested answers:

a) Reread the description above and identify the study’s variables.

Variable name	What were this variable’s levels?	Was this manipulated or measured?	Was this variable an IV? Or a DV?	For the IV: was it independent groups or within groups?
Clown condition	Medical clown or Standard care only	Manipulated	IV	Independent groups
Hours in the hospital	0 up to many (100 perhaps?)	Measured	DV	n/a

B) This was a posttest only design because the IV was manipulated as independent groups and the participants were measured on the DV only once.

Now let’s apply the four big validities to this design.

c) Random assignment would be important here to ensure that kids in the two groups were approximately equal in how sick they were, how old they were, how much they like clowns, how good their doctors were—in short, equal on almost any relevant individual differences that might provide an alternative explanation for the “clown” group getting out of the hospital sooner.

If, instead of random assignment, the researchers asked parents which group they’d like their hospitalized child to be in (the clown group or the standard care group), that would  introduce an internal validity problem—a confound. Specifically, it would introduce a selection effect in which there were systematically different types of participants in the two groups. The researchers couldn’t be sure if the medical clown kids had shorter hospitalizations because of the clowns, or because the kids’ parents were more motivated to provide them with extra services and care.

d) The journalist is addressing the effect size of the study—how strong or how large the difference was between the two conditions. Effect size is one of the standard things to ask about when it comes to Statistical Validity (you’d also want to ask about the precision of the estimate [the 95% CI] and about replication—what other studies have found).

e) You might ask how they got this sample—was it a random sample of children from this hospital? (if so, you could at least generalize to that hospital’s population.) You might also ask if the medical clown results would generalize to kids who are hospitalized for other conditions, such as cancer or other illnesses. (This would require conducting additional studies using samples from these populations.)

f) You might disagree, but to me, it seems that “time spent in the hospital” is a very reliably measured variable. It is probably included clearly on each kid’s medical records and is less susceptible to problems such as response biases.

Emotional and behavioral outcomes of bed-sharing with infants

Is this non-bed-sharing baby more or less likely to show emotional or behavioral symptoms? Photo: morrowlight/Shutterstock

Bed sharing, in which parents sleep in the same bed with their infants, is a fairly common practice in North America and the United Kingdom. Parents who bed-share sometimes do so to facilitate breastfeeding, or because they believe it helps babies feel safe or that it will lead to closer emotional relationships. 

However, many people believe that bed-sharing might be bad for children's development. And, the American Association of Pediatrics officially recommends against adults sharing a sleeping surface with infants under the age of six months because of the potential risk of sudden infant death.  

A large study aimed to see if there were relationships between bed sharing and emotional and behavioral outcomes. The study was summarized by PsyPost here. The study illustrates the use of regression (Chapter 9) and is an example of a null effect (Chapter 11). 

The researchers aimed to clarify whether bed-sharing during infancy has any lasting influence on a child’s development, particularly concerning emotional stability and behavioral tendencies.

The study utilized data from the Millennium Cohort Study, a large, nationally representative longitudinal study from the United Kingdom. This cohort followed over 18,000 infants born in the early 2000s, tracking their development through various stages of childhood. For this particular research, the focus was on children who were bed-sharing at 9 months of age and their subsequent emotional and behavioral development up until they were 11 years old.

a) Given the information above, what can you infer about the external validity of this study--can it generalize to the population of children in the UK?

To assess emotional and behavioral outcomes, the researchers used the Strengths and Difficulties Questionnaire, a widely respected tool for measuring psychological adjustment in children. This questionnaire, completed by parents when the children were 3, 5, 7, and 11 years old, allowed the researchers to track patterns of internalizing symptoms, such as anxiety and depression, and externalizing symptoms, like aggression and hyperactivity, over time.

b) The passage above addresses construct validity. It tells you how emotional and behavioral outcomes were measured, but it doesn't say much about whether this measure is reliable and valid. What might you want to know about this measure to decide if it is construct valid? 

The study also accounted for several other factors that could influence a child’s development, including gender, socio-economic status, night-waking frequency, breastfeeding practices, maternal psychological distress, and parenting beliefs. By considering these variables, the researchers aimed to isolate the specific impact of bed-sharing on child development.

Interestingly, while bed-sharing at 9 months was more common among children in the groups with elevated symptoms [of emotional and behavioral symptoms], the researchers found no direct evidence linking bed-sharing to an increased risk of these symptoms once other factors were taken into account. In other words, after considering variables like parenting beliefs and maternal distress, bed-sharing itself did not predict whether a child would belong to one of the higher-risk groups.

c) Read the passage above and restate the simple bivariate relationship between bed-sharing and emotional/behavioral symptoms. 

d) Now consider the regression analysis. What is the criterion variable? 

e) What are the predictor variables? (Hint--there are seven of them)

f) Given the results described above, what might the beta have been for the bed-sharing predictor variable? 

To summarize the results so far: There was a bivariate relationship between bed-sharing and symptoms. However, it seems that this bivariate relationship was attributable to some third variable like parenting beliefs or maternal distress because when these variables were controlled for, the relationship between bed-sharing and emotional/behavioral symptoms went away. 

g) In Chapter 11's section on null effects, you learn that we can be more confident that null effects are real in the population when the study has more power and precision. What do you think--to what extent does this study help support that there is no relationship between bed-sharing and emotional/behavioral symptoms? 

 

Suggested Answers to Selected Questions

b) we might want to know why the Strengths and Difficulties Questionnaire is described as "well-respected." Do parent's reports on this scale also go with other evidence of children's difficulties or psychologists' ratings?

c) The bivariate relationship is that kids who bed-shared at age 9 months had more emotional/behavioral symptoms at age 10-11.

d) The criterion variable was emotional/behavioral symptoms

e) The predictor variables were bed-sharing, gender, socio-economic status, night-waking frequency, breastfeeding practices, maternal psychological distress, and parenting beliefs.

f) The beta for bed-sharing would have been zero, or close to zero, because the text says, "after considering variables like parenting beliefs and maternal distress, bed-sharing itself did not predict whether a child would belong to one of the higher-risk groups."

g) This study's strength is its large sample of 16,000 children. Such large samples lead to more narrow 95% CIs and more precision in our estimates. If we can't find an effect of bed-sharing with this large of a sample, we probably won't find it. In other words, it would take a lot of exceptional cases to budge our "zero" conclusion. I think we can trust it. 

Can stress turn your hair gray?

Correlational and experimental research seems to converge on the conclusion that stress causes gray hair. Photo: Kunlathida6242/Shutterstock

Can stress turn your hair gray? Here's a story on gray hair research, written by a New York Times science journalist. In our culture, we certainly believe that gray hair can be caused (in some cases at least) by stress. There is clear evidence that gray hair is inherited genetically, apparently stress plays a role, too 

Here's how the journalist introduces the first study:

In past studies, researchers have asked participants to fill out questionnaires about their hair color and stress levels, and then the scientists would see if they could link them.

In one study published in 2016, for instance, scientists surveyed more than 1,100 young Turkish adults and found that the 315 who reported prematurely graying hair had higher stress levels than those who didn’t. 

a) Identify the two variables in this study.

b) Were the variables manipulated or measured?

c) Was this a correlational or experimental study? 

d) Can the results support a claim that stress causes premature gray hair? Why or why not? Apply the three criteria for causation: Covariance, Temporal Precedence, and Internal Validity.

For question d above, you should have responded no; although this study did show a relationship between premature graying and perceived stress, the study doesn't establish temporal precedence since both variables were measured at the same time. And, it's possible some third variable is responsible--for example, maybe older participants have both more stress and more gray hair. (Note that even though genetics is the biggest predictor of premature gray hair, genetics isn't a confound here because the genes for gray hair don't also predict perceived stress.)

To show causation, it helps to have an experiment. Here's an example: 

But a mouse study published in 2020 took the research a step forward. In it, researchers stressed mice in various ways, including by injecting them with a chili-pepper-like chemical that induced a “fight-or-flight” response. This caused them to release the stress hormone norepinephrine, which, in turn, depleted their hair follicles of the stem cells involved with adding pigment to mouse fur. The hair then grew in gray.

But studies on this topic are challenging to perform on people, because researchers can’t ethically induce artificially high stress responses in humans like they can in animals or cells, Dr. Hsu said.

e) What is the independent variable in this study?

f) What were the dependent variables (there were two)? 

g) If you've read Chapter 9, you'll recognize this as a mediator argument: "[The chili-pepper chemical] caused them to release the stress hormone norepinephrine, which, in turn, depleted their hair follicles of the stem cells involved with adding pigment to mouse fur. The hair then grew in gray." Can you draw this mediator argument using boxes for each variable? 

Here's one more study on humans that was better able to manage temporal precedence: 

One small human study published in 2021 still advanced the narrative: Researchers plucked various strands of hair from 14 volunteers who had at least some graying. Several of the strands were fully gray, some were partially gray and some hadn’t grayed at all. The scientists then created high-resolution digital images of the hairs and calculated when each strand went gray using estimates of how quickly hair grows.

They also asked the participants to plot out stressful experiences from the past year on a timeline, and rank them from least to most distressing. The researchers found that when a strand turned gray frequently corresponded with the most stressful moment of that volunteer’s previous year.

h) There seem to be two variables in this study. Can you name them and their levels?

i) Was this a correlational or experimental study? 

Taken together, the three studies here can be seen as an example of "pattern and parsimony." Some of the studies are correlational, others are experimental. Some are on humans and some are on animals. But they all seem to parsimoniously support an argument that stress precedes gray hair. What do you think?

 

Burrito bowl variability

But how big will the next burrito bowl be?
Generic burrito bowl photo: B Young/Shutterstock

An important concept in both statistics and methods is variability. And this analysis conducted by Wells Fargo consultants does a nice job illustrating the concept of variability. They'd been seeing social media complaints about a popular restaurant chain wasn't always providing a full burrito bowl, and decided to look into it. 

Chipotle recently got a wave of social media attention from users claiming the fast-and-casual chain was holding back. Analysts at Wells Fargo sought to put the debate to rest and ordered 75 identical burrito bowl orders across eight locations in New York City.

You need to see the results of the burrito bowl analysis, so click here for a figure from Sherwood.com  (a business news site).

a) Orient yourself to this figure. What do you see on the y-axis? What's on the x-axis? What does one bar represent? 

b) Find the bar on the graph representing the largest/fullest burrito bowl. Find the bar on the graph representing the lightest one. 

c) If you were this fast food chain, what would be the shape of the graph that you'd most like to see? What actions would you  take in your stores and franchises to achieve this shape?

d) The phenomenon depicted in the graph originally reminded me of inter-rater reliability, but it's really not the same. Why not? 

e) How do you think they sampled the restaurants for this analysis? Would it be important to use a random sample, or not? 

Selected answers

a) Here, one bar represents a single burrito bowl; its height corresponds to how heavy it was. 

b) The lightest bowl is all the way on the right.

c) The restaurant chain probably wants to see a rectangular distribution, in which all of the bars are the same height--this would indicate that every burrito bowl weighed the same.

To get closer to this shape, you'd probably need to do some training of your chefs, setting clearer standards for how many ingredients to use in each burrito bowl. 

d) In inter-rater reliability, you're looking to see if two or more raters make consistent observations of a set of people or objects. In other words, you want to make sure that the behavior of a set of targets (say, schoolchildren) is rated the same no matter who rates them. (In this example, Wells Fargo might have wanted to make sure that two analysts who weighed a set of burrito bowls, would come up with exactly the same weights.) In contrast, if you are a restaurant owner, you'd be looking for something we might call "inter-burrito bowl reliability," in which burrito bowls weigh the same no matter who makes them. 

That said, if people have poor inter-rater reliability, you'd see too much variability between them. And if burrito chefs have poor inter-bowl reliability, you'd see a graph that looks a lot like the one we're talking about. 

e) Sure, random sampling of restaurants would support external validity--it would mean that we could generalize from this data--we'd know that the median and the standard deviation in this sample could be generalized to the population restaurants. However, we probably don't need a random sample to support a proof of concept--that is, it's probably a problem that in this sample of 75 burrito bowls, there's a great deal of variability, even if those 75 bowls and restaurants weren't sampled randomly. 

Thanks to Jess Hartnett and Stephen Chew, who shared this one on twitter.

Running on cannabis

How did the researchers manipulate the IV in this study?
Photo credit: mansong suttakarn/Shutterstock

Cannabis is considered a performance-enhancing drug in the Olympic Games and other competitions, but it's also (stereotypically) associated with being too chill to work out. In the past, psychologists could not easily test the relationship of marijuana and exercise because the drug was illegal. Now, however, since recreational marijuana is legal in several U.S. states, researchers can ethically test its effects on humans. 

One such study investigated the effect of cannabis on how runners experience their workouts. Here's a summary of the research by the online source, ScienceAlert. 

In a recent experiment, 42 healthy adult runners from Colorado who identified as regular cannabis users ran on a treadmill at a moderate pace for 30 minutes. Before, during, and after the run, scientists monitored their physical and mental state. On another occasion, participants were given the choice to use either THC or CBD products before a similar 30-minute run.

THC, or tetrahydrocannabinol, is the compound in cannabis largely responsible for the plant's psychoactive effects. Whereas CBD, or cannabidiol, is a compound in the plant that offers muscle-relaxing, anti-inflammatory effects without producing the trippy mind-altering effects traditionally associated with marijuana.

"The bottom-line finding is that cannabis before exercise seems to increase positive mood and enjoyment during exercise, whether you use THC or CBD. But THC products specifically may make exercise feel more effortful," says psychologist Laurel Gibson from UCB.

[...]Compared to running sober, running high was more enjoyable for participants, even if the inclusion of THC did make the exercise feel slightly harder.

a) Here's a table on which you can classify the variables in this study. Here's a hint: there were two conceptual dependent variables. 

Name of the variable (Conceptual level)	What are the variable's possible levels?	Is this manipulated or measured?	Is this an IV or DV?	For IVs: Is it manipulated as independent groups or within-groups?

 

b) This was an experiment. Why? (Use the term, "manipulated" in your response). 

c) What type of experiment is this? Posttest only? Pretest/posttest? Repeated measures? Concurrent measures?

d) Sketch a graph of the "enjoyable" result that is described in the journalist's coverage above. 

Now let's consider the internal validity of the study. Read this detail: 

Because of ethical considerations, the study was not double-blinded or randomized, and dosage among participants wasn't regulated.

e) The study was "not double-blinded." What are the names and definitions of the internal validity threats (see Table 11.1 in Chapter 11) that would be impacted by this decision? 

The journalist says that the study was "not randomized." What do you think they are criticizing? Is this critique relevant for a within-groups design? Why or why not? 

And, if dosage among participants wasn't regulated, is that a serious problem for the design, or not? Think through how that might have affected this design. 

Here's another detail from the journalist's coverage: 

Furthermore, the group of participants was limited to include regular cannabis users who ran a lot, which means enrollment could very well be biased towards those with positive cannabis experiences while exercising.

f) Which of the four big validities is this comment addressing? 

Really? Prenatal cannabis exposure and early language development

Photo: Viacheslav Lopatin/Shutterstock

A story in PsyPost was headlined with, Prenatal cannabinoid exposure appears to have a strange impact on early language development. 

You might have noticed that the verb phrase, "has a strange impact on" is a causal one. We are being asked to believe that prenatal exposure to cannabis causes something in children's early language development.  I hope your causality suspicions are activated here, since it's almost certain that both cannabis exposure and language development are measured variables. This was probably a correlational study. 

Let's read on to find out what the mysterious "strange impact" is.

The journalist reported on a study in the open-source journal, Frontiers in Pediatrics. The PsyPost journalist drops the "strange impact" in the first paragraph:

...infants exposed to cannabis in the womb tend to exhibit improved language development by their first birthday. This surprising finding sheds light on the impacts of prenatal cannabis exposure on one aspect of early childhood neurodevelopment. However, experts advise that women avoid cannabis use during pregnancy due to known risks and unknown long-term effects.

Given past research, it's a bit surprising to find a positive relationship between cannabis use and child development, so that was the "strange impact" PsyPost mentions in their headline.

Here's how the journalist summarized the study's methods:

The researchers recruited 207 pregnant individuals and their 12-month-old infants who were part of The Safe Passage Study. This was a large prospective study that followed participants from 2007 to 2015.

Talavera-Barber and colleagues assessed the infants according to the Mullen Scale of Early Learning, a tool that assesses cognitive development in children from 2 days old to 68 months. 

[...]Prenatal cannabis exposure was collected from the mothers through self-report. Participants were categorized based on whether the exposure to cannabis occurred early (first trimester only; 51 participants) or late (second or third trimester; 18 participants), and they were randomly matched with unexposed participants for comparison (138 participants).

Don't be misled by the term "randomly matched with unexposed participants." A peek at the full empirical article shows that these were simply people in the sample who did not use cannabis. They were randomly selected from a larger set of participants in the study--but that doesn't mean they were "randomly matched." (In fact, I'm not sure I know what the journalist meant by "randomly matched" here.)

Questions

a) Based on what you've read so far above, complete this table of the variables in the study:

Variable name (stated at the conceptual level)	How was this variable operationalized?	Is this a measured or a manipulated variable?

Some of the results were summarized this way by PsyPost:

Unexpectedly, infants who were exposed to cannabis later in pregnancy scored higher in both expressive and receptive language areas than those who were not exposed.

Additionally, infants exposed early in pregnancy showed better gross motor skills, although there was no difference in fine motor skills and visual reception skills.

b) Sketch a bar graph of the relationship between exposure to cannabis later in pregnancy and expressive language scores.

c) Now apply the three criteria for causation. Sketch three diagrams following the model in Chapter 3, assigning one variable (cannabis exposure) to A and another (expressive language) to B. 

d) For the internal validity step, think of a specific third variable, C, that goes both with high levels of cananbis exposure and with higher levels of language development. 

e) Notably, if you read the original empirical article, you can see that the researchers used multiple regression to control for some "C" variables (i.e., third variables). They controlled for age, race, education, household income, prenatal alcohol and tobacco use, parent's anxiety, and infant sex. That means if you used one of these variables as your "C" variable in question d, we can probably rule it out. Can you think of an additional "C" variable--one that they did not control for and that is plausibly associated with both cannabis use and higher language development? 

When questionable survey methods reach the Supreme Court

According to the report, the Court may have been misled by the quality and true findings of survey on the habits of gun owners. 
Photo: B Christopher/Alamy Stock Photo

Polling research can play a critical role in shaping public opinion and public policy, so it's crucial that it be conducted well. Polling (survey) research is increasingly used in court cases, too. Because survey research can have an important impact, it's important to scrutinize a survey's research methods.

An example from the United States is a landmark gun-rights case (known as the Bruen case) from 2022, one of many pro-gun rulings in recent years.  In the Bruen case, the Court ruled that New York State could not outlaw concealed-carry handguns. In their written opinions, several of the justices cited polling research that supposedly found that  guns are "commonly" used for self-defense. 

The full story can be found here, and it was covered in the podcast, The Daily, on June 20, 2024 (available here).

According to reporting by New York Times journalists, the U.S. Supreme Court's decision in the Bruen case depended, in part, on the results of expert research: a survey of gun owners conducted by Georgetown business professor William English. English's survey data was used to argue that many gun owners use guns for self-defense. The Court accepted the research as expert testimony and allowed it to influence their reasoning.  However, when journalists started looking into the survey and the researcher who conducted it, they found instances of questionable scientific practice.

First, the researcher who conducted the survey has not made himself available to journalists to answer questions about his research and he also did not respond to requests that he testify in court about his work on the survey. This reticence to answer questions about one's research goes against scientific norms of transparency, or as Merton called it, communality. As the textbook says, "Scientists should transparently and freely share the results of their work with other scientists and the public", p. 15. From the journalist:

Government lawyers tried to talk with [Dr.English] in December, after his research surfaced in a Firearms Policy Coalition lawsuit seeking to overturn Washington State’s ban on assault weapons. But Dr. English did not respond to emails, phone calls or a certified letter, and a process server bearing a subpoena could not find him at the university or his home, where someone inside “looked at the door, then turned away,” according to court records.

Faced with efforts to compel his testimony, the pro-gun plaintiffs in [a different] Washington case chose to drop all references to his research.

Second, the Dr. English misled people about his impartiality as a scientist. Specifially, the journalists learned that the surveys he conducted were funded by pro-gun legal groups connected to the National Rifle Association. Failing to disclose one's funding is a potential conflict of interest and violates Merton's norm of disinterestedness, which says "scientists strive to discover the truth, whatever it is; they are not swayed by conviction, idealism, politics, or profit," (p. 15). From the story:

Dr. English seems at first glance to be an impartial researcher interested in data-driven insights. He has said his “scholarly arc” focuses on good public policy, and his lack of apparent ties to the gun lobby has lent credibility to his work.

But Dr. English’s interest in firearms is more than academic: He has received tens of thousands of dollars as a paid expert for gun rights advocates, and his survey work, which he says was part of a book project, originated as research for a National Rifle Association-backed lawsuit....

At least one of Dr. English's papers on the gun survey in question (which is published here) does not disclose this funding. 

Third, the journalists reported that Dr English's research methods probably led to misleading results and interpretations of the data. Specifically, the journalists reported:

... the wording of some questions could elicit answers overstating defensive gun use, and that he cherry-picked pro-gun responses.

An example comes from the self-defense question on the survey. In reporting his results, English asserted that 31% of his survey respondents said "yes" when asked, "Have you ever defended yourself or your property with a firearm, even if it was not fired or displayed?" But in that presentation, English left out the full text of the question. The Times report highlighted in blue the omitted portions:

Many policymakers recognize that a large number of people participate in shooting sports but question how often guns are used for self-defense. Have you ever defended yourself or your property with a firearm, even if it was not fired or displayed? Please do not include military service, police work, or work as a security guard.

In Chapter 6, you can read that this is a potential example of a leading question. Gun owners might be more motivated to answer "yes" to it, given its lead-in.

Here's another example of leading question wording (with a leading portion highlighted in blue), coupled with English's misleading presentation of results:

Some have argued that few gun owners actually want or use guns that are commonly classified as ‘assault weapons.’ Have you ever owned an AR-15 or similarly styled rifle? You can include any rifles of this style that have been modified or moved to be compliant with local law. Answering this will help us establish how popular these types of firearms are.

Elsewhere in his findings, Dr. English quotes about 30 gun owners attesting to the usefulness of large-capacity magazines for self-defense — but only two reported actually firing a gun, both at animals. He did not quote any of the hundreds of negative responses from gun owners, including: “There is no reason to have a magazine like that unless you’re fighting in a war.”

 

The journalist concludes that while English's research was only one piece of the full argument in Bruen and other pro-gun rulings, "...his credentials and findings carried weight. At least five briefs cited his research."

Reflection Questions:

a) What's the difference between writing a leading survey question and writing a misleading summary of one's research? 

b) What do you think might have happened in the court cases if Dr. English had been more up-front about who funded his research? 

c) What do you think might have happened in the court cases if briefs about Dr. English's presentation of his research had included the full texts of the questions he used?