Everyday Research Methods

Evidence in favor of U.S. stimulus checks

Which type of quasi-experimental design does this study seem to fit best? The dark line on top represents respondents with children. The red line represents all respondents. The lighter blue line at the bottom represents respondents without children. Credit: Courtesy Patrick Cooney and H. Luke Shaefer. From Poverty Solutions, May 2021, University of Michigan,

Since the COVID-19 pandemic began, several governments, including the U.S. federal government, have distributed cash payments to citizens in an attempt to alleviate the  financial hardship caused by the global shutdown. 

As the New York Times reports here, policymakers wonder if such payments are useful. Now we have data to help address that research question. 

In offering most Americans two more rounds of stimulus checks in the past six months, totaling $2,000 a person, the federal government effectively conducted a huge experiment in safety net policy. Supporters said a quick, broad outpouring of cash would ease the economic hardships caused by the coronavirus pandemic. Skeptics called the policy wasteful and expensive.

[...]

A new analysis of Census Bureau surveys argues that the two latest rounds of aid significantly improved Americans’ ability to buy food and pay household bills and reduced anxiety and depression, with the largest benefits going to the poorest households and those with children. The analysis offers the fullest look at hardship reduction under the stimulus aid.

The full report is available here, and provides some excellent graphed examples of quasi-experimental data. We'll focus on Figure 1 from the report, reprinted above with permission from the authors.

The y-axis on the figure represents the percentage of Americans who "sometimes or often had not enough food to eat in the last seven days." The question is asked by U.S. Census' Household Pulse survey about once per month to a random sample of Americans. The dates of the two U.S. stimulus checks are indicated in the laddered, light-blue vertical lines. Payments occurred in December, 2020 and March, 2021.

Questions

a) The journalist writes that "the federal government effectively conducted a huge experiment in safety net policy".  What is the independent variable in this "experiment?"  What is the dependent variable (i.e., the one depicted in Figure 1)?

b) The independent variable here is not a true IV--it is a quasi-independent variable. What makes this variable quasi-independent?

c) The authors claim that the results from Figure 1 are consistent with the argument that the stimulus checks helped people. The authors wrote in their report: "Based on the speed with which we see hardship fall, we suspect much of this drop was the result of EIP checks, which the federal government was able to quickly deliver to bank accounts for most U.S. households following passage of both bills." Locate the areas on the figure that the authors are referring to. 

d) This quasi-experimental study is probably best described as an interrupted time-series design. What is the "interruption" here? What is the time series?

e) A history threat is one potential internal validity problem in an interrupted time-series design. Explain what a history threat is and also why a history threat might be a problem here. What do you think--to what extent can we be sure it was the stimulus checks, and not some other pair of events, that is responsible for the increased food security depicted here? 

You can see results for other variables in the report here, including figures that tracked people's feelings of anxiety and their feelings of depression. 

 

 

 

 

 

Be quiet! Psychological effects of noise

In one of the studies, researchers found that students whose classrooms were on the train-track side of the school made less progress in reading than those on the quieter side.  Credit: Alon Adika/Shutterstock

The weekly Freakonomics podcast recently put together an episode about the psychological impact of loud noises in our environment. You can listen to it (or read the transcript) here. 

The host and guest review several studies with a variety of methodologies, which include some excellent examples of quasi-experiments.

The introduction to the transcript reads, 

The modern world overwhelms us with sounds we didn’t ask for, like car alarms and cell-phone “halfalogues.” What does all this noise cost us in terms of productivity, health, and basic sanity?

 

Example I.

The first example comes from animal behavior and behavioral ecologist Peter Tyack, who studied whales in the Bay of Fundy. The Bay provided lots of food for the whales, but as a site of many shipping channels, the same place always had the background noise of ships going to and fro. 

Researchers like Tyack had never really thought much about this background noise. It was just there. But then all of a sudden it wasn’t. The change came with the 9/11 terrorist attacks.

TYACK: All of a sudden, the ships that were plying the ocean in that area stopped. 

This drop in ship traffic was only temporary. But it happened to coincide with some other whale research that was happening in the Bay of Fundy:

TYACK: Researchers from the New England Aquarium had been sampling feces from whales to look at stress hormones. It wasn’t part of a noise experiment at all. 

It was just your standard whale-feces research.

TYACK: But what they found was that if you compared the stress hormones in whales before 9/11 and after 9/11, their stress hormones actually went down after 9/11. 

a) In the quasi-experiment described here, what is the quasi-independent variable? What are its levels? What is the dependent variable?

b) Let's assume that the researchers collected and tested whale feces nearly every day, several days before 9/11 and several days afterwards, as well. What type of quasi-experiment is it (nonequivalent control groups, nonequivalent control group prettest-posttest, nonequivalent control groups interrupted time series, nonequivalent control groups interrupted time series)?  

c) Sketch a graph of the results of the whale study. 

Example 2

Here's another example: This one took place in a public school in New York City. The school was built right next to an elevated subway track, so kids in one of the school were exposed to the loud noise of subway trains approximately every four minutes. 

One side of the school building faced a nearby elevated subway; the other side faced away. Bronzaft matched second-, fourth-, and sixth-grade classrooms on the quiet side and on the noisy side, where a passing train would push the sound readings from 59 decibels to 89 decibels. Then she compared the average reading scores from the two sets of classrooms.

BRONZAFT: And the children exposed to the transit noise were nearly a year behind in reading by the sixth grade, and the teacher had difficulty teaching.

d) In the quasi-experiment described here, what is the quasi-independent variable? What are its levels? What is the dependent variable?

e)  What type of quasi-experiment is it (nonequivalent control groups, nonequivalent control group prettest-posttest, nonequivalent control groups interrupted time series, nonequivalent control groups interrupted time series)?  

f) Sketch a graph of the results of the reading study. 

g) In the podcast, they make two caveats:

It’s worth noting that Bronzaft’s subway research, as with similar studies at airports and elsewhere, have some limitations. For one thing, Bronzaft couldn’t randomly assign students to the noisy versus quieter classrooms. There were also relatively few classrooms to choose from, so there might have been some natural variation.

Which of the four big validities is the first caveat about? Which of the four big validities is the second caveat about?  How big of a problem would this be? 

Follow up: Bronzaft conducted and published additional studies, in which, after noise reduction efforts (such as adding acoustic tiles and treating the rails), children on the train-side of the building and children on quiet side of the building were reading at the same level.  

Would you like to read more about the effects of noise on child development and mental health? Visit this resource. There's also an educational module available about the effects of sound in NYC here.

 

 

Too sleepy to choose that major

These students might misattribute their early-morning fatigue to their interest in the subject. Photo credit: Gorodenkoff/Shutterstock

Podcasts have been gaining in popularity in the last few years, and several feature psychological research. I'm going to be devoting my next few posts to some podcasts. We'll start with this very short, 3-minute episode/ interview from the NPR podcast, Hidden Brain. The episode is called "How does the way you feel shape your life?" and it's about how students might be influenced by situational factors (such as how early a class is) when they select their majors. 

The podcast opens with a story:

...behavioral economist Kareem Haggag at Carnegie Mellon University... sometimes teaches early morning classes. And he's worried that when his students are tired or sleepy, they might draw the wrong conclusion about him and about the class.

The researcher himself states (around minute 1)

DR. KAREEM HAGGAG: Our hypothesis was that students who are assigned to an early morning section of the class, or to multiple back-to-back classes before a class, might mix up how tired they are in that class with how much they like the subject, thus leading them to be less likely to choose the subject as their major.

The host, Shankar Vedantam, reports:

This is a psychological phenomenon called misattribution. You know, you go to an amusement park on a sunny day and you think the park is great. You go when it's raining - you think the park is terrible. You're unconsciously confusing your feelings about the weather for the quality of the park.

The researchers used thousands of data points from students at the U.S. Military Academy at West Point to test their hypothesis. As Vedantam points out:

... students at West Point are randomly assigned classes that are scheduled at different times of the day, which means that some students are taking this introductory class first thing in the morning. Some students are taking the very same class later in the day. The researchers then measure the chance that students would later on choose that subject as their college major.

The researchers, led by Dr. Haggag, found that students were more likely to major in the topic of that introductory class if they'd taken it at a later time of the day: 

HAGGAG: We find that students who are randomly assigned to the first period, 7:30 a.m. section, are about 10% less likely to choose the corresponding major compared to a student who takes that class later in the day. 

Questions

a) What are the two main variables of interest in this study (the independent and dependent variables)? What are the levels of each variable?

b) Is this a true experiment or a quasi-experiment? If it's a true experiment, what kind is it (prettest-posttest, posttest-only, repeated measures, concurrent measures)?

If it's a quasi-experiment, which of the four types is it (nonequivalent control groups, nonequivalent control group prettest-posttest, nonequivalent control groups interrupted time series, nonequivalent control groups interrupted time series)?

c) Which is the IV and which is the DV here?

d) Sketch a small bar graph of the results they describe.

e) Can these researchers support the causal claim that "having an early morning introductory class makes you less likely to want to major in that discipline?" Apply the three rules of causation: covariance, temporal precedence, and internal validity in your response.

f) Ask a question about this study's internal, external, statistical, and construct validity.

Bonus content:

In the study above, the main IV was "time of day of the introductory class". As a bonus, here is another independent variable they studied--"number of breaks before the introductory class":

DR. KAREEM HAGGAG: We also compared two students who are sitting in the exact same classroom, but one of whom just had a free period as a break before, and the other came from one or more back-to-back classes. We find that each additional back-to-back class reduces the likelihood that that student enrolls in the major by about 12%.

g) Which is the IV and which is the DV here?

h) Sketch a small bar graph of the results they describe.

i) Can these researchers support the causal claim that "having breaks before your introductory class makes you more likely to want to major in that discipline?" Apply the three rules of causation: covariance, temporal precedence, and internal validity.

 

 

Temperature data as quasi-experiment

Which government-led social distancing measures are effective at stopping the spread of the novel Coronovirus?  One company has contributed data to answering this question. A private health technology company, Kinsa sells a "smart thermometer" which links to an app that guides people through what to do if they have a fever. When people use the app, they agree to have their temperature information (including fever and other symptoms they report) shared with Kinsa. 

Kinsa has used body temperature data points from all over the U.S. to create maps showing areas of "atypical illness." They define atypical illness on their website as "an unusual incidence of elevated flu-like illness levels."

It's really interesting to explore Kinsa's atypical illness maps of the U.S. here. 

More relevant for a research-methods course are the figures that Kinsa presents that map "flu-like illness" rates over time in certain high-population counties (Boston, New York, Los Angeles). I've pasted two of them here. Take a look at these figures, and then answer the questions below. 

Image courtesy of Kinsa, Inc.

 

Image courtesy of Kinsa, Inc

 

a) These figures should remind you of one of the quasi-experimental designs covered in Chapter 13. Which design is it?

b) The Kinsa company has used these charts to argue that social distancing measures work to reduce cases of COVID-19. Journalists (for example) have used the charts to specifically argue that limiting large gatherings and closing schools alone does not do much to stop the spread; it was only when bars and restaurants were shuttered that these counties saw reduced rates of atypical illness. Look carefully at the three counties above. What parts of the data are relevant to this argument? Do you think the data support the claim that closing bars and restaurants was effective? Can you think of alternative explanations for the pattern, other than "closing bars and restaurants"?

c)  Finally, consider the ethics and scientific openness principles. Do you think it's ethical for Kinsa to use and publicize the private health information of people who buy its products? Consider arguments both for and against the scientific ethics of this practice. Frame your arguments around the ethical principles of respect for persons, beneficence, and justice. 

In your reflections, you might consider that users of Kinsa's products agree to data sharing via its terms and conditions. You might consider that the data from users is kept anonymous and is aggregated over more than a million users. You might consider the public health benefits of sharing the data. And you might also consider the extent to which Kinsa is following open science practices--that is, sharing all the data the way most university researchers now do (relevant Kinsa research information here). 

d) If you're construct-validity-curious, you might be wondering if Kinsa's definition of flu-like illness actually tracks COVID-19 cases. Kinsa do not claim to be diagnosing actual COVID (you need a nasal swab for that). However, they present some maps showing how their temperature data has tracked with confirmed COVID cases in several counties. The data are in the middle of this webpage.  

By the way, here's another example of a company that collects our health data, using it to share national patterns. This one is Fitbit, sharing how sleep time has increased (in most age groups) since the COVID quarantine started.

 

A serendipitous study of long-term memory in birds

Here's a photo of a toutouwai (North Island Robin) sporting its bands for identification. Photo: Shutterstock

We don't include many animal behavior studies in ERM blog, but here's a great one about animal memory. How long do you think wild animals can remember new skills they have learned? In this report in The Scientist, we read that the North Island Robin (known in the Maori language as toutouwai) of New Zealand can apparently remember a skill up to 22 months, without any reminders in the meantime. Here's how the study worked. 

[Originally, scientist Rachel Shaw] demonstrated that wild toutouwai living in the Zealandia Wildlife Sanctuary in Wellington, New Zealand, are capable of learning a novel task. Using a rectangular box with three compartments, each covered by a lid that swivels to the side to reveal its contents, she incrementally taught the birds how to peck the lids aside and retrieve mealworms hidden inside two of the three wells.

(By the way, the story in The Scientist includes a photo of the experimental apparatus and a video of a bird using it.)

Roughly a year later, Shaw was giving a public talk at the sanctuary about conservation and her work with birds. As she explained the memory apparatus she had used in her previous research, a toutouwai appeared.

“I haven’t given this bird this task in over a year,” she recalls telling the audience, “so let’s see how he does.” As if on cue, the bird instantly flew down to the apparatus and started opening the lids. That got her wondering: how many of the other birds could repeat the memory task and how long could they remember what they were taught?

This serendipitous event prompted Dr. Shaw to design a study to find out. Before reading on, how would you study this question? 

In the new study, Shaw and her coauthor, Annette Harvey, tested 32 of the initial 64 birds that were trained in 2015 and 2016 and banded for individual identification, 30 of which performed the memory task by spontaneously pecking the lids and opening them on their first try. None of the trained birds had seen the box between the initial training and retesting, though some of the untrained control birds had encountered the apparatus in a previous research project. The time between when the birds had first learned the lid-opening behavior and subsequent testing ranged from 10 to 22 months. By contrast, the 17 untrained birds were unable to complete the task.

The article mentions several other examples of long-term memory in animals, including pigeons, tortoises, nutcrackers, and chickadees. Read more if you are interested!

Questions

a)  This study might be considered a simple observational study but it might also be classified as a quasi-experiment. What do you think?

b) What role does the untrained comparison group play in this research? 

c)  Do you think the effect size in this study would be considered small or large? Explain your answer.

d) Chapter 6 describes important construct validity considerations for observational studies, such as reactivity, observer bias, and observer effects. Which of these might be relevant in this study on toutouwai? 

Thanks to my friend Dr. Stephen Chew for sharing this story on Twitter!

Racial stereotypes behind how we explain school shootings

Participants and newspapers were more likely to blame video games when a school shooter was white than when he was black. Photo: Richard Levine / Alamy Stock Photo

What causes mass shootings? Of course, we have no experimental evidence on this question. Correlational evidence (both within the U.S. and evidence that compares the U.S. to other countries) is consistent with the argument that gun availability is a factor. 

However, this post isn't about what really causes school shootings. It's about what people think causes them. And it turns out that the race of a school shooter influences what people believe about the shooter. 

After some school shootings (a subset of mass shootings), people have sometimes looked for answers not in the availability of guns, but in video games. Now, there is actually no evidence linking video games to mass shootings. However, this ABC news story talks about when people are more likely to think that video games are responsible: 

There's little evidence showing a relationship between violent video games and school shootings, but video games are often blamed for overall violence in our communities. A recent study published in the Psychology of Popular Media Culture set out to find if we look for an external explanation for violence by whites more often than we do for African Americans.

"We try harder to make excuses for white perpetrators," said investigator Dr. James Ivory, a professor in the Department of Communications at Virginia Tech, in an interview with ABC News.

The first study gave people a mock news story about a school shooting. When asked what caused the shooting, people were more likely to blame video games if the shooter was white than if the shooter was black.

It turns out that in the original study (linked here;  if it's paywalled you may be able to access via your university library), participants read the mock news story and were randomly assigned to one of two photos of the perpetrator--a mugshot showing either a Black or a White man. 

a)  There were two variables in this study. Name each one. For each, was it manipulated or measured?

b) What kind of study is this, correlational or experimental?

c) Sketch a graph of the results of this study. What will you put on the y-axis? 

d) In the study above, what role does random assignment play? Why did the researchers randomly assign people to see either a Black or White perpetrator?  

 

Here is how ABC news described the second study:

The second study looked at thousands of news articles -- video games were mentioned more often in stories of white shooters [compared to black shooters]. 

e) What are the variables in the study above? Name each one; for each, was it manipulated or measured?

f) What kind of study is this, correlational or experimental?

g) Sketch a graph of the results of this study.

h) In the study above, the researchers could not randomly assign news article authors to write about a black vs. a white perpetrator. How does that change the impact of this study? 

i) Which of the two studies do you think provides the strongest evidence that race influences people's willingness to think video games are a cause of the shooting?

 

Suggested answers

a) One variable is race of the perpetrator (levels are Black or White), this was manipulated. The other variable is "likelihood to blame video games". This was measured.

b) This was an experiment because one variable was manipulated. 

d) Random assignment contributes to the study's internal validity. They can be sure that the same kinds of participants were in each group (reading about Black vs. White perpetrators). Therefore, people who play a lot of video games themselves were equally likely to be in the two groups. You might be interested to know that in the original study, the researchers also pretested the Black and White photos they used to be sure that the faces differed on race but not on perceived attractiveness or dangerousness. This pretesting also contributed to the experiment's internal validity.

e) The first variable was race of the perpetrator in the article (Black or White), which was measured. The second variable was whether or not the article mentioned video games (yes or no).

f) This was a correlational study because all variables were measured. 

h) Because this study used naturally-occurring crimes, the researchers may have an internal validity problem. It is possible that the Black and White perpetrated shootings differed not just on race but also on number of victims, age of victims, geography, or some other variable. This introduces an alternative explanation for the result--perhaps people blamed video games for White perpetrated shootings because they had more victims, had older victims, and so on.

i) You might have selected the first study as the strongest because it was an experiment and allows us to make a causal inference. Or you might have selected the second study because it was conducted on "real world" stimuli such as news stories--therefore it appears to have good ecological validity or external validity. 

Thanks to Marianne Lloyd for this idea!

Did the show "Thirteen Reasons Why" cause an increase in suicides?

The study used a quasi-experimental design.  Credit: Beth Dubber/Netflix/Kobal/Shutterstock

Update 7/19/2019: Netflix has decided to remove the scene depicting the main character's actual suicide, perhaps in response to data from this and other studies. Here's one story on the change. 

The Netflix series "Thirteen reasons why" tells the story of a teenage girl who commits suicide, leaving behind audio recordings about people who contributed to her decision. When the series was released in April, 2017, mental health and school professionals became concerned about the possibility that young viewers of the show would experience increased risk of suicide themselves. The phenomenon of "suicide contagion" occurs when a publicized suicide leads to other suicide attempts. Young people may be particularly susceptible to its effects. Suicide is the third leading cause of death among American teenagers (ages 12 to 19, according to the CDC). 

This Mashable journalist covers an empirical journal article published in JAMA Psychiatry in 2019.  The Mashable story reports: 

A new study, conducted by different authors, came to a similar conclusion: In the three months after 13 Reasons Why launched, the youth suicide rate for 10- to 19-year-olds rose unexpectedly by 13 percent. Based on historical trends dating back to 1999, 94 more American adolescents and teens died by suicide during the three-month period than otherwise expected. Similarly, the study published last month looked at suicide deaths in the nine months following the show's launch and found an additional 195 fatalities. 

Students and instructors will find it useful to consult the publicly-available JAMA Psychiatry article that contains graphs of this study. Specifically, the solid orange lines in Figure 2 show the suicide rates for 24 months of time: 15 months before the show debuted, and about 9 months after. 

a) This study used a quasi-experimental design. Which of the designs from Chapter 13 was it? (Non-equivalent control group posttest only? Non-equivalent control group pretest-posttest? Interrupted time series design? Non-equivalent control group posttest-only design?)

b) In the quote above, the journalist reported that an 13 percent increase in suicide equals an additional 94 adolescent teen deaths. This figure captures the effect size of the study's result. What do you think of this effect size? Would you call it small or large? 

The empirical journal article also documents the publicity the show received via social media. Here's how the journalist describes the evidence: 

Niederkrotenthaler and his co-authors looked at potential exposure to the show between April 1 to June 30, 2017, when the first season of 13 Reasons Why had just debuted and was most popular on Twitter and Instagram. Using social media data as a proxy for understanding peak interest in the show, the researchers suggest the show's portrayal of suicide created a 'contagion effect' The phenomenon can happen when media coverage of suicide leads to suicide clusters. 

Students and instructors can consult the empirical article's Figure 1 to see the results of the social media analysis. 

b) Why did the authors of the JAMA article include the social media piece? How does this evidence help them make their argument? 

The researchers cannot establish for certain that viewing the show caused the increase in suicide. Here's the Mashable journalist: 

While the researchers couldn't determine whether the children who took their own lives watched the series, an accompanying editorial written by researchers at Harvard and Stanford argues the study provides "strong evidence" that the Netflix show may have led to that increase. 

"One cannot draw definitive causal conclusions from such data, but that no similar increase was seen in suicide rates other than for the age group to which the media portrayal pertained provides some further compelling evidence that the excess suicides may indeed have been owing to the series," the editorial's authors wrote. 

c) The quote above adds an additional detail about the study. The researchers not only tracked the suicide rates of people aged 10-19; they also tracked suicide rates of people in the 20-29 and 30+ age group. The researchers found an increase in suicide after April 2017 only for the teenage group. How does this pattern of results support the argument that the show led to increased teen suicide? 

If you want to talk to someone or are experiencing suicidal thoughts, text the Crisis Text Line at 741-741 or call the National Suicide Prevention Lifeline at 1-800-273-8255. Here is a list of international resources. 

Who's more likely to claim expertise?

The journalist's article was illustrated with a photo of a businessman. Is that the best choice, given the study? Photo credit: Yuri Arcurs/Getty Images

Are you familiar with proper numbers? How about declarative fractions? 

If you claimed that you knew these concepts, you're probably fooling yourself....because declarative fractions and proper numbers are not real math things. 

A Washington Post journalist summarized a study on who is more likely to falsely claim expertise, or as some folks might put it: to "BS." The article's headline reads, "Rich guys are most likely to have no idea what they are talking about, study suggests." 

What did the study do?

Study participants were asked to assess their knowledge of 16 math topics on a five-point scale ranging from “never heard of it” to “know it well, understand the concept.” Crucially, three of those topics were complete fabrications: “proper numbers,” “subjunctive scaling” and “declarative fractions.” Those who said they were knowledgeable about the fictitious topics were categorized as BSers.

The sample for the study was international:

The study drew from the Program for International Student Assessment, which is administered to tens of thousands of 15-year-olds worldwide. The test included a background questionnaire that captures demographic information, along with students’ attitudes toward the subjects they study in school. That section of the test included the questions about math knowledge.

Here is the pattern of results for gender (You can see tables of the data in the journalist's article).  

The data revealed that boys across all nine countries were significantly more likely than girls to pretend expertise, with the difference between the two working out to nearly half a standard deviation in some countries — a big gap, statistically speaking.

And here's the description of the results for income:

There’s also a significant class-based difference, with respondents from the wealthiest households showing a greater proclivity toward overstatement than those from the poorest.  

a) Here's a statistical validity question. When the authors say that the boys are about a half a standard deviation higher than girls in some countries and report it's "a big gap, statistically speaking," they are referring to effect size. In fact, if two groups are half a standard deviation apart, the effect size is probably about d = 0.5. Is that considered a small, medium, or large effect according to Cohen's conventions? 

b) Here's an external validity question. What were the age and income levels of the students in the study? Reflect on the headline, which refers to "rich guys," and the photo accompanying the story, which shows an adult man. What do you think--what external validity issues are raised here? 

c) Here's a construct validity question. The construct the journalist claims is being measured here is the "tendency to BS." How was that construct operationalized in this study? What do you think of that operationalization? Can you think of other ways to operationalize it? 

d) You might think we're ready for an internal validity question. However, internal validity doesn't seem relevant here, because nobody seems to be making a causal claim....or are they? Could you make a causal claim here? 

e) Advanced question: You might wish to investigate the table of data in the article that depicts the "BS gaps between boys and girls." Do you think that this table really needed separate columns for boys, girls, and "gap"? Why was it formatted this way? How else might you present these data? 

f) At the end of the article, the scientists are quoted about the study's implications and limitations. One of the study's original authors says, 

The authors also point out that their study was narrowly defined and restricted to the realm of mathematics. “Ideally,” they wrote, “future research should try to include a greater number of fake constructs in order to maximize precision of the bulls--- scale.”

Which of the four big validities seems to be addressed in this quote? 

Human intelligence over time

When people knew they could look up facts online, they didn't remember them as well.

Here's an article on how intelligence might be changing over time, from Australia's Sydney Morning Herald. It's a great opportunity to practice classifying claims and identifying studies. 

One point the article makes is that humans have been getting more intelligent in general. For example:

For the last century, we’ve been getting smarter - fast. The average IQ of the population has been increasing by about three points per decade.

Even Mensa has noticed. “Our membership is growing at about 10 per cent a year,” says Teresa Wong, the organisation’s recruitment officer. “And about a third of them are under 18.” [...]

The same increase in IQs over the last century has happened in nearly every other developed nation. It’s known as the Flynn effect after James Flynn, the New Zealand researcher who discovered it.

There are several theories for why this has happened: we’re better educated, modern education systems mould our brain in ways IQ tests tend to like, and many of us will do cognitively-taxing things like play video games during our leisure hours.

a. What kind of claim is it to say that "the average IQ of the population has been increasing by about 3 points per decade"? (Frequency, association, or cause?)

 b. What kind of data would you need to support this claim?

The journalist then shifts the focus: 

But something strange and troubling appears to be happening. In several countries – Norway, Denmark, Finland, Britain and France – IQ scores that rose for so long now seem to be dropping, at a rate of about 2.4 points per decade in recent decades.

c. If you were to sketch the average IQ scores of these nations over several decades, what would the graph look like? 

The journalist asks, 

Could it be that our smartphones are making us dumber? We can look up any fact in an instant. We never need to remember facts or directions or friends’ phone numbers. Is that making us more stupid?

d. What kind of claim is the journalist making, above?

e. Below are three studies the journalist describes. Two are probably correlational and one is probably experimental. Which is which?  (and, what are the two variables in each study?):

i. Studies have linked smartphone use with a decreased ability to exert high levels of focus and poorer attention control – although video games seem linked to better multitasking abilities.

ii. In one influential study, volunteers were asked to type newly-learned trivia facts into a computer. Those who were told the computer would remember the facts for them performed much worse on a later memory-recall test. The researchers dubbed it the Google Effect.

iii. In another study, volunteers were given a series of questions that pushed the limits of their brain. Those who said they were heavy smartphone users tended to perform worse and be less analytical. 

f. What kind of data would it take to convince you that the Internet is responsible for the recent declines in IQ around the world? 

You can read one of the original "Google Effect" studies here. 

Does loud music make you eat bad food?

That sounds delicious, thank you! Photo credit: Nathan Motoyama/EyeEm/Getty Images

June is food month on the ERM blog! This story's about the headline, Loud music makes you eat bad food?!?!?Whaaaaaaatever!

QRock 100.7 was one of several news outlets that had fun describing this study for its readers. 

Let's find out what kind of study was conducted to test the claim. The Q100.7 journalist wrote:  

A new study found loud music makes us more likely to order unhealthy food when we’re dining out.  A new study in Sweden found loud music in restaurants makes us more likely to choose unhealthy menu options.  And we’re more likely to go with something healthy like a salad when the music ISN’T so loud.

Researchers went to a café and played music at different decibel levels to see how it affected what people ordered.  Either 55 decibels, which is like background chatter or the hum from a refrigerator . . . or 70 decibels, which is closer to a vacuum cleaner.

And when they cranked it to up 70, people were 20% more likely to order something unhealthy, like a burger and fries.

They did it over the course of several days and kept getting the same results.  So the study seems pretty legit.

a) OK, go: What seems to be the independent variable in this study? What were its levels? How was it operationalized?

b)  What seems to be the dependent variable? How was it operationalized? Think specifically about how they might have operationalized the concept "unhealthy."

c) Do you think this study counts as an experiment or a quasi-experiment? Explain your answer. 

d) This study can be called a "field study" or perhaps a "field experiment". Why?

e) To what extent can this study support the claim that loud music makes you eat bad food? Apply covariance, temporal precedence, and internal validity to your response.

f) If you were manipulating the loudness of the music for a study like this, how might you do so in order to ensure it was the music, and not other restaurant factors, were responsible for the increase in ordering "unhealthy" food?

g) The Q100.7 journalist argues that the study seems "pretty legit." What do you think the journalist meant by this phrase? 

h) The study on food and music volume are summarized in an open-access conference abstract, published here. You might be surprised to read, contrary to the journalist's report, that the field study was conducted on only two days--with one day at 50db and the other at 70 db. How does this change your thoughts about the study? 

g) Conference presentations are not quite the same as peer-reviewed journal publications. Take a moment (and use your PSYCinfo skills) to decide if the authors, Biswas, Lund, and Szocs, have published this work yet in a peer reviewed journal.  Why might journalists choose to cover a story that has only been presented at a conference instead of peer-reviewed? Is this a good practice in general?