Everyday Research Methods

What behaviors make you popular? What behaviors make you likable?

The study investigated two types of behaviors and two types of liking. Photo credit: Rawpixel.com/Shutterstock

Can psychological science make you more popular?  That's what a journalist from Forbes claims in an article, "A science-backed way to up your popularity and friendship." Let's see what the journalist is claiming and see if we buy the evidence. 

The journalist's summary focuses on an empirical article published in Psychological Science (may be paywalled). 

The researchers started with a theory about...

...two types of behavior that can lead to liking during a first encounter:

1. 1. Agentic behavior is usually exhibited when one wants to gain someone’s respect or admiration. Examples include dominant or confident behavior.
   2. Communal behavior, on the other hand, is shown when the goal is to form mutually supportive and trusting bonds with someone. Examples include warm, welcoming, and friendly behavior.

Both agentic and communal behavior have been studied in other social psychological research.

Here's how the study was conducted: 

The study recorded brief, first encounter conversations between two participant groups such that each member of one group interacted with each member of the other.

Specifically, each member of a small group had a 5 minute introductory conversation with each other member of the group. After everybody had talked to everyone, each member of the group rated how much they liked each other person. 

Importantly, the journalist explained that the researchers  were interested in two types of liking: 

1. Popularity, i.e., how much is an individual generally liked by others
2. Unique liking, i.e., how much an individual is particularly liked by a specific person

a) Now that you've got this background, make a prediction in your head. Which kind of behavior--agentic or communal--do you think will be most associated with popularity? Which kind of behavior will be most associated with unique liking? 

Here's how the journalist summarized the study's results:

The study produced two key findings:

1. 1. Showing frequent agentic and communal behavior can lead to high popularity.
   2. When it came to unique liking, exhibiting high levels of communal, but not agentic, behavior led to being particularly well-liked by the interaction partner, more than they liked their other partners as well as more than their other interaction partners liked them on average.

The Forbes journalist doesn't tell us much about how agentic and communal behaviors were measured in this study. Therefore, I turned to the original empirical journal article to find out. It turns out that they had a separate group of raters watch videos of the interactions, rating each participant's behaviors. Here's a passage from the study's Method section:

After the study was complete, independent observers watched the video material and rated the degree to which each participant showed agentic or communal behaviors toward each other group member during the getting-acquainted conversations. (Observers saw videos showing both interaction partners but were told to focus on one of them.)

A rating manual provided behavioral anchors of the four agentic and the four communal behaviors so that raters would have a common understanding of what these behaviors meant. As agentic behaviors, the observers indicated how “leading,” “dominant,” “confident,” and “boastful” a participant’s behavior was. As communal behaviors, the observers indicated how “polite,” “benevolent,” “warm,” and “friendly” a participant was .... Ratings were made on a 6-point Likert scale ranging from 1 (not at all) to 6 (very much).

b) In Chapter 5, you learn about three types of dependent measures: self-report, observational, and physiological. This is a clear example of an observational measure. Which of the three types of reliability is relevant for an observational measure? 

Here is another passage from the empirical journal article: 

The observer ratings demonstrated moderate to good interrater agreement for all eight interpersonal behaviors—the intraclass correlation coefficients(2, k) ranged from .66 to .84. 

c) Imagine there are only two coders....could you graph the result of the passage above, following models in Chapter 5? Pick one of the ratings to focus on. 

d) This was a correlational study, not an experimental one, which means that the journalist probably jumped the gun when he suggested "Showing frequent agentic and communal behavior can lead to high popularity."  How might you change this study be experimental? Specifically which variables would you have to manipulate? Which would you measure? 

Personality correlates of liking spicy foods

Photo credit: marcoscastillo/Deposit photos

Here's another article from the content farm website, StudyFinds.  This one's on the personality correlates of liking spicy foods, and it includes a combination of frequency and association claims. The same poll was also covered by the NYPost and by Yahoo! News (with a video)

Below I will post a series of quotes and questions from the StudyFinds version of the story. 

Ninety-three percent of respondents prefer some level of heat in their food. However, nearly four in 10 (36%) say they go for medium heat, 33 percent go for mild, and just 24 percent like their spice level set to hot.

a) Based on this statement, there seem to be four levels of the "Liking Spicy Foods" variable. What are they? 

The methodology of the survey is not always clear from the press stories. Here is one example:

Spicy food eaters are more likely to enjoy trying new things (76%), consider themselves attractive (62%), and are more content with their lives (66%) than those with milder heat preferences.

b) In the quote above, what is your guess about who counts as a "spicy food eater"? That is, which answer would a person have to give to the "Liking Spicy Foods" variable you listed in a) in order to be counted as a "spicy food eater"?

In the quote above, another problem is they only report the percentages for the "spicy food eaters". We'd also need to know the corresponding percentages for "those with milder heat preferences."

c) What would these corresponding percentages be able to tell us about effect size? Propose some alternative outcomes for the variable "trying new things" that would show a large effect size vs. a small one. 

Here is a quote from later in the story:

Almost one-third (32%) of spicy food fanatics follow a certain diet like vegan or vegetarian— more than double than no spice fans (13%) and more than any other spice preference.

It's great that this quote at least tells us the percentage in each group that are vegan/vegetarian. However, it still does not tell us how they operationalized being a "spicy food fanatic" vs a "no spice fan."  

d) Here's another quote from the journalist's story. Based on what you answered so far (in previous questions), what is good and bad about the coverage in this quote? 

Of those who prefer hot foods, 21 percent consider themselves extroverted, compared to 15 percent of mild food eaters. Those who pack on the heat are also more likely to describe themselves as creative (54%), confident (51%), and adventurous (44%).

Another consideration here is the cultural background of the participants. One of my colleagues grew up in India, where children often learn to eat spicy foods from an early age.

e) How might culture moderate the relationships reported here, for example, how might it moderate the relationship between liking spicy food and being extroverted? 

f) This study was conducted by a commercial polling firm, and this particular study was commissioned by the brand Frank's RedHot. Do you think this commercial funding should affect our interpretation of the data? Why or why not? 

Selected Answers:

a) The variable "Liking Spicy Foods" has four levels: No heat, mild heat preference, medium heat preference, and hot heat preference.

b) It's not clear--but it's possible that they counted anybody who picked mild, medium, or hot (that is, 93% of the sample) as a "spicy food eater"?

c) Here's an example. What if 76% of spicy food eaters like trying new things but 70% of non-spicy food eaters like trying new things? That's not a very big effect size! But if 76% of spicy food eaters like trying new things but 20% of non-spicy food eaters like trying new things, that would be a very large effect size. 

d) This quote has the same problem of not defining what a spicy food eater is. But at least they give the percentages of extroverts for two different groups.

e) If a culture serves a lot of spicy foods and even children learn to enjoy them, then it's possible that "liking spicy food" will not be strongly correlated with personality (there's a ceiling effect on the spicy food variable). The personality correlations may exist only for cultures where spicy foods are more variable.

 

 

 

 

 

 

 

Taking Poli Sci predicts likelihood of voting

After matching students based on their pre-college voting behaviors, the researchers found that students who had taken a political science class became more likely to register to vote. Photo credit: Lakshmiprasad/Depositphotos

In the online news outlet called The Conversation, researchers write summaries of their own research studies for a general audience. 

Here's an example from political scientists Frank Fernandez and Matthew Capaldi, both of the University of Florida. They introduce their study by writing,

...improving college student voter turnout is a national issue....According to data from the National Study of Learning, Voting, and Engagement, about 1 in 4 students – including at both two- and four-year colleges – were not registered to vote in the 2016 or 2018 elections.

The researchers conducted a study of 2,000 students at community colleges. All the students had completed an extensive survey about their civic behaviors both before and during college. Students also reported the classes they had taken in college.

The researchers used a "propensity score matching procedure", which is similar in some ways to multiple regression. Specifically, they matched up students who had the same demographic characteristics (e.g., race, parental education, family income, gender). They also matched students on whether, before going to college, they had been registered to vote or voted in elections. The researchers were mainly predicting whether students had newly registered to vote (since starting college) as well as whether they had actually voted since starting college. The authors write about their results: 

After taking students’ prior civic engagement and other college experiences into account, we found that students who took at least one political science course were 9% more likely to register to vote than those who did not.

Additionally, we found that students who took at least one political science class were 8% more likely to vote.

Questions

a) Name the variables in this study--there are at least seven variables mentioned in the summary provided above.

b) Is this a correlational study or an experiment? Explain your answer.

c) The authors imply that taking a political science course seems to cause students to be more likely to register to vote and to actually vote. Let's apply the three criteria for causation to this statement. First, do the results support covariance? 

d) Second, does the method of the study clearly establish temporal precedence? Why or why not? 

e) Finally, the authors used the matching technique to control for several alternative explanations such as race, parental education, family income, gender, as well as whether, before going to college, they had been registered to vote or had voted in elections. Can you think of any additional alternative explanations that they did not control for? 

f) In their summary in The Conversation, the authors discuss some of the downsides of their methodology. For example, they write, "We relied on self-reported data, so there is no practical way to confirm that they registered to vote or turned out to vote."  This critique is focused on construct validity--specifically, criterion validity. That is, they are admitting that they don't know whether or not a self-report item about voting actually predicts the behavioral criterion of voting. How could you run a pilot study to confirm that people who say they voted in a certain election have actually voted (and vice versa--people who say they did not vote have actually not voted)?

Here's a link to their study , which was published in Educational Researcher (sorry--paywalled).

Suggested answers:

a) The variables include race, parental education, family income, gender, being registered to vote before college, voting before college, and registering to vote during college and voting in college.

b) It's a correlational study--the survey method simply measured all of the variables, and a study where all the variables are measured is correlational. 

c) Yes, the results support covariance because students who took a poli sci course were 9% more likely to have registered to vote in college (and 8% more likely to actually vote). 

d) In my opinion, temporal precedence is not established here--students reported on whether they took a poli sci course at the same time they reported on whether they had registered to vote in college. It's not clear from this whether the course came first in time, before the voting behavior.

e) The authors did control for several likely explanations, including baseline interest in voting--and that is good. Another variable they might consider could be local political climate--perhaps some community colleges are more politically active than others. Students who attend more politically active CCs could be more likely to start to vote, and also be more likely to take a poli sci course. 

f) You could recruit a sample of people and ask them if they voted in the last election. Then, you can use publicly available voting records to confirm whether or not the actually voted. You should see a strong association:  People who say they voted, should have actually voted, and people who say they did not vote, should have actually not voted. 

Kids who eat more fruits, vegetables have better mental health

Which alternative explanations for the relationship between diet and mental health does this study allow us to rule out? 
Photo credit: ArturVerkhovetskiy/Deposit Photos

Here's a simple headline to work with: "Kids who eat more fruits, vegetables have better mental health". The story appears on a mega-site called studyfinds.org, which provides multiple summaries of science research. 

a) What are the variables in this headline?

b) Does the headline make a frequency, association, or causal claim (and why?)

c) Before reading on, do you think that the variables in this study were measured or manipulated? 

Now here are some details about the study:

“While the links between nutrition and physical health are well understood, until now, not much has been known about whether nutrition plays a part in children’s emotional well-being,” says says lead researcher Ailsa Welch ... “So, we set out to investigate the association between dietary choices and mental wellbeing among schoolchildren.”

Researchers studied data from more than 8,500 children at 50 schools in England. 

d) Based on this description, can you tell how the "diet" and
"well being" variables were probably operationalized--Self-report? Observational? or Physiological? (If not, what would you guess?)

Here are some details of the results:

The team looked at the association between nutritional factors and mental well-being and took into account other factors that might have an impact, such as adverse childhood experiences and home situations. “We found that eating well was associated with better mental well-being in children. Among secondary school children, in particular, there was a really strong link between eating a nutritious diet, packed with fruit and vegetables, and having better mental well-being,” explains co-author Dr. Richard Hayhoe....

e) Start with this quote: "there was a really strong link between eating a nutritious diet, packed with fruit and vegetables, and having better mental well-being,” Sketch a scatterplot of this relationship, labeling your axes mindfully. 

f) Now consider this statement: The team looked at the association between nutritional factors and mental well-being and took into account other factors that might have an impact, such as adverse childhood experiences and home situations. The phrase "took into account" should alert you that regression was used. In this regression, what was the criterion (dependent) variable? What were the predictor (independent) variables? (hint: there were at least three predictor variables).

g) Suppose someone wanted to claim, based on this study, that eating fruits and vegetables causes better mental well-being. What are some internal validity problems with that claim? Come up with one or two possible C-variables that could be responsible for this relationship, and explain how your C variable goes with both A (eating fruits and veg) and B (higher mental well-being).

h) When the researcher reported that "among secondary school children, in particular, there was a really strong link between eating a nutritious diet..and having better mental well-being," he is suggesting a moderator. The term, "in particular," suggests that the relationship between diet and well-being was stronger for secondary school children than it was for younger children. Sketch what this moderator would look like, either using two scatterplots (one for each age subgroup, kind of like Figure 8.18) or making a table, similar to Table 8.5. 

i) If you like, think about mediators. Let's say that the researchers have a theory about why eating more fruit and veg is associated with better mental well-being. What might mediate this relationship? That is, why might eating fruit/veg lead to better mental well-being? Think of a mediator and draw a small diagram of it here, like Figure 9.12. 

(Please keep in mind that a mediator relationship is a causal relationship, and that can't really be supported by this correlational study. However, that does not stop us from proposing some potential mediators that we can test in a future study.)

 

 

 

 

 

 

 

People believe claims of sexual harassment more when women are attractive

The coded stereotypicality of the drawing was a dependent variable in one of the studies. Photo: Courtesy of Dr. Jin Goh. Citation: Goh, J. X., Bandt-Law, B., Cheek, N. N., Sinclair, S., & Kaiser, C. R. (2021). Narrow prototypes and neglected victims: Understanding perceptions of sexual harassment. Journal of Personality and Social Psychology. Advance online publication. https://doi.org/10.1037/pspi0000260

Sexual harassment (unwelcome and inappropriate sexual remarks or physical contact) harms people in schools and workplaces. One potential problem with sexual harassment is whether people who report it are taken seriously. Unfortunately, when people report sexual harassment, their claims are not always believed, which means perpetrators can go uninvestigated and victims' harassment might continue. 

The press recently covered an important series of studies that investigated the role that physical appearance plays in the reporting (and believing) process. One of the summaries appeared in the Seattle Times; another  appeared on the APA website.  The Seattle Times journalist summed up the studies like this: 

The study found people are more apt to believe sexual harassment claims by women who are young, “conventionally attractive,” and appear and act feminine. Women who don’t fit that prototype not only are less likely to be believed, but also are presumed to be unharmed by harassing behavior, the study said. 

The APA website does a better job detailing the methods:

The researchers conducted a series of 11 multimethod experiments, involving more than 4,000 total participants, designed to investigate the effect a victim’s fit to the concept of a typical woman had on participants’ view of sexual harassment and the consequences of that mental association.

In five of the experiments, participants read scenarios in which women either did or did not experience sexual harassment. Participants then assessed the extent to which these women fit with the idealized image of women, either by drawing what they thought the woman might look like or selecting from a series of photos. Across all the experiments, participants perceived the targets of sexual harassment as more stereotypical than those who did not experience harassment.

The details above describe a set of experiments (do you know why they are experiments?)

a) What was the main independent variable (IV) in the five studies above? What were its levels? 

b) What was the main dependent variable (DV) in the five studies? What do you think its levels were? 

c) The image (with the two drawings, above) that goes with this post came from the first study (Study 1), in which participants read a scenario in which " women either did or did not experience sexual harassment" and made a "drawing what they thought the woman might look like". Which drawing do you think came from each condition? 

d) The drawings from Study 1 had to be coded for stereotypical femininity. Since this variable required coding, inter-rater reliability would have been important. How do you think the researchers established inter-rater reliability for their coders? (Hint: you can find relevant information in Chapter 5.)

e) Sketch a small bar graph depicting the results of these studies, based on the journalist's description. 

Now here's some more description from the APA website:

In the next four experiments, participants [read] ambiguous sexual harassment scenarios, such as a boss inquiring about a woman’s dating life. These scenarios were paired with descriptions or photos of women who were either stereotypical or not. The participants then rated the likelihood that the incident constituted sexual harassment.

f) What was the main independent variable (IV) in these four studies? What were its levels? 

g) What was the main dependent variable (DV) in the four studies? What do its levels seem to have been? 

h) Sketch a small bar graph depicting the results based on the journalist's description. 

i) Assess the plausibility of this claim: "Being more stereotypically feminine causes people to think a woman's sexual harassment claim is true"  Apply the three causal criteria (covariance, temporal precedence, and internal validity). 

Instructors: Students are likely to enjoy reading the studies in the original journal article, which appears on my list of journal articles that are student-friendly.

Frequency claims: Fitter and better rested during lockdown

One of the positive changes people reported was being able to spend more time with family. Photo: Steve Skjold / Alamy Stock Photo

The first peer-reviewed studies about the COVID-19 pandemic  are starting to appear in journals, and journalists are starting to summarize them in the popular press.

Let's explore this summary of a survey study conducted in Scotland and summarized for a general audience by the scientist herself, in The Conversation. The scientist writes, 

 ... for some people lockdowns have provided an unexpected opportunity to make positive changes to their lives...

Here are some details: 

In May 2020, my colleagues and I surveyed over 3,000 people in Scotland to find out what positive changes people had made in their lives during the lockdown period. 

And some of the key results:

More than half the people we surveyed reported these changes for the better: being more appreciative of things usually taken for granted (reported by 83% of participants), having more time to do enjoyable things (by 67%), spending more time in nature or outdoors (by 65%), paying more attention to personal health (by 62%), doing more physical activity (by 54%) and spending more time with a partner or spouse (by 53%).

Each of the above results constitutes a frequency claim, because it talks about the level, or percentage, associated with one variable. Consider this one, for example: "doing more physical activity (by 54%)". The variable here is "doing more physical activity under the lockdown (or not)."

Now let's use the four big validities to systematically interrogate this frequency claim, "54% of people in this Scottish survey reported doing more physical activity during the lockdown." 

a) Construct validity: How do you think they measured the variable "doing more physical activity"? (Self-report? Observational? Physiological?) . What are your initial impressions of, or questions about, how well this variable was measured? 

b) Construct validity: Let's say you wanted to establish the criterion validity (See Ch 5) of the (presumably self-report) measure of "doing more physical activity".  What behavior should self-reports of this variable correlate with, if the self-report measure is valid? 

c) Statistical validity: The point estimate, 54%, is one element of statistical validity. What about the precision of this estimate? That is, what is the 95% CI, or margin of error of this estimate? It's not provided in the article, but you can visit this calculator to estimate the CI. Do that now. 

d) Statistical validity:  Replication is another aspect of statistical validity. The scientist-author cites at least one more, similar study that came to a similar conclusion: 

An Australian study (still in preprint, meaning its findings are yet to be reviewed by other scientists) also sought to find out similar information. In a survey of over 1,000 people, it found that 70% of participants reported having experienced at least one positive effect of the pandemic. 

e) External validity:  The sample for this survey was N =3000. Don't get distracted by the size of the sample (remember--it's not the sample size that helps establish external validity). Instead, what do you need to ask in order to decide whether the study's sample can generalize to their population of interest (Consult Ch 7)?

f) External validity: I visited the peer-reviewed article (here--open access) and found this information about their sampling technique.

The target population of the survey was adults, aged 18 years or older, currently residing in Scotland, who were interested in sharing their experience of positive change. Participants were primarily recruited through social media advertisements on Facebook and Twitter which directed participants to the online survey on Qualtrics. 

What kind of sampling did they apparently use in this study? (Name a sampling technique that is discussed in Chapter 7). 

What is their population of interest?

Does the sampling technique they used allow them to generalize from their sample to their population of interest? Why or why not? 

g) Internal validity: In a frequency claim, nobody is making a causal statement, so we don't actually need to interrogate internal validity.

h) Now that you've looked at all four validities, what do you think about the quality of this study? Make a nuanced comment about its strengths and weaknesses. 

 

P.S. Thanks to Marianne Lloyd for the reference!

Teaching with 100 Humans

In 2020, Netflix released an 8-episode series called "100 Humans" in which 100 (American) humans from a variety of ages and social groups participate in a series of studies. It's a gold mine of examples for critical thinking in research methods. You'll find examples of posttest-only, pretest-posttest designs, HARKing, ethics, and more. Most of the studies have flaws, but that makes them more fun to critique. 

I've cataloged ways you can use the series in your teaching: www.tinyurl.com/teachingwith100humans.

Reach out if you use it! Let me know how it works for you. 

 

Temperature data as quasi-experiment

Which government-led social distancing measures are effective at stopping the spread of the novel Coronovirus?  One company has contributed data to answering this question. A private health technology company, Kinsa sells a "smart thermometer" which links to an app that guides people through what to do if they have a fever. When people use the app, they agree to have their temperature information (including fever and other symptoms they report) shared with Kinsa. 

Kinsa has used body temperature data points from all over the U.S. to create maps showing areas of "atypical illness." They define atypical illness on their website as "an unusual incidence of elevated flu-like illness levels."

It's really interesting to explore Kinsa's atypical illness maps of the U.S. here. 

More relevant for a research-methods course are the figures that Kinsa presents that map "flu-like illness" rates over time in certain high-population counties (Boston, New York, Los Angeles). I've pasted two of them here. Take a look at these figures, and then answer the questions below. 

Image courtesy of Kinsa, Inc.

 

Image courtesy of Kinsa, Inc

 

a) These figures should remind you of one of the quasi-experimental designs covered in Chapter 13. Which design is it?

b) The Kinsa company has used these charts to argue that social distancing measures work to reduce cases of COVID-19. Journalists (for example) have used the charts to specifically argue that limiting large gatherings and closing schools alone does not do much to stop the spread; it was only when bars and restaurants were shuttered that these counties saw reduced rates of atypical illness. Look carefully at the three counties above. What parts of the data are relevant to this argument? Do you think the data support the claim that closing bars and restaurants was effective? Can you think of alternative explanations for the pattern, other than "closing bars and restaurants"?

c)  Finally, consider the ethics and scientific openness principles. Do you think it's ethical for Kinsa to use and publicize the private health information of people who buy its products? Consider arguments both for and against the scientific ethics of this practice. Frame your arguments around the ethical principles of respect for persons, beneficence, and justice. 

In your reflections, you might consider that users of Kinsa's products agree to data sharing via its terms and conditions. You might consider that the data from users is kept anonymous and is aggregated over more than a million users. You might consider the public health benefits of sharing the data. And you might also consider the extent to which Kinsa is following open science practices--that is, sharing all the data the way most university researchers now do (relevant Kinsa research information here). 

d) If you're construct-validity-curious, you might be wondering if Kinsa's definition of flu-like illness actually tracks COVID-19 cases. Kinsa do not claim to be diagnosing actual COVID (you need a nasal swab for that). However, they present some maps showing how their temperature data has tracked with confirmed COVID cases in several counties. The data are in the middle of this webpage.  

By the way, here's another example of a company that collects our health data, using it to share national patterns. This one is Fitbit, sharing how sleep time has increased (in most age groups) since the COVID quarantine started.

 

Screen time (again!)

The effect of screen time on well-being is likely to depend on moderators such as what's on the screen, when it's used, and who's using it.
Photo: Eric Nathan / Alamy Stock Photo

I can't promise this is the last article on screen time research that I will blog about. But this long-form piece by journalist Kim Tingley was irresistable because it illustrates several of the concepts in the Research Methods course. (Instructors--it would make a nice final exam review.)

The questions here are set up as an end-of-semester scavenger hunt. After you read each question, read a bit further down in the article and find the quote or paragraph section that fits that question. 

Questions:

a) There are two frequency claims that open this piece. What are they? 

b) What's the first association claim presented in this article? What are the variables in the claim? 

c) The second association claim contradicts the first one. What is the second association claim and how does it differ? 

d) Reading on, you'll find some information relevant to the external validity of these studies. What is the relevant information? 

e) The journalist makes a claim about self-report data. What is it? What concept in Chapter 5 is being discussed here?

f) The journalist does a quick job of covering the three criteria for causation next. Which passage addresses that topic?

g)  Next the journalist walks us through the potential for p-hacking in the Monitoring the Future (M.T.F.) survey. Which paragraphs address this?

h) Next look for a discussion of effect size (and especially effect size in context) in these data. 

i) What about moderators? The article has those too. Which passages address moderators? Identify the core relationship of interest (the two variables) and identify at least one moderator. You can make a sentence of this format: _______ might moderate the relationship between _____ and ______.

j) Finally, ethics. Which passage addresses this issue in screen time? 

 

Suggested Answers

a) The two frequency claims are "95 percent of teenagers had access to a smartphone, and 45 percent said they were online 'almost constantly.'"

b) "the longer adolescents were engaged with screens, the greater their likelihood of having symptoms of depression or of attempting suicide."  The variables are Time engaged with screens  and Likelihood of depression or suicide attempts"

c) The second claim is about  "a tenuous relationship between adolescent well-being and the use of digital technology". 

d) The journalist mentions that epidemiological surveys involve "conducting phone interviews with thousands of randomly selected people," which would enhance external validity. 

e) She mentions "For starters, people are notoriously bad at self-reporting how often they do something or how they feel." This is probably about criterion validity. People's self reports (for example, of "how often they use their phones" may not correlate with their actual behavioral phone use.  (For the record, I disagree with the journalist's point that people are bad at self-reporting how they feel. How else will you find out about a person's subjective state?)

f) She writes, "there’s no way to say if the technology use caused their low mood or vice versa" (that's temporal precedence) and "or if other factors were involved" (That's internal validity.)

g) The paragraphs that start "A researcher must decide..." and "To show how many legitimate outcomes..." provide excellent overviews of the potential for unintentional p-hacking.

h) The journalist is discussing effect size when she describes "smoking marijuana and being bullied were more closely linked with decreased well-being than tech use was" and "the strength of the association screen time had with well-being was similar to neutral factors like wearing glasses or regularly eating potatoes."

i) The phrase "depends on" is a good signal that there's a moderator.  Your sentence might be:
The type of person (perhaps good student vs. poor student)  might moderate the relationship between screen time and well-being. 

j) Ethics come into play when we ask whether companies should be collecting data on our screen use, and if they are doing so, whether they have our permission and if they should share that data with researchers. 

 

Which Ice Cream Flavor Are You?

Many flavors, many personalities: How could we test whether an ice cream personality test is reliable and valid? Photo credit: Okanmetin/Getty Images

At the height of summer, we all start considering the same, timeless question: Can your personality can be accurately summarized by an ice cream flavor? This Buzzfeed poll asks you to answer a series of summer-activity-related questions and then delivers a verdict. Go take it yourself! I'll wait. 

The first time I took the poll, I answered the questions seriously and was told that I am characterized as "Coffee," along with this description: 

You’re a go-getter, a hard worker who doesn’t pause to reflect on what you have accomplished in the past because you are so focused on what you want to accomplish in the future. You seek to inspire the people around you and help to propel them to greater achievements. At the same time, you are approachable and kind rather than snotty and entitled. It’s basically the best of all worlds.

I sure felt good reading that personality description! The second time I took the quiz, I clicked on questions randomly haphazardly, just to see what other feedback there might be. This time I got "Chocolate", with this description:  

Mmm, you’ve got a truly rich and decadent side that you aren’t afraid to share. You make people feel good — really good. You’ve got a knack for excelling in social situations and know just how to be a crowd-pleaser. Keep doing your thing and living that luscious life.

Of course, nobody is going to mistake a Buzzfeed poll for a legitimate personality measure. But it does provide a fun way to consider issues raised in Chapters 5 and 6. Take the quiz yourself and then prepare these questions:

 

1.     What flavor ice cream did you get? What did the results say about you?

 

2.     Do you think this quiz is reliable? What kind of reliability is relevant for this measure? How would you empirically establish the quiz's reliability? Here's a hint: You'd start with a large sample of people and have them take this ice cream quiz, and then..... (what?)

3.  Now think about the measurement validity of this quiz. I'll narrow down the question for you here: Are people who score in the Coffee category actually more likely to be "hardworking," "inspiring," and "approachable," as the description suggests? How would you establish the criterion validity of this quiz? Here's a hint: You'd start with a large sample of people and have them take the ice cream quiz. You'd separate them into people who got the Coffee category and those who did not get Coffee. How should the Coffees be different from the non-coffees?

 

4.     What might we change about the quiz if we wanted to study ice cream preferences in a scientific way? 

 

5.  Read this description on the Barnum effect. Reflect on how the Barnum effect might be at play for people who take this Buzzfeed quiz. 

 

Many thanks for this idea to Josh Maxwell of the University of New Mexico!

 

Suggested answers to selected questions.

2. Test-retest reliability is the most relevant. To establish it, you'd have a large sample of people take the Buzzfeed quiz at one time point and record everyone's category (e.g., Coffee, Chocolate, etc.) Then you'd track down the same group of people again some time later (maybe 4 weeks later) and have them take the same quiz again. People who were scored Coffee at Time 1 should also score Coffee at Time 2, and people who were scored as Chocolate at Time 1 should also score as Chocolate at Time 2. (You'd compute the correlation between these two category assignments using Cohen's Kappa.)

3. To establish this criterion validity, you'd recruit a large sample of people and have them take the quiz, recording those who got Coffee and those who did not. Then you'd need to measure variables such as "hardworking" "inspiring" and "approachable." I'm not sure how to measure "inspiring" or "hardworking," but you could probably measure "approachable" using the Big Five trait of Agreeableness (here's one measure of that). If the Buzzfeed poll is valid, then a simple bar graph (and an independent groups t-test) should show that people who are assigned Coffee should score higher on agreeableness than people who were assigned some other flavor.