Everyday Research Methods

What ends up on the Yelp Top 100 list?

What construct and external validity considerations affect Yelp ratings, like this 4-star review of a restaurant near my campus? Screenshot from Yelp website (by author)

In this interesting piece, Slate writer Will Oremus asks why the top-rated restaurants on Yelp are places that "nobody has ever heard of."  He explains that the top 10 rated restaruants on Yelp change from year to year, and furthermore, they tend to be places like Copper Top BBQ of CA and Art of Flavors of Las Vegas.  The top 100 are not big, famous restaurants--in fact, they tend to be simple, local, even touristy places with "styrofoam and paper plates."

Oremus points out that Yelp ratings are subject to "biases are quite different from the ones we’re used to" in other ratings websites.  Are these biases of the external validity variety? Or the construct validity variety?  I'll quote some passages from Oremus' Slate article for your consideration.

Here's Oremus's first observation:

I have not eaten at Copper Top, but I have little doubt that these restaurants also share a consistently high quality of food for the money. On Yelp, that’s usually a recipe for a four-star rating. Compared to professional critics, Yelp reviewers skew young and budget-conscious, which is part of the site’s appeal. By and large, they’re happier paying $8 for a very good burrito than $23 for a fancy one, and the ratings reflect that.

a.  What kind of validity is this conclusion directed at--external or construct? Can you say anything specific about this type of validity, as used by this journalist? 

Here's another observation from Oremus' article.

Part of the explanation lies in the distribution of ratings on the site’s five-star scale. Only a handful or restaurants in the world rate three Michelin stars. But more than 40 percent of all Yelp reviews are perfect scores, suggesting that five stars on Yelp entails satisfaction rather than perfection. Average hundreds of reviews of the same establishment, and you’ll find that its overall rating is influenced far more by the number of dissatisfied customers than by how much the five-star reviewers loved it. The best-rated restaurants on Yelp, then, are not so much the most loved as the least hated.

b.  What kind of validity is this conclusion directed at--external or construct? Can you say anything specific about this type of validity, as used by this journalist?

Oremus also points out the power of incidental influences, such as neighborhood and weather, on Yelp reviews. To wit:

...researchers at Georgia Tech and Yahoo Labs found that online restaurant reviews are significantly influenced by at least three factors that have nothing to do with the operation of the business:

• Neighborhood demographics: Restaurants in neighborhoods with high education levels don’t get better reviews, but they do get more reviews. That matters, because Yelp’s top-100 rankings are based not only on average ratings, ...so a place with 100 five-star reviews will rank higher than one with 50.
• Time of year: Restaurants get more reviews in July and August than they do in the winter, but the average ratings in the summer months are lower.
• Weather: One of the strongest exogenous effects on restaurant ratings, according to the study, is the weather at the time of the review. As you might guess, warm temperatures and sunshine mean higher reviews. Cold temperatures or extreme heat mean lower reviews, as does precipitation of any kind. The researchers attribute this to weather’s well-documented effects on mood and memory.

c. What kind of validity is the above research concerned with?

d.  How might this information affect your own use of Yelp in the future?  Think of two possible ways you can use this information.

 

Suggested Answers

a. When Oremus writes that Yelp's users tend to be younger and budget-conscious, he's describing how the sample of people who choose to post on Yelp is biased. This is an external validity point. One could say that the sample of Yelp reviews is self-selected, and is biased toward restaurants that are cheaper.  Cheaper restaurants may get more reviews on Yelp than more expensive ones; in addition, cheaper restaurants may be rated more positively, all because of this sampling bias.  Therefore, ratings on Yelp might not generalize to how older people would evaluate the same restaurants. The situation seems similar to conducting an opinion poll and including (or not) cell-phone only households.

b. This is a construct validity point. People tend to use the 5-star end of the Yelp scale the most, he says. This means that people have a "yea saying" bias on Yelp reviews (to use a Chapter 6 term). As a result, it might be hard to decide if a positive review on Yelp is truly good, or if people just tend to like everything! 

Another point Oremus made is that a high Yelp rating means the restaurant is more consistent--not necessarily more delicious. That again is a construct validity point.

c. The weather bias suggests problems with construct validity. We might not know if a positive rating reflects the quality of the restaurant (the construct in question) or the type of weather outside!  However, the "neighborhood bias" seems to be an external validity issue--restaurants in highly educated neighborhoods get more reviews, so this is a sampling bias. 

 d.  Answers will vary. All in all, it seems that these construct and external validity issues mean that Yelp reviews are unlikely to parallel what a professional restaurant critic would say. Does that affect your restaurant behavior, or not?  

 

Thanks again to Carrie Smith of Ole Miss, who, as usual, is a fount of bloggable Slate pieces!

 

 

 

 

Resource: Recent articles that are student-accessible: Updated August 2017

Hi! I've updated and reposted this list. Check here. 

A list of recent articles that are student-accessible 

Compiled by Beth Morling and Jeong-Min Lee, University of Delaware

These articles primarily come from Psychological Science or Personality and Social Psychological Science. Our students have used these articles in their final projects for Research Methods in the past. In this final project, students summarize and analyze the validities for two empirical studies. (I can provide the assignment upon request: [email protected].)

Most of these articles report more than one study, and at least one of the studies is simple enough for a student with basic skills in research methods to understand. The most complex experimental design would be a 3-way factorial design. The most complex correlational design uses multiple regression and (sometimes) mediation.

 

Brummelman, E., Thomaes, S., de Castro, B. O., Overbeek, G., & Bushman, B. J. (2014). “That’s not just beautiful—that’s incredibly beautiful!”: The adverse impact of inflated praise on children with low self-esteem. Psychological Science, 25, 728-735. doi:http://dx.doi.org/10.1177/0956797613514251

Bryan, C.J., Yeager, D.S., Hinojosa, C.P., Chabot, A., Bergen, H., Kawamura, M., & Steubing, F. (2016). Harnessing adolescent values to motivate healthier eating. PNAS, 113, 10830-10835. doi: 10.1073/pnas.1604586113

 

Cantú, S. M., Simpson, J. A., Griskevicius, V., Weisberg, Y. J., Durante, K. M., & Beal, D. J. (2014). Fertile and selectively flirty: Women’s behavior toward men changes across the ovulatory cycle. Psychological Science, 25, 431-438. Retrieved from http://search.proquest.com/docview/1501839110?accountid=10457

Chan, K. Q., Tong, E. M. W., & Tan, Y. L. (2014). Taking a leap of faith: Reminders of God lead to greater risk taking. Social Psychological and Personality Science, 5, 901-909. doi: 10.1177/1948550614537309

 Fraley, R.C., Griffin, B.N., Belsky, J., & Roisman, G.I. (2012). Developmental antecedents of political ideology: A longitudinal investigation from birth to age 18 years. Psychological Science, 23, 1425-1431. doi: 10.1177/0956797612440102

 Frenda, S., Patihis, L., Loftus, E., Lewis, H., Fenn, K.  (2014) Sleep deprivation and false memories. Psychological Science, 25, 1674-1681. doi: 10.1177/0956797614534694

 Gino, F., & Wiltermuth, S. S. (2014). Evil genius? How dishonesty can lead to greater creativity. Psychological Science, 25, 973-981.doi: 10.1177/0956797614520714

Gino, F., Norton, M. I., & Ariely, D. (2010). The counterfeit self: The deceptive costs of faking it. Psychological Science, 21, 712-720. doi:http://dx.doi.org/10.1177/0956797610366545

 House, J., DeVoe, S. E., & Zhong, C. (2014). Too impatient to smell the roses: Exposure to fast food impedes happiness. Social Psychological and Personality Science, 5, 534-541. doi:http://dx.doi.org/10.1177/1948550613511498 [use study 3]

Jirout, J. J., & Newcombe, N. S. (2015). Building blocks for developing spatial skills: Evidence from a large, representative U.S. sample. Psychological Science, 26, 302-310. doi:http://dx.doi.org/10.1177/0956797614563338

 Kruse, E., Chancellor, J., Ruberton, P. M., & Lyubomirsky, S. (2014). An upward spiral between gratitude and humility. Social Psychological and Personality Science, 5, 805-814. doi:http://dx.doi.org/10.1177/1948550614534700 [Study 2]

Lee, K., Talwar, V., McCarthy, A., Ross, I., Evans, A., & Arruda, C. (2014). Can classic moral stories promote honesty in children? Psychological Science, 25, 1630-1636. doi:10.1177/0956797614536401

Levy, K. N., & Kelly, K. M. (2010). Sex differences in jealousy: A contribution from attachment theory. Psychological Science, 21, 168-173. doi:http://dx.doi.org/10.1177/0956797609357708

 Mrazek, M. D., Franklin, M. S., Phillips, D. T., Baird, B., & Schooler, J. W. (2009). Mindfulness training improves working memory capacity and GRE performance while reducing mind wandering. Psychological Science, 24, 776-781. DOI: 10.1177/0956797612459659.

Nelson, S. K., Kushlev, K., English, T., Dunn, E. W., & Lyubomirsky, S. (2013). In defense of parenthood: Children are associated with more joy than misery. Psychological Science, 24, 3-10. doi:http://dx.doi.org/10.1177/0956797612447798

 Nusbaum, E. C., & Silvia, P. J. (2011). Shivers and timbres: Personality and the experience of chills from music. Social Psychological and Personality Science, 2, 199-204. doi:http://dx.doi.org/10.1177/1948550610386810 [Focus on “personality and chills during music” and “do music preferences mediate effects of openness” results]

 Ronay, R., & von Hippel, W. (2010). The presence of an attractive woman elevates testosterone and physical risk taking in young men. Social Psychological and Personality Science, 1, 57-64. doi:http://dx.doi.org/10.1177/1948550609352807 [focus on “skateboarding” and “testosterone mediation” results]

 Saslow, L. R., Muise, A., Impett, E. A., & Dubin, M. (2013). Can you see how happy we are? facebook images and relationship satisfaction. Social Psychological and Personality Science, 4, 411-418. doi:http://dx.doi.org/10.1177/1948550612460059 [Study 2]

 Seder, J. P., & Oishi, S. (2012). Intensity of smiling in facebook photos predicts future life satisfaction. Social Psychological and Personality Science, 3, 407-413. doi:http://dx.doi.org/10.1177/1948550611424968 [focus on Study 1]

Tice, D. M., & Baumeister, R. F. (1997). Longitudinal study of procrastination, performance, stress, and health: The costs and benefits of dawdling. Psychological Science, 8, 454-458. Retrieved from http://search.proquest.com/docview/619221802?accountid=10457

 Valor-Segura, I., Expósito, F., Moya, M., & Kluwer, E. (2014). Don't leave me: The effect of dependency and emotions in relationship conflict. Journal of Applied Social Psychology, 44, 579-587. doi:http://dx.doi.org/10.1111/jasp.12250 [use Study 1]

 Vezzali, L., Stathi, S., Giovannini, D., Capozza, D., & Trifiletti, E. (2014). The greatest magic of harry potter: Reducing prejudice. Journal of Applied Social Psychology, 45, 105-121. doi:http://dx.doi.org/10.1111/jasp.12279  [Study 1]

 Wong, E., Ormiston, M., Haselhuhn, M. (2011). A face only an investor could love: CEOs’ facial structures predict their firms’ financial performance. Psychological Science, 22, 1478-1483. doi: 10.1177/0956797611418838.

Take a walk!

Was "taking a walk" in the study a between groups or within groups independent variable? © Dave & Les Jacobs / AGE FotoStock

The New York Times reported recently on a study suggesting that taking a walk can be beneficial.

The article reports on a study showing that taking a lunchtime walk improves workers' enthusiasm and relaxation at work. Along with five other researchers, Dr. Thøgersen-Ntoumani randomly assigned some physically inactive staff at a large university in England to walk three times a week for 30 minutes, while researchers asked those in the control condition not to change their typical exercise behavior. Prior research led researchers to hypothesize that those in the experimental (walk) condition would report greater well-being than those in the control condition.

Questions:

To introduce the topic, the journalist reports that

It is not news, of course, that walking is healthy and that people who walk or otherwise exercise regularly tend to be more calm, alert and happy than people who are inactive.

a) What kind of claim is this? What are the variables in the claim?

The Times goes on to report that

A new study finds that even gentle lunchtime strolls can perceptibly — and immediately — buoy people’s moods and ability to handle stress at work.

b) What type of claim is this? What are the variables in the claim?

The journalist also writes that

Most of the resulting 56 volunteers were middle-aged women. It can be difficult to attract men to join walking programs, said Cecilie Thogersen-Ntoumani, the study’s lead author and now a professor of exercise science at Curtin University in Perth, Australia. Walking may not strike some men as strenuous enough to bother with, she said. But she and her colleagues did attract four sedentary middle-aged men to the experiment.

c) What type of study validity does this statement concern?

For the first set of analyses, the NY Times goes on to report that

The volunteers completed a series of baseline health and fitness and mood tests at the outset of the experiment, revealing that they all were out of shape but otherwise generally healthy physically and emotionally. Dr. Thogersen-Ntoumani and her colleagues then randomly divided the volunteers into two groups, one of which was to begin a simple, 10-week walking program right away, while the other group would wait and start their walking program 10 weeks later, serving, in the meantime, as a control group.

d) Is participation in the walking program a between-subjects or within-subjects variable at this stage of the study? Why did the researchers use random assignment?

During the intervention period, the researchers also compared the responses to questions about mood given first thing in the morning to those given after the lunchtime walk by participants’ who were in the walking program, and they compared their responses after lunch on days they had walked with days they did not walk. As the Times reports,

On the afternoons after a lunchtime stroll, walkers said they felt considerably more enthusiastic, less tense, and generally more relaxed and able to cope than on afternoons when they hadn’t walked and even compared with their own moods from a morning before a walk.

e) After reading this description, you might have noticed that, in addition to their overall between subjects design, these researchers embedded a within groups design.  What is the primary advantage of a within-groups comparison over a between-groups comparison?

f) Is the causal claim made by the Times (listed in question b) justified? Apply the three criteria.

 

Suggested answers:

a)     This is an association claim, as the comparative wording (tend to be more …) suggests. The two measured variables are walking and emotional/mental state (degree of calmness, alertness, and happiness).

b)     This is a causal claim, because the word buoys is a causal verb meaning "to increase" or "lift up." The variables are walking (IV) and mood and ability to handle stress (DVs).

c)     This statement concerns the external validity of the study, as the sample was not randomly selected and is not representative of the population of sedentary office workers in terms of gender.

d)     Participation in the walking program at the start of the study was a between-groups variable. Random assignment typically results in equivalent groups and rules out selection effects, such as  individual differences among the participants. The use of random assignment means that participation in the walking program was a between-subjects variable.

e)     The primary advantage of a within-groups comparison is that there are no subject variable confounds (selection effects), because the participants are the same in both groups (i.e., it is the same people completing the ratings in the morning before the walk and in the afternoon after the walk).

f)      The study meets the three criteria for causality. There is covariance; that is, participants in the walking group gave higher mood ratings than those in the control condition. The IV, walking, took place prior to the participants’ giving their ratings of mood, establishing temporal precedence. Finally, the researchers used random assignment to address potential third variables, suggesting strong internal validity.  

 

Warmest thanks to Chantal Poister Tusher, Lecturer at Georgia State University, for guest blogging today!

 

 

Is Costco Making us fat?

The Atlantic ran a story under the headline, "Is Costco Making us Fat?"

It talks about the rise of obesity in the U.S., and proposes that cheaper food might be the main culprit. What I like about it is the description of how they used multiple regression to decide which of many factors has the strongest relationship to obesity.

Along with three other researchers, Charles Courtemanche, an assistant professor of economics at Georgia State University, analyzed a number of these theories for a recent study.

Courtemanche realized that a lot of past studies came to conclusions like, "the increasing popularity of driving to work is correlated with the rise of obesity." But he wasn't sure what, specifically, was having the greatest effect on obesity's rise.

He gathered 27 things he thought might be contributing to obesity—everything from working hours to exercise trends to gas prices—and put them in what he calls a "statistical horserace." Essentially, he took all the studies that associated some factor with obesity and saw what would happen if he held everything else constant, like demographic or economic changes. (For example, he told me, "If you're just looking at cigarette prices, was it really cigarette prices, or is it just that they're related to food prices and gas prices? It's controlling for everything together.)

Only two of the factors ended up being meaningful drivers of obesity: 1) the proliferation of restaurants and 2) the rise of warehouse clubs, like Costco, and super-centers, like those Walmarts that have grocery stores in them.

Of course, it's fascinating to read that the increasing prevalence of warehouse clubs and restaurants is associated with increasing obesity, controlling for so many other factors.

But despite The Atlantic's headline, these data can't support their causal claim. All the variables are measured. And even though they analyzed (and held constant) 27 things, there could still be that 28th, unmeasured variable out there....