Everyday Research Methods

Case study in online personality tests: Psychopathy

What would you do if an online quiz told you you were a psychopath? Photo: Selimaksan/ iStockphoto

The Guardian's science section has an editorial about whether it is possible, or responsible, to offer online quizzes for people to self-diagnose their own degree of psychopathy.

They provide a definition of psychopathy in the article:

Psychopathy is a rare and serious personality disorder, which is primarily diagnosed in criminal justice settings. Individuals with psychopathy lack empathy and remorse, do not emotionally connect with other people, are manipulative, use other people to their own ends and are often aggressive or violent. Psychopaths are estimated to make up approximately 1% of the population, but comprise up to 20% of the prison population.

You may have seen some online personality tests (or "quizzes" as they are sometimes called), that purport to tell you whether you have the qualities of a psychopath. I was surprised that such a test is embedded in the dating site OKCupid. And you can find others online, like this one. 

The authors of The Guardian piece raise two points about such online tests. First, do the online personality tests work--do they accurately detect psychopathy? In the terms of Chapter 5, on Identifying Good Measurement, do the online quizzes have good construct validity?

In order to have construct validity, these tests should be both reliable and valid.

a) What kind(s) of reliability do you think is (are) important to establish for these online tests--interater reliability? internal reliability? test-retest reliability?  How would you decide if the tests have each kind of reliability?

Even if you learned that an online test has reliability, you would also want to establish the test's validity.

b) What kinds of validity might be relevant here? What would you do to establish criterion validity of one of the online tests, for example?

As the authors of The Guardian piece write:

...self-rating instruments are never perfect and there is a great deal of room for error – particularly when the instruments have not been subjected to rigorous empirical study assessing their reliability, validity and ability to capture individual differences in the population. We see no evidence that the online quizzes have undergone these procedures and as such what constitutes a high score is likely to just represent someone’s subjective opinion.

The following sections in The Guardian piece that discusses content validity, too. (What's the definition of content validity?)

To get a diagnosis of psychopathy, an individual has to score a minimum of 30/40 on a standard diagnostic instrument that relies on recorded, independently verified information from institutional files, as well as an in-depth interview administered by a trained professional

People may endorse that they have some psychopathic traits without actually having a full-blown psychopathic personality disorder. But scoring relatively high on some of the features of psychopathy does not make a person a psychopath. Consequently, there is a concern that psychopathy quizzes may suggest to people with psychopathic traits that they in fact are bona fide psychopaths.

c)  Explain why this section (quoted above) is about content validity of a psychopathy test. There are several points you could make.

In sum, the first major argument in The Guardian piece is that these online psychopathy tests may not be construct valid.

The second major argument in the article is that these quizzes are also not responsible. After the authors took the online tests and endorsed a few psychopathic traits on purpose, they got this feedback from OKCupid's test:

Wow, you are a genuine psychopath. You lack empathy, are highly manipulative, disregard the law, and don’t even have any delusions to blame for your behaviour. Therapy is unlikely to help you and would in fact just make you better at manipulating others. Chances are that most people don’t even realize just how sick you are.

As the authors point out, this kind of feedback may upset certain people who are actually not  psychopathic! And in a way, the feedback is almost congratulatory. The authors write:

In particular, those who encounter these quizzes on dating websites might be especially concerned about how such feedback reflects on their social abilities. At worst, the feedback was irresponsibly congratulatory and even appeared to exhort people to capitalize on their “psychopathic personality” to use others for personal gain. We were also concerned about some of the feedback “diagnosing” the respondent as a psychopath and telling them that they cannot change and that no therapy will work for them. Such feedback is misinformed...

Overall, I thought this was a thoughtful scientific analysis of these online "tests".

d) What other online personality or mental health tests have you encountered? What kind of data (inspired by Chapter 5) would convince you that these tests are reliable and valid?

 

 

Fabricated data in the news

Yet another case of fabricated data, this time from political science, is being analyzed in the news lately. Last December, Science published an article showing that certain types of canvassing techniques could dramatically change people's attitudes toward marriage equality. However, the data for the study were apparently fabricated by one of its key authors. Here are two good pieces that cover the timeline of the article, its discovery, and its retraction:

This one is from the New Yorker and focuses on the graduate student who looked into the case and raised his suspicions.

This one is from This American Life, who originally included the study in a story, and later retracted it.

 

Thanks to Dr. Renee Carrico of Saint Michael's College for suggesting that I add this to the blog!

Kids learn through play

Do you think this was the kind of kindergarten play the authors had in mind? Photo: gmutlu/ iStockphoto

Kindergarten is in the news. Specifically, teachers, parents, and school administrators are discussing how much time kids should spend in kindergarten just playing. This story tells how some schools are re-introducing water and sand tables into their classrooms.  And this story describes some research about whether kindergarten is too early to teach formal skills in reading or math. 

There are three studies described in this article--all quasi-experimental. Interestingly, there are three different patterns of results, but all are used to support the argument that kindergarteners should play more. Let's present each one in turn. 

Here's the first study:

A 2009 study by Sebastian P. Suggate, an education researcher at Alanus University in Germany, looked at about 400,000 15-year-olds in more than 50 countries and found that early school entry provided no advantage. 

a) What seem to be the key independent and dependent variables in this study?  What makes this one quasi-experimental, and which design would it likely be? 

b) Could you sketch a graph of this result?

c) This study shows a null result. In Chapter 11, you learned how to interrogate a null result to decide if it's a problem of a weak manipulation, insensitive measures, and so on. Apply the tools from Table 11.2 to decide if this study really does support a null effect. 

Here's the second study:

Advocates say that starting formal education earlier will help close [acheivement] gaps. But these moves, while well intentioned, are misguided. Several countries, including Finland and Estonia, don’t start compulsory education until the age of 7. In the most recent comparison of national educational levels, the Program for International Student Assessment, both countries ranked significantly higher than the United States on math, science and reading.

d)  What seem to be the key independent and dependent variables in this study?  What makes this one quasi-experimental, and which design would it likely be? 

e) Could you sketch a graph of this result?

f) Can this second study support the claim that "starting school entry later causes kids to score higher on math, science, and reading? 

And here is the third study:

Other research has found that early didactic instruction might actually worsen academic performance. Rebecca A. Marcon, a psychology professor at the University of North Florida, studied 343 children who had attended a preschool class that was “academically oriented,” or one that encouraged “child initiated” learning. She looked at the students’ performance several years later, in third and fourth grade, and found that by the end of the fourth grade those who had received more didactic instruction earned significantly lower grades than those who had been allowed more opportunities to learn through play. 

g)  What seem to be the key independent and dependent variables in this study?  What makes this one quasi-experimental, and which design would it likely be? 

h) Could you sketch a graph of this result?

i) Can this second study support the claim that "receiving didactic instruction causes kids to get lower grades in fourth grade?"

Suggested/selected answers:

a) This sparse description makes it difficult to determine, but the variables seem to be "early school entry or not" (as the IV) and "academic performance" (as the DV).  This study is apparently a non-equivalent control groups post-test only design.

b) A graph of this result might look like this (data are fabricated for the purpose of illustration):

c) You might conclude that with 400,000 kids, there's plenty of power in this study to find an effect of early school entry. So apparently, there is no covariance between the variables of "early school entry" and "school success at age 15." 

d) The IV could be stated as "country" (levels being Finland, Estonia, and U.S.) or the IV could be stated as "age at compulsory education" (levels being age 7 or age 5). The DV would be math, science, and reading scores on the Program for International Student Assessment (PISA) This study is apparently a non-equivalent control groups post-test only design.

e) Here's one possible graph (Data are fabricated for purpose of illustration):

f) No; although the covariance is in the appropriate direction and although temporal precedence is established (Because the age of starting school comes before grades on the PISA), there is no internal validity. As  a non-equivalent control groups post-test only design, this study has selection effects, because  as the article points out, Estonia and Finland "are smaller, less unequal and less diverse than the United States."

g) The IV is type of preschool learning, (levels were "didactic" vs. "child initiated").  The DV would be grades in 4th grade. This study is also a non-equivalent control groups post-test only design.

h) A graph of this result might look like this (data are fabricated for the purpose of illustration):

 

i) Once again, we have covariance and temporal precedence here (kids in child initiated instruction did better, and the type of instruction came before the 4th grade grades). However, the study does not mention that students were randomly assigned to the two types of instruction, so we have no internal validity. There is a likely selection effect.