The Guardian's science section has an editorial about whether it is possible, or responsible, to offer online quizzes for people to self-diagnose their own degree of psychopathy.
They provide a definition of psychopathy in the article:
Psychopathy is a rare and serious personality disorder, which is primarily diagnosed in criminal justice settings. Individuals with psychopathy lack empathy and remorse, do not emotionally connect with other people, are manipulative, use other people to their own ends and are often aggressive or violent. Psychopaths are estimated to make up approximately 1% of the population, but comprise up to 20% of the prison population.
You may have seen some online personality tests (or "quizzes" as they are sometimes called), that purport to tell you whether you have the qualities of a psychopath. I was surprised that such a test is embedded in the dating site OKCupid. And you can find others online, like this one.
The authors of The Guardian piece raise two points about such online tests. First, do the online personality tests work--do they accurately detect psychopathy? In the terms of Chapter 5, on Identifying Good Measurement, do the online quizzes have good construct validity?
In order to have construct validity, these tests should be both reliable and valid.
a) What kind(s) of reliability do you think is (are) important to establish for these online tests--interater reliability? internal reliability? test-retest reliability? How would you decide if the tests have each kind of reliability?
Even if you learned that an online test has reliability, you would also want to establish the test's validity.
b) What kinds of validity might be relevant here? What would you do to establish criterion validity of one of the online tests, for example?
As the authors of The Guardian piece write:
...self-rating instruments are never perfect and there is a great deal of room for error – particularly when the instruments have not been subjected to rigorous empirical study assessing their reliability, validity and ability to capture individual differences in the population. We see no evidence that the online quizzes have undergone these procedures and as such what constitutes a high score is likely to just represent someone’s subjective opinion.
The following sections in The Guardian piece that discusses content validity, too. (What's the definition of content validity?)
To get a diagnosis of psychopathy, an individual has to score a minimum of 30/40 on a standard diagnostic instrument that relies on recorded, independently verified information from institutional files, as well as an in-depth interview administered by a trained professional
People may endorse that they have some psychopathic traits without actually having a full-blown psychopathic personality disorder. But scoring relatively high on some of the features of psychopathy does not make a person a psychopath. Consequently, there is a concern that psychopathy quizzes may suggest to people with psychopathic traits that they in fact are bona fide psychopaths.
c) Explain why this section (quoted above) is about content validity of a psychopathy test. There are several points you could make.
In sum, the first major argument in The Guardian piece is that these online psychopathy tests may not be construct valid.
The second major argument in the article is that these quizzes are also not responsible. After the authors took the online tests and endorsed a few psychopathic traits on purpose, they got this feedback from OKCupid's test:
Wow, you are a genuine psychopath. You lack empathy, are highly manipulative, disregard the law, and don’t even have any delusions to blame for your behaviour. Therapy is unlikely to help you and would in fact just make you better at manipulating others. Chances are that most people don’t even realize just how sick you are.
As the authors point out, this kind of feedback may upset certain people who are actually not psychopathic! And in a way, the feedback is almost congratulatory. The authors write:
In particular, those who encounter these quizzes on dating websites might be especially concerned about how such feedback reflects on their social abilities. At worst, the feedback was irresponsibly congratulatory and even appeared to exhort people to capitalize on their “psychopathic personality” to use others for personal gain. We were also concerned about some of the feedback “diagnosing” the respondent as a psychopath and telling them that they cannot change and that no therapy will work for them. Such feedback is misinformed...
Overall, I thought this was a thoughtful scientific analysis of these online "tests".
d) What other online personality or mental health tests have you encountered? What kind of data (inspired by Chapter 5) would convince you that these tests are reliable and valid?