Everyday Research Methods

What is an evidence-based treatment? Why should you use one?

My textbook (and this blog) helps teach readers to become systematic, critical consumers of the information they find in the press and in journal articles. But a recent article on therapies for PTSD calls attention to another important realm: being a consumer of psychotherapies.

What do I mean by this? For one, if you plan to see a therapist or mental health counselor, you should be an educated consumer of their services--which treatments are the best?

Second, what if you plan to train to become a psychologist, therapist, or licensed profesional counselor someday? Which of the many types of therapy should you learn? Which should you practice?

The answer in both cases is: evidence-based treatments (also called "empirically supported therapies" "evidence-based practice" and "evidence-based psychotherapy"). Therapists should ideally use treatments that have been shown to work through rigorous scientific testing.There are some therapies out there for which there is no evidence. Trying an untested therapy might be like taking an expensive pill when you're not sure it works. When people are suffering, we know that it's critical for them to get treatment--but it's a good idea to look for the best treatment available.

A journalist at ScienceNews Daily interviewed psychologist Edna Foa, who developed and tested a therapy for PTSD called Prolonged Exposure therapy (PE). In this therapy, 

...patients approach -- in both imaginary and real-life settings -- situations, places, and people they have been avoiding. The repeated exposure to the perceived threat disconfirms individuals' expectations of experiencing harm and, over time, leads to a reduction in their fear.

Over years of testing, PE and other forms of cognitive behavioral therapy have proved highly effective in addressing the distress and dysfunctional problems that trauma victims experience.

 Dr. Foa and her collaborators are in the news because of a recent article they wrote, which concluded that

...the majority of mental health professionals do not use such evidence-based treatments (EBTs) when working with patients suffering from PTSD. 

Many clinicians believe that good psychotherapy should be individualized and should focus on the underlying causes of one's problems and symptoms. But studies show scant evidence that psychodynamic therapy -- which focuses on such issues as difficult childhood relationship with parents -- effectively eases PTSD symptoms, according to the report.

Unfortunately for those who are suffering, there's a large gap between what clinical scientists know and what therapists practice in their offices. And as you read in Chapters 1 and 2, just because a therapist thinks a technique works based on his or her experience with it, that does not mean that the therapy is truly effective. Personal experience has no comparison group!

What can you do?  For one, if you're looking into psychotherapy for yourself or a friend, ask potential counselors if they use evidence-based therapies. If you or a friend suffer from PTSD, look around for a therapist who uses an evidence-based therapy like PE or cognitive-behavioral therapy. My colleagues Dr. Tim Fowles and Dr. Ryan Beverige recommend the following websites, which list interventions and indicate their supporting evidence.

The Substance Abuse and Mental Health Services Administration (SAMHSA; U.S. Dept. of HHS) maintains lists more than 280 interventions with information about the intervention and its supporting evidence:

http://www.nrepp.samhsa.gov/

And there is also this list from the APA:

http://www.psychologicaltreatments.org/

And if you're considering becoming trained as a counselor or psychologist someday, ask your professors and mentors if the therapies they are teaching you are evidence-based. Research shows us which therapies are better than others--wouldn't you want to learn the ones that actually have been shown to work?

 

Related articles

Most Effective PTSD Therapies Are Not Being Widely Used, Researchers Find

What You Should Know About Treatment for Posttraumatic Stress Disorder

Measuring charter schools

Are charter schools effective?  Philadelphia Inquirer columnist Joel Naroff wrote about the data we might use to decide whether charter schools are doing well. He describes the pitfalls of using achievement test scores to measure school effectivness. His article uses the language of multiple regression and internal validity to describe how difficult this process is.

Specifically, Naroff mentions the following about charter schools, discussing

....what researchers call "selection bias." Parents have to choose to send their children to a charter school, meaning that charters start with students who have high parental involvement.

a) To translate his comment into our own research methods terms:  Imagine a study that compared achievement test scores in a city's charter schools with the city's traditional public schools. Sketch a simple bar graph showing that charter schools score higher in achievement than traditional schools. Does this result mean that charter schools cause better school performance?  What are some internal validity issues here?  Apply the term  "selection bias" or "selection effect," as the author is doing above.

This article also states that

Unfortunately, we don't adjust test scores for differences in factors such as intelligence, income, parental involvement, or school facilities. Instead, we use misleading, unadjusted test-score comparisons. That means some schools that test well might not be doing all they can, while those with failing scores might be making outstanding progress. We are largely clueless about what is happening.

In this paragraph, the author seems to be advocating for using multiple regression to evaluate the effectiveness of charter schools. 

b) Why would multiple regression help us rule out the selection bias problem the author mentioned earlier?

c) Sketch a possible table of multiple regression results, as the author suggests above: Assume that achievement test score is the dependent variable of interest. What would be your predictor variables? What results would show that charter schools really are more effective than traditional schools, even controlling for these selection effect variables?  What results would show that charter schools are not more effective than traditional schools?

d) Do you agree that "we are largely clueless" about what is going on? Do you think the regression approach he mentions is a good idea? Are there any downsides to the regression approach?

 

Suggested answers to question c:

Below is a possible table of (fabricated) results that would show that charter schools really are more effective than traditional schools, even controlling for the selection effect variables the author mentioned.

DV: Achievement test scores

Predictor                                                                                    Beta

Traditional vs. Chater (0 = traditional, 1 = charter)       0.15**
Entering achievement test score                                         0.59*
IQ of child                                                                                0.20**
Child's family income                                                            0.11**
Child's parental involvement                                               0.18*
School's level of facilities                                                       0.08

Note: Data are fabricated for teaching purposes.

* represents a statistically significant effect. This table suggests that charter schools are associated with better achievement test scores, even when controlling for earlier scores, IQ of child, income, parental involvement, and school facilities.

 

Now, below is a possible table of (fabricated) results that would show that charter schools are not more effective than traditional schools, even controlling for these selection effect variables

DV: Achievement test scores

Predictor                                                                                    Beta

Traditional vs. Chater (0 = traditional, 1 = charter)       0.03
Entering achievement test score                                        0.57*
IQ of child                                                                               0.24**
Child's family income                                                           0.13**
Child's parental involvement                                               0.16*
School's level of facilities                                                       0.06

Note: Data are fabricated for teaching purposes.

* represents a statistically significant effect. This table suggests that charter schools are not associated with better achievement test scores, when controlling for earlier scores, IQ of child, income, parental involvement, and school facilities.