Everyday Research Methods

Self control as a child can predict employment status as an adult

The study used teacher perceptions of each child's self control as the measure of self-control. Photo: Sean Gallup/ EdStock/ iStockphoto

A recent empirical research article in Psychological Science (psychology's premier journal) has received some attention from the press. The Pacific Standard headlined the story this way: "Learn self-control, stay off the dole:  A study from Britain links poor self-control with unemployment"

The journalist reports:

An analysis of decades worth of data on two large, nationally representative groups of British citizens finds those who had problems with self-control as children had more trouble finding, and keeping, jobs as adults.

"Low childhood self-control predicted unemployment in adulthood, even decades later at age 50,” a research team led by Michael Daly of the University of Stirling writes in the journal Psychological Science. 

At age 10, each participant was rated by their teacher on their level of “attentional control” (Do they pay attention in class? Are they easily distracted?), and perseverance (whether they typically complete tasks).

The researchers looked at that data, and compared it with the participants' total months of unemployment from 1986 to 2008.

Later in the article, you'll read:

the researchers note that “Childhood self-control was still a significant predictor (of unemployment) after we controlled for variation in intelligence, social class, and an extensive range of family and health factors.”

Apply your knowledge of Chapter 9 to this article, by answering the following questions:

a)  What were the variables in this multivariate correlational study?  How might you set up a regression table with mock results that are consistent with the pattern described?

b) What information do you have about the external validity of the study? Is this result generalizable? If so, to what population of interest?

c) What information do you have about the construct validity of the study's variables? What more information would you like to see?

d) Here is some information about the effect size of the study.  These quotes help you assess the study's statistical validity:

Our analysis indicated that from youth to age 38, participants with low self-control experienced 1.6 times as many months of unemployment as those with high self-control,” they conclude.

“The predictive strength of differences in childhood self-control was equal to, or greater than, that of intelligence."

What does the second quote ("the predictive strength...") tell you about the betas for childhood self-control and intelligence?  Should you revisit the mock results that you created in part a?

e) Finally, what about internal validity?  The headline, "Learn self-control, stay off the dole" implies that if we teach kids self-control, we will cause a reduction in unemployment.  Is this claim warranted by the study's design and results? Apply the three criteria for causation to this claim.

To feel smart...look something up on Google

 

When's the last time you looked up the answer to a question on the Internet?  I know I do this daily, if not hourly. It turns out that even when Google is the one that answered our questions, we end up thinking we are the smart ones!  The online version of Pacific Standard has a story about how searching for answers to basic questions on the Internet makes us think that we know more than we probably do.  The journalist discusses a multi-study article by researcher Matthew Fisher and colleagues of Yale University.

Here's how the journalist described the first study:

The first experiment featured approximately 200 Americans recruited online. They were presented with four explanatory questions, such as "How does a zipper work?" Half were instructed to look up the answers on the Internet, while the others were told to figure them out "without using any outside sources."

Afterwards, all participants were given four questions on topics unrelated to the first set, such a "How do tornadoes form?" and "Why are cloudy nights warmer?" They were then asked to rate, on a one-to-seven scale, how well they could answer detailed questions similar to those.

"Participants who had looked up explanations on the Internet ... rated themselves as being able to give significantly better explanations to the questions in the unrelated domains," the researchers report. "The effect was observable across all six domains for which participants were asked to assess their knowledge."

a) The journalist described the study as an "experiment."  Was it really?  (Sometimes, journalists call studies "experiments" when they are not.)  If so, what kind of experiment was it (Pretest posttest?  Repeated measures? etc.)?

Next, consider this study:

Follow-up experiments duplicated these results under different conditions, such as when people were asked to assess the level of their knowledge both before and after the Internet-search portion of the test.

b) What kind of experiment is described by the phrase "people were asked to assess the level of their knowledge both before and after the Internet-search portion of the test"?  What would be the IVs and DVs in this study?

Now here's another study in the series:

In a fascinating variation, 233 people performed the same experiment described above with a twist at the end. Rather than assessing their own knowledge, they looked at a series of brain-scan images indicating "varying levels of activation" and asked which best corresponded to their own cerebral activity as they considered the second set of questions.

Those who had just searched the Internet chose images with higher levels of activation than those who had not. This suggests they believed their brains were working harder than the people who did not have access to the Web.

c) What kind of experiment is being described above?  What are the IVs and DVs in this study?  Sketch a graph of the results they are describing.

d)  From the study above, can we support the causal claim that "people who look up information on the Internet believe their brains are working harder than when they don't have access to the Web?"

Finally, the researchers dug a little bit deeper to find out if it was the process of searching the web, or the outcome of getting some information (the answer) that mattered.  The article reports:

The effect was also found when those looking things up on the Web were directed to specific sites, and those who did not use the Internet were provided with the precise information found on those pages. Those who did the searching still thought of themselves as more knowledgeable.

This means that using the Google makes us feel smarter. It's not about the answers we get there--it's about the process of using it in the first place!

 

Suggested answers

a) This really is an experiment (the journalist used the term correctly!).  They manipulated whether or not people were instructed to look up answers on the Internet (or not) and measured people's perceived ability to answer questions.  This was apparently a posttest-only experiment, since the independent variable was between groups, and because the dependent variable was asked only once (after the manipulation).

b) This is a pretest-posttest design--the independent variable (whether people were instructed to look things up on the Internet or not) was between groups, and the dependent variable was measured both before and after the manipulation.

c) The independent variable is, again, whether or not people were instructed to look up answers on the Internet.  This time, the dependent variable is people's perceived level of brain activity--as operationalized by which brain scan image they selected. There is not a lot of information to go on, but it appears that this independent variable was also manipulated as an independent-groups design, making this a posttest-only design. 

A bar graph of this study might have "Use of Internet" and "No use" on the x-axis, and "Likelihood of selecting the most active brain scan image" on the y-axis. The bar for "use of Internet" should be higher than the "no use" bar.

d) Can we support the causal claim? Probably. There is covariance--the results showed that people who were told to use the Internet also reported thinking their brains had higher levels of activity.  There is temporal precedence--because this is an experiment, we know that the Internet searches came before the brain activity inferences.  What about internal validity?  There's not much information provided that helps us decide for sure. But if we assume that the researchers randomly assigned people to the Internet vs. No internet conditions, there would not be selection threats to internal validity.  What about design confounds?  If we assume that the exact questions they were attempting to answer were the same (and not, for example, more difficult questions in the Internet use group), then we can state that internal validity was good.