Everyday Research Methods

The Gender Shades project

The design was a 2x2 factorial. What were the IVs?
Photo credit: Djomas/Shutterstock

Researcher Joy Buolamwini of MIT has conducted a fascinating study of how accurately image-processing algorithms detect and classify the faces of people from different racial and gender backgrounds. She coined the term "the coded gaze" for a bias she found in these algorithms. Her research was inspired by a personal experience.  Joy is a Black woman, and she noticed that computers were able to correctly recognize the faces of her light-skinned friends. However, when she gave the facial recognition computers an image of herself, they did not accurately detect it was a face at all. Other times, it detected her face, but misgendered it as male. The tools she tried included commercially-available image processors from Microsoft, Kairos, IBM, and Face++. 

Joy's research is an example of how artificial intelligence (AI) tools can recreate biases in the world. Specifically, because AI is trained on a corpus of images and text that contains gender and racial biases, its algorithms will tend to reproduce those biases. 

To get started with the questions, watch this 5-minute video in which she describes the overall patterns. 

The research Dr. Buolamwini designed can be described as a factorial design. Let's walk through it. In addition to watching the video, you could also explore the website, www.gendershades.org. 

Buolamwini's project systematically tested how accurate software from three companies (IBM, Microsoft, and Face ++)  was, across a large sample of faces that varied in terms of skin type and gender. Buolamwini used publicly-available photos from members of parliament from 8 countries. Because she used known people, she knew ahead of time which faces were men and which were women. 

In her analysis, the dependent variable was accuracy, and its levels were yes (the computer correctly identified the target's gender) and no (the computer did not)

Here are the accuracy results from the IBM tool, based on data presented in the video (minute 2:42-3:00)

Darker subjects accuracy: 77.6%

Lighter subjects accuracy: 96.8%

However, these two percentages could be further broken down by target sex:

Darker male accuracy: 88%

Darker female accuracy: 65.3%

Lighter male accuracy: 99.7%

Lighter female accuracy:  92.9%

As you can see, the results mean that IBM's product failed to identify darker skinned women up to 1/3 of the time! The other two products were just a little bit better. 

Buolamwini points to the fact that although we need to know more about why this bias exists, it can almost certainly traced back to the fact that the companies didn't train their AI tools on enough dark-skinned male and female faces. 

Questions

a) If you consider only the data from IBM, what were the two independent variables in this analysis? What were each IV's levels? 

b) Convert the data above (The four categories) to a carefully labelled 2x2 grid, as you see throughout Chapter 12 of your text.

c) Use your grid (and your calculator) to estimate the marginal means for each of the IVs. (Hint: You can see two of the marginal means above!)

d) Now make a bar graph or a line graph of the four data points. Label your axes mindfully.

e) Next, assuming that differences larger than 2% are statistically significant, write sentences for each main effect and interaction. Use these sentence templates:

There is/is not a main effect for _[IV #1]_, such that ____ faces are recognized more accurately than ____ faces.

There is/is not a main effect for _[IV #2]_, such that ____ faces are recognized more accurately than ____ faces.

There is/is not an interaction between  _[IV #1] and [IV #2]_, such that for darker skinned faces, ____ are recognized more accurately than ____, but for lighter skinned faces, ____ are recognized more accurately than_____.

f) Buolamwini's research illustrates that machines are not neutral. What is one real-world application of this research that you can anticipate? 

 

 

 

 

Latinos' experiences with the Spanish language

Does the methodology of this survey allow us to generalize the result to all U.S. Latino adults? Photo credit: Monkey Business Images/Shutterstock

It's been awhile since we've interrogated a frequency claim. Pew Research has some new data about U.S. Latinos' experience with speaking Spanish. Pew's synopsis is here.  There's a lot of fascinating content and several interesting graphs on this link--visit it!

Before reading on, make a prediction: What percentage of U.S. Latinos would you estimate are able to carry on a conversation in Spanish? 

Let's see if your prediction was accurate.  The Pew report reads:

Most U.S. Latinos speak Spanish: 75% say they are able to carry on a conversation in Spanish pretty well or very well. And most Latinos (85%) say it is at least somewhat important for future generations of Latinos in the United States to speak Spanish.

Make another prediction: When they break it down by first-generation, second- or third-generation Latinos, who is most (and least) likely to have Spanish ability? 

Here's the report:

24% of all Latino adults say they can only carry on a conversation in Spanish a little or not at all. Among third- or higher-generation Latinos, a much higher share are not Spanish speakers: Close to two-thirds (65%) of third- or higher-generation Latinos say they cannot carry on a conversation well in Spanish.

Another theme in Pew's poll was about whether Latinos who do not speak Spanish ever feel ashamed, or are ever teased, for their lack of ability. Here are some quotes from the report. First, some data on how many have heard such teasing:

Four-in-ten Hispanic adults say they hear other Hispanics make jokes, extremely often or often, about Hispanics who do not speak Spanish or don’t speak it well.

And some data about feeling shamed: 

... not all Hispanics are Spanish speakers, and about half (54%) of non-Spanish-speaking Hispanics have been shamed by other Hispanics for not speaking Spanish.

Questions

a)  The statement: "24% of all Latino adults say they can only carry on a conversation in Spanish a little or not at all" is a frequency claim. What is the variable, and what are its likely levels? 

b) While most of the statements in this report are frequency claims, here's one that's not: "Hispanics who consider their Hispanic identity to be extremely or very important to how they think about themselves are more likely than other Hispanics to say it’s important for future generations to speak Spanish."

What kind of claim is this? 
What are the two variables in this claim? 

To support the statement, "75% say they are able to carry on a conversation in Spanish pretty well or very well", we need to know more about how well the question was asked (Chapter 6 issues--construct validity of a poll), and also how the study got its sample (Chapter 7, external validity issues). Fortunately, Pew makes its methods completely transparent (the questionnaire is available as a pdf on the link here...click on "topline questionnaire").

The item in question was worded: How well, if at all, would you say you can carry on a conversation in SPANISH, both understanding and speaking? 
The response options are Very well, pretty well, just a little, not at all

c) What do you think of this question? Is it double-barreled? Leading? Negatively worded? 

d) What do you think of the response options? Would they foster yea-saying? fence-sitting? Why or why not?

Now here's a quote from the methodology section about the sampling used in the survey. I found this information from clicking on the "Methodology" link on report's home page.

In the quotes below, I've added bold to several terms you learned about in Chapter 7: 

The American Trends Panel (ATP), created by Pew Research Center, is a nationally representative panel of randomly selected U.S. adults. Panelists participate via self-administered web surveys. Panelists who do not have internet access at home are provided with a tablet and wireless internet connection. Interviews are conducted in both English and Spanish. The panel is being managed by Ipsos.

Data in this report is drawn from the panel wave conducted from Aug. 1-14, 2022, and included oversamples of Hispanic, Asian and Black adults, as well as 18- to 29-year-old Republicans and Republican-leaning independents in order to provide more precise estimates of the opinions and experiences of these smaller demographic subgroups. These oversampled groups are weighted back to reflect their correct proportions in the population. 

[...]

The overall target population (i.e., population of interest) for this survey was noninstitutionalized persons ages 18 and older living in the U.S., including Alaska and Hawaii. It featured a stratified random sample from the ATP in which Hispanic, Asian and Black adults, as well as 18- to 29-year-old Republicans and Republican-leaning independents, were selected with certainty. The remaining panelists were sampled at rates designed to ensure that the share of respondents in each stratum is proportional to its share of the U.S. adult population to the greatest extent possible. Respondent weights are adjusted to account for differential probabilities of selection as described in the Weighting section below. [...]

e) Read the quote above and identify the bold-faced terms. Look up any you don't remember in Chapter 7.

f) Based on the information above, can we generalize the result from this survey to the target population of U.S. Latinos? That is, can we conclude that about 76% of U.S. Latinos can carry on a conversation in Spanish? Why or why not? 

Finally, here's a comment from the methodology section that informs us about how Pew checks for response biases: 

Data quality checks

To ensure high-quality data, the Center’s researchers performed data quality checks to identify any respondents showing clear patterns of satisficing. This includes checking for very high rates of leaving questions blank, as well as always selecting the first or last answer presented. As a result of this checking, 12 ATP and seven KP respondents were removed from the survey dataset prior to weighting and analysis.

g) "Satisficing" is not a term you learned in Chapter 6, but. can you figure out what it means from this context?