Recently, the National Institute of Justice (NIJ) — the research arm of the Department of Justice — put out a call for researchers to participate in what they called the “Recidivism Forecasting Challenge.” The challenge was designed to use information about people on parole in Georgia to “improve the ability to forecast recidivism using person-and place-based variables,” encourage “non-criminal justice forecasting researchers to compete against more ‘traditional’ criminal justice researchers,” and provide “critical information to community corrections departments.” Challenge contestants were awarded a collective total of $723,000 for their submitted models.

While heralded by the NIJ as a successful effort that “demonstrate[d] the value of open data and open competition,” in reality, the challenge was marked by serious and fundamental flaws. One of the winning papers encapsulated the issues with the challenge best when they said, “We are hesitant to accept any insights gained from submitted models and question the reliability of their performance. We would also discourage the use of any submitted models in live environments.” Six of the other 25 winning papers also expressed their concerns about the use of models created for the challenge in real-world environments.

So, what contributed to the challenge’s failures?

We argue in a new research study critiquing the challenge that a failure to engage impacted communities (those whose data was used for the challenge) as well as public defenders and other advocates for impacted communities contributed in part to some of the failures of this project. The standard going forward for developing predictive tools should draw on recent resources from the federal government to inform decision-making around whether to develop predictive tools. These efforts should center around developing strong protections for the people whose data is used to build automated systems and the people who may ultimately be evaluated by those systems if they are deployed.

So, why does this matter?

The NIJ has a lot of power, given its position within the Department of Justice, to shape the way that local community corrections departments think about recidivism. We submitted a Freedom of Information Act request to the DOJ to try to better understand how the results of the challenge have been or will be used but have not yet received a response to our request. While it is not fully clear yet how the results of the challenge will be used by the DOJ, the NIJ has already signaled that these types of tools are important to it by spending close to $1 million creating and executing the challenge. Furthermore, the DOJ, through the Bureau of Prisons, already uses a risk assessment tool, PATTERN, to make critical decisions about incarcerated populations. The use of this tool has been roundly criticized by several civil rights organizations.

Beyond influencing decisions about imprisonment and government surveillance, the data produced by law enforcement agencies and the predictions generated from risk assessment tools are often used in making decisions that can have a catastrophic impact on people’s lives — including loss of parental rights, homelessness, prolonged job insecurity, immigration consequences (including deportation), and inability to access credit. The voices of those impacted by these tools should be embedded in the design and implementation of these tools, as they are the individuals who will have to suffer the consequences of poorly designed systems. By involving impacted communities in the development of predictive tools, the design of these types of systems may look dramatically different, or these tools may be determined to not be useful at all.

For more information about the NIJ’s Recidivism Forecasting Challenge and its shortcomings, check out our paper below. Our paper was presented at the Association for Computing Machinery’s Conference on Equity and Access in Algorithms, Mechanisms, and Optimization at the end of October, where it won an Honorable Mention for the New Horizons Award.

Employers today rely on various kinds of artificial intelligence (AI) or other automated tools in their hiring processes, including to advertise job opportunities, screen applications, assess candidates, and conduct interviews. Many of these tools carry well-documented risks of discrimination that can exacerbate existing inequities in the workplace. Employers should avoid tools altogether that carry a high risk of discrimination based on disabilities, race, sex and other protected characteristics, such as personality assessments and AI-analyzed interviews. But where an employer is considering using or is already using an AI tool, robust auditing for discrimination and other harms is one critical step to address the dangers that these tools pose and ensure that it is not violating civil rights laws.

But as usual, the devil is in the details.

Know Your Rights | Know Your Digital Rights: Digital Discrimination in Hiring

Learn more about how automated tools are used in the hiring process and your digital rights under these laws.

A rigorous and holistic discrimination audit of an automated tool — both before and periodically after deployment — can provide employers information to help them determine whether to adopt a tool at all, what mitigation measures may be needed, and whether they need to abandon a tool after adoption. Auditing can also bring much needed transparency when audits are shared with the public, providing critical information for job applicants, researchers, and regulators. On the other hand, algorithm audits that are not carefully crafted can be gamed to present a misleading picture of the system in question or can serve as a cursory box-checking exercise, potentially legitimizing systems that may be discriminatory.

As regulators and legislators are increasingly focused on addressing the impacts of automated systems in critical areas like hiring and employment, including creating requirements for auditing, these efforts must be carefully crafted to ensure that the audits increase accountability in practice. While there is no one-size-fits-all approach to algorithm auditing, audits for bias and discrimination should:

• Evaluate the system’s performance using carefully selected metrics — metrics that consider both when the system works and when it fails.
• Break down performance for people in different groups, including but not limited to race, sex, age, and disability status, and the intersections of those groups.
• Use data that faithfully represents how the system is used in practice.
• Be conducted by auditors who are independent from the entity that built or deployed the algorithm.

In many cases, audits can and should be conducted by interdisciplinary teams of subject matter experts, including social scientists, lawyers and policy researchers, that consult with people who will be impacted by these tools and the users of the system itself. Researchers and practitioners have created many different resources describing how these kinds of audits can be operationalized.

Examining emerging “bias audits” produced in connection with a recently enacted law in New York City (Local Law 144) helps demonstrate why these details are so critical. Because of this law, employers using some of these kinds of technologies are required to publish “bias audits” with statistics about how often job applicants advance in the hiring process when an automated tool is used, broken down for people of different races and sexes.

Some news coverage has described this law as requiring employers to “prove their AI hiring software isn’t sexist or racist.” But a closer look at these “bias audits” indicates that they are incomplete evaluations of bias and discrimination. First, the auditing requirement only applies to a limited set of the types of automated tools used in hiring processes today. So far, we’ve only been able to locate around a dozen bias audits — even though 99 percent of Fortune 500 companies reportedly use some type of automated system in their hiring processes. The law also doesn’t require the audits to assess possible biases related to many characteristics where discrimination in hiring and employment has long been a concern, including disability, age, and pregnancy.

The Historic New Law Protecting Fairness for Pregnant Workers

After a decade of advocacy, the Pregnant Workers Fairness Act is finally going into effect.

When it comes to what’s in the audits, the statistics required to be calculated and reported can provide some basic information about which automated tools employers are using in their hiring processes and the number of job applications being evaluated by these tools. But these audits fall short of meaningful transparency in several ways. For one example, some of the audits we’ve seen so far don’t even provide the name or vendor of the tool being audited. The audits also don’t examine whether the tools work as advertised or whether they accurately assess the relevant skills or capabilities needed for a job. In addition, these bias audits may not fully portray the experiences of candidates or practices of employers for multiple reasons. Several of the audits, including this one of an AI-driven candidate screening tool and this one of an AI-driven applicant scoring tool are missing a lot of data on candidates who were evaluated by the automated tool in question.

The published audits also frequently rely on data that is pooled together from multiple employers that use the same tool, even though they may be using the tool in very different ways. Companies characterize these audits as designed to “ensure non-discrimination against protected groups,” when in fact this data pooling may mask stark disparities or discriminatory practices by employers.

More generally, algorithm audits should be publicly available and easy to access as a matter of transparency. Even though employers are required to publish the audits on their websites, so far, we’ve found it quite difficult to locate these bias audits. That’s why we worked with the New York Civil Liberties Union to create a public tracker of all the ones we’ve seen so far (if you know of Local Law 144 bias audits that employers have posted that we missed, let us know by emailing analytics_inquiry@aclu.org).

As automated systems become more entrenched in every part of our lives, audits of these systems can be crucial to identifying and preventing their harms. But for that to be the case, algorithm audits must be holistic, ongoing, and reflective of the ways automated systems are used in practice. Technologists, civil rights advocates, policymakers, and interdisciplinary researchers should work together to ensure that algorithm audits live up to their potential.