The Strengths and Limitations of Risk Assessment: Professor Susan Turner of the University of California-Irvine
In this podcast recorded at the Courts, Community Engagement, and Innovative Practices in a Changing Landscape symposium held in Anaheim in December 2015, Susan Turner, professor in the department of Criminology, Law & Society at the University of California-Irvine, explains how risk assessment tools are developed and discusses the strengths and limitations of risk assessment.
The following is a transcript:
RAPHAEL POPE-SUSSMAN: This is Raphael Pope Sussman, with the Center for Court Innovation. This podcast is part of a series of dispatches from the Court's Community Engagement and Innovative Practices in a Changing Landscape Symposium held in Anaheim in December 2015. The conference focused on justice reforms, including recent developments in California, public safety realignment and Proposition 47. Public safety realignment refers to changes brought about by 2011 legislation that shifted responsibility for certain populations of offenders from the State to the county level. Proposition 47, a ballot initiative passed by referendum in 2014 reclassified certain low level felonies as misdemeanors. I hope you enjoy listening.
Hi, this is Raphael Pope Sussman, with the Center for Court Innovation, and I'm here today at the Court's Community Engagement and Innovative Practices in a Changing Landscape Conference in Anaheim. Right now I'm sitting with Susan Turner, professor in the Department of Criminology, Law and Society at University of California, Irvine. Susan, thank you for speaking with me today.
SUSAN TURNER: Glad to be here.
POPE-SUSSMAN: What's the process for developing a new risk assessment tool?
TURNER: Well, risk assessment tools can be developed by an organization. There are also ones that are available that have been developed by others, so some of the first decisions that jurisdictions make is whether to do it home grown or whether or not to buy something that's been developed and is available for purchase, or is perhaps free. If you're interested in developing a risk tool for your jurisdiction, one of first things you need to decide is what are you interested in predicting? Most jurisdictions in risk assessment are interested in predicting some kind of recidivism behavior, such as arrest or incarceration or conviction. Then we generally use a set of variables that are related to an offender's prior record, age or gender, and then some other factors that may be related to their drug use, mental health status, their association with criminal peers. You basically need to make sure that you have the data for your left hand and your right hand side of the equation.
You gather the information on an individual level, so for every person you're interested in, you gather information on whatever outcome and then their record, their variables such as their gender, their age, and their prior record. You want to have a sample that's large enough. You put them generally in a regression equation, where you try to figure out whether or not these factors are predictive. You're able to gather weights for the variables based on these mocks. For example, if you're predicting whether somebody would be convicted, you might have age and you'd have a weight of 5, gender you'd have a weight of 3, different prior record variables would have various weights. You can just imagine sort of scoring everybody up, and the higher the scores, generally the more likely someone will recidivate. Once you do that on one half of your sample, you build your model. Then you do something called validation on a separate set of data that you haven't looked at before and that tells you whether or not what you've developed on your first sample validates or is predictive on your second sample. That's basically a kind of simplistic view, but the overall gist is that you have data available, you develop your instrument on one half of the sample and test it on the second.
POPE-SUSSMAN: What are the limitations of these tools?
TURNER: Well, there's a lot of good things about the tools and there are, of course, always limitations with any kind of tool. Risk assessment for recidivism is, like many other risk tools, we have risk tools in our regular lives. You think of your car insurance premiums that you pay are based on risk assessment tools. Risk assessments tools are only as good as the data that are behind them, number one. Another limitation to risk assessments tools that we all need to be aware of is that they're never 100% accurate, and probably will never be 100% accurate. They generally are what's referred to as a moderately good predictor, but because of the nature of human behavior and variables that we don't know why people do the things we do, we can never gain 100% accuracy, which sometimes gets us into trouble with say politicians who have very high standards for whether or not you want to use an instrument. Sometimes so high that they're really not practical.
POPE-SUSSMAN: What are the great strengths?
TURNER: Great strengths are that we normally discuss risk assessments tools contrasted with what we refer to clinical judgment, or some people call it gut. The research has shown that actuarial tools or risk prediction instruments are more accurate than gut or clinical experience. Many people don't like to believe that because they're often trained in a clinical background, or you can imagine a parole agent with 30 years of experience is sometimes looking askew at risk assessments tool, but they're more accurate. They can also result in decision making that's more consistent, because you can imagine the great variability if everyone follows their own gut. You have lots of different ways the same person might be scored, so there's consistency. There's also, with some risk assessments tools the advantage of time. Some of them are automated so for large numbers of people they can be scored pretty quickly, allowing an agency to be able to get actuarial tool on a number of offenders without the resources that may be required for, say, 45 or 50 minute instruments that are collected individually.
POPE-SUSSMAN: What roles do you see for risk assessments tools in the larger project of decarceration?
TURNER: One of the things we've talked about here today was the use of risk assessment and one of the things I mentioned was that risk assessment's been around for a long time. It's got a resurgence nowadays, I think partly because we're trying to reduce the current populations that we have incarcerated and we don't have enough money to be able to keep everyone locked up. Money is driving a lot of decisions and one of them I think actually may be risk assessment. We need to decide who are the highest risk people, the ones that we need to devote our resources to, our public safety resources, and to not spend the resources on the ones who are the lower risk.
POPE-SUSSMAN: How do we ensure that risk tools don't show bias against individuals and communities that may have disproportionate rates of established risk factors, like housing or employment or disability?
TURNER: Very good question. I think there are a number of people who are concerned with risk assessments tools based on dimensions that may impact certain groups unfairly. Many risk assessments tools use an offender's prior record as a major variable in their recidivism outcomes. Then there's obviously the question, if prior records are somehow biased in their very creation within communities of color, what does that mean in terms of risk assessments tools. It's a very valid question. One of the ways that we help address this is when we develop tools, we take a look at and see how well the tools works with the different populations of interest.
For example, when we developed a tool for the California Department of Corrections and Rehabilitation, we were particularly concerned about whether the tool predicted for females as well as males, because there's a lot of work suggesting that some of the risk factors for women are different and for men. Also, we were concerned about whether the risk tool worked equally well for blacks and Hispanics and whites and other groups. What we do with those is that we separate the samples into the groups of interest and check to see whether the tool is as predictive in each of the groups. If you find a tool that doesn't work very well in one of the segments that you're interested in, then you need to go back to the drawing board.
POPE-SUSSMAN: I've heard that misdemeanor populations are particularly difficult to assess. Why is that?
TURNER: Well, the discussion of misdemeanor risk assessments has really come into the light here in California with proposition 47. Traditionally, most of us deal with risk assessments in the felony world. All the risk assessments tools I've done are pretty much predicting felony recidivism. There is a question, are these tools good for misdemeanors. There are tools out there that have been developed for misdemeanors. There's a tool for misdemeanor DUIs and some tools that are developed by the University of Cincinnati team have established one that's for misdemeanors. They basically work the same as the felony tools. They're predicting a different kind of an outcome, which is maybe a lower level of behavior. What we find with our risk tools is that these tools can be very robust to different populations and different kind of outcomes.
POPE-SUSSMAN: Do you have any big vision going forward, the next step for risk assessment?
TURNER: Well, one of the risk assessment has always been used pretrial for a long time. It's been used the last 20 or 30 years at the probation stage. I think more recently it's being considered at the sentencing stage, which I think in many ways has a little more ethical issues, whether or not in sentencing actually considering risk as opposed to the sentence type.
POPE-SUSSMAN: What do you mean by ethical issues?
TURNER: I mean, what's the purpose of sentencing and some people say the purpose of sentencing is that it should be proportionate to the crime that was committed and that our statutes are based, sentences are based on perceived severity of violent offences being more serious that property offenses and you reserve the highest sentence lengths for the more violent. If you move to a risk based system and you say, well as we found in our research, people convicted of violent crimes are not necessarily the most likely to re-offend, so you basically a conflict going with what is the justice based, in terms of the offense, versus a risk based, which is about future behavior. That's sort of conflict that you've got to figure out how you're going to deal with. In our own development of the risk tool for the State, in our training and discussion of the tool, we've come across people who are very perplexed by our findings that show that people who have convictions for violent offenses, there's actually what we refer to as negative weights on that, so that people with violent offenses are actually less likely to have recidivism than those that don't, which is very counter-intuitive.
POPE-SUSSMAN: I guess that would also get to people's ideas about what the justice system is supposed to do. Is it supposed to be punitive? Is it supposed to be rehabilitative? Is it supposed to be utilitarian?
TURNER: Yeah, I think, as I teach my students, a couple things you have to remember. If you remember a couple of things, you go out after school it'll be very helpful. One of the them is to remember always the different purposes of punishment. One is deterrence, and that's what we we think risk based decision making is based on. But there's retribution, incapacitation. Yeah, there's many philosophies of what the goal of punishment is. I think risk prediction sets you straight in the cross hairs of some of the other theories of punishment.
POPE-SUSSMAN: Well, thank you. I really appreciate you taking the time to speak with me.
TURNER: Thank you very much for the opportunity.
POPE-SUSSMAN: This is Raphael Pope Sussman, of the Center for Court Innovation, and I've been speaking with Susan Turner, Professor in the Department of Criminology, Law and Society at the University of California, Irvine. For more information on the Center for Court Innovation, visit www.courtinnovation.org