The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement

"This is where it gets into what’s important to you ethically and philosophically and politically. In my world, I’ve got some serious concerns about the wholesale automation of discrimination by machine. These decisions that are being driven by computers are often very consequential and have impacts on lives. Who gets access to housing, to credit? In cases where the predictions directly inform a judge’s sentencing decision or a parole board’s decision, it’s even: How long do you stay in jail as a convicted felon? There are a couple of levels of problems. One is: does the system make a decision based directly on a protected class, like race or ethnicity or national origin? In general, no. But there are places in the law where that could be allowed, and there are examples of where it takes place. It turns out that there are a lot of needle-nosed technical experts—even in machine learning ethics—who are proponents of allowing that type of direct discrimination, where the model output by machine learning is permitted access to those protected classes and therefore can base a decision, at least partially, directly on those factors. It could say, ‘Hey! Look, you’re black. We’re going to increase your risk score by seven points.’ That’s literally the kind of thing it could do, and it potentially would do, depending on the data from which it learns. Support Five Books Five Books interviews are expensive to produce. If you're enjoying this interview, please support us by donating a small amount . Getting rid of that problem, ensuring that the model is “colorblind” in that sense and doesn’t have direct access, is, I would say, the bare minimum first step. But that doesn’t eliminate the problem because it turns out you still have what some call ‘machine bias.’ The most famous citation everybody refers to is an article by that name in ProPublica , which talks about predictive policing, where it turns out that there’s a high rate of errors. The system’s always going to make an error, just like humans do. We don’t have a crystal ball or clairvoyance. They can predict better than guessing, potentially better than humans, and potentially less biased than humans. But, because of the state of the world today and historic unfairness, it turns out that underprivileged groups are going to suffer these costly mistakes that would unjustly keep them in prison longer or lacking approval for a credit application or housing. To put it in technical terms, it’s a higher false positive rate. It’s a model saying, ‘I’ve identified you as a positive member of this high-risk group’ when they didn’t deserve it. It’s going to turn out to make that error proportionally more often with underprivileged groups. Those are some of the issues that I think are really important and that I’ve written about in my op-eds . There are a lot of books on this area, and a lot of them I think are really terrific and then always have some point of fault. By process of elimination, I landed on this one book that I think is an exemplar. It addresses at least some of the responsible machine learning, ethical areas without missing the forest for the trees."