Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

"Authority is shifting from humans to Big Data algorithms."

"Yes, it’s a good one to start with because it’s very well-written and super accessible. This is a great book with a memorable title. It’s very comprehensive and she has excellent examples—about getting insurance or jobs, how we evaluate people at work and in civic life. The book also has a lot of authority, because the author is a mathematician arguing that maths is not neutral, and that values are always baked into different algorithms. One of the examples that most stuck with me is how we evaluate both students and teachers and how metrics change the activity that is being measured. So, if you focus, say, on test scores, then universities try to game the system by getting students to sit the exams many times and get the grades up. There’s another example in which there was an algorithm that was used to assess how good teachers were, and the algorithm was quite complicated—it took into account different things, including the grades of the students, which of course can vary depending on how difficult the exam was that year or other random things. In the end, people got sacked for this algorithm that was later proved to be tracking nothing at all. It was an algorithm that was self-referencing, but people had to go to court for that to come out."

"This book is a really fantastic analysis of how quantification, the collection of data, the modelling around data, the predictions made by using data, the algorithmic and quantifiable ways of predicting behaviour based on data, are all built by elites for elites and end up, quite frankly, screwing over everybody else. Cathy gives great examples, from predictive policing systems to insurance systems, of how biases are baked into the technologies that are supposedly serving us. For example, an insurance algorithm—and this is actually illegal—will take poorer people and consider them a greater risk and therefore make it impossible for them to afford health or life insurance. She shows how bank loans are contingent on credit, but these algorithms are taking poor people and seeing them as less creditworthy. I understand that, but the question is, ‘What kind of world should we be living in? What do we think about credit or insurance based on those principles?’ So this is a very important book that addresses the dangers of hidden, opaque, biased quantification. Another key point is that it’s not simply the biases of the people who build those technologies in the absence of auditing and transparency and regulation and collaboration, it’s also the datasets that those systems learn from. We know our world doesn’t treat women as equally as men. We know that black people are treated much worse in many countries, including the United States. These are statements of fact. So if we’re going to build technologies that are learning from the world, they’ll end up implementing activities based on a racist or sexist world. “This is a common theme we see in the US: we socialize the costs and privatize the profits.” Because people treat technologies as neutral—especially because they don’t know how the technology works; people don’t even know when they’re interacting with an AI system because there’s no disclosure—we’re going to end up building a world that engineers the worst forms of inequality. Another example that Cathy O’Neill touches on briefly is facial recognition systems. From Michelle Obama to Serena Williams to Oprah Winfrey, every major facial recognition system, whether Chinese or American, thinks these people are men. Google’s image recognition algorithm also mistook various pictures of black people for gorillas. How did Google deal with the situation? They removed gorillas from their image system. Instead of dealing with the underlying architecture that is computationally producing these inequalities and injustices, we’re scratching at little scabs on the surface. We need to do better than that. I use Facebook. I use Twitter. I use Amazon. I’m trying to be mindful of how much reliance and dependency I have on them, but I believe that is a question that can only be answered by each of us individually, unless we’re talking about doing something on massive scale that would actually have an impact. The interventions would need to be systemic, if they occurred through a ‘delete Facebook’ movement, for example. But I don’t feel the burden of this should be on individual users. I would like the burden to be on activist regulators, journalists and, most importantly, our tech gazillionaires. It’s partly my personality. I’m critical, but I’m hopeful and believe in movements. I’m not a cynical person. And I see opportunities. My work has been getting a lot of attention and there’s a reason people are interested in these themes. Also, I don’t want to be so naive as to think that the nice people I’ve been on panels with from Facebook are going to change things, but they’re not sociopaths. They just don’t know what’s going on. Some of them do know and are doing terrible things—and we’ve seen some evidence that Zuckerberg has tread on that ground a little bit—but let’s hope they’re mostly just ignorant and that by bringing these issues to light we can encourage them to experiment with other models of being profitable and worth a lot of money. I actually met Mark 10 years ago. I sat with him and talked to him for over an hour. He offered me Facebook data at that time. So what we saw happening with Cambridge Analytica and Russia was a tacit understanding and assumption at Facebook. I don’t think they thought of what they were doing as necessarily wrong. Part of the problem is they’re a bit clueless about these things. Maybe they are a bit more clued in now, but that’s why these books we’re all writing are important."