Superintelligence: Paths, Dangers, Strategies

"The catalyst for his concerns regarding the control problem in artificial intelligence."

"Nick is another remarkable individual. A nice story he tells is that when he was at university in Sweden, his tutor said to him, “It’s come to my attention that at the same time as you’re doing this degree you’re also doing another one. You can’t do that, you have to stop.” Bostrom says he didn’t stop and he also didn’t tell her he was doing a third degree at the same time! He’s a bit of a genius, brain-the-size-of-the-planet individual. He is a philosophy professor at Oxford University and runs the Oxford Martin School’s Future of Humanity Institute. He’s said, for a long time, that Kurzweil is half right. If we get AGI, the outcome could be absolutely wonderful. But it could also be terrible. He warns about the possibility not so much of a superintelligence going rogue—like Skynet, or HAL in 2001—but more simply of an immensely powerful entity that would not set out to damage us but have goals that could do us harm. He uses what he calls a ‘cartoon’ example: The first AGI turns out to be developed by someone who owns a paperclip manufacturing company. The AI has the goal of maximizing the production of paperclips. After a little while, it realizes, “Well these humans, they’re made of atoms, they could be turned into paperclips.” So it turns us all into paperclips. Then it turns its gaze towards the stars and thinks, “Well there’s an awful lot of planets out there and I can turn all those into paperclips!” So it develops a space program and travels around the cosmos and turns the entire universe either into paperclips or something that makes paperclips. It’s an absurd idea, but it shows the possibility of inadvertent damage. An AI doesn’t have to hate humans in the way Hollywood often shows them disliking us. It can just have goals that do us damage. If you think about it, if you have a superintelligence around and it’s capable of changing the environment on the earth radically, there’s only a narrow range of positions that are good for us. We can’t have anyone tinkering with the mixture of oxygen and nitrogen in the air. We don’t want anyone changing the way gravity works. We don’t want anything taking away rare earths or materials that we use in our smartphones or for food. We need the superintelligence to leave things pretty much as they are and not make any radical changes. But a superintelligence could have any goal. We’ve got no idea what goals it may give itself. Get the weekly Five Books newsletter What Bostrom is saying is, ‘These are possibilities which we have to take seriously, because we may get AGI in the next few decades. We need to make sure that the first AGI, and all future AGIs, are safe.’ There’s a project called Friendly AI or FAI, which is to make sure that AIs are safe. It’s a very, very difficult job, but the good news is we’re a smart little mammal and we’ve got quite a lot of time to find the answers. He is. The Future of Humanity Institute is one of the four main existential risk organizations around the world which are thinking hard about this problem and trying to raise awareness—which is what I’m trying to do as well—so that more smart people can be applied to solving it. Very few people have. I’ve been slightly obsessive about AI for 15 years or so. Bostrom’s book, Superintelligence, really changed the landscape. It’s a really, really good book. It’s quite technical, in a philosophical sense, it can be quite hard going. I talk to everybody I meet about AI and I get this glazed look because people think it’s just Hollywood, that it’s nonsense. Bostrom’s book was so thorough and his credentials so solid that suddenly a lot people—like Bill Gates and Stephen Hawking and Elon Musk—took notice and started to speak publicly about it. That’s what’s really brought it to the public attention over the last year or so. For a long time, it was thought to be blasphemous. You shouldn’t do it, even if you could, because it was God’s preserve. But yes, stories about either mortals or gods making other humans have been around pretty much forever. Things are always impossible until they become possible. Ever since people looked at birds, we’ve tried flying. It was always impossible until suddenly a couple of bicycle makers managed to do it. It is a hard thing to do: To create a human-level AI is a massive, massive task. Although, in many ways, AI is near the beginning of its history, it’s come a long way. It was only in 1956 that the discipline got going, at the Dartmouth Conference in America. It’s had periods of over-hype, followed by periods of winter when you couldn’t raise any money for it. One thing that has happened recently is the application of deep learning to AI, the use of clever statistics and big data. It turns out that giving algorithms datasets of billions or trillions makes them unreasonably useful. If you only give them datasets of millions, they can’t do very much. Deep learning has led to really enormous strides in image recognition and real time machine translation, for instance. You can also see the glimmerings of an AI becoming sensible. Geoff Hinton, one of the founders of deep learning, reckons we’ll have the first computers with common sense in 10 years. That’s startling. That’s taking current AI systems, what’s known as artificial narrow intelligence, and improving them — bearing in mind that much of the hardware and a lot of the understanding is on an exponential curve of improvement. Another way to create an AGI would be to copy the machine that we know works already: the human brain. People have thought for a while that if you slice a human brain incredibly thinly and map where all the neurons are and how they connect to each other, you can then reconstruct that inside a computer. That’s always sounded like a massive task but we now know that it’s harder than we originally thought because neurons are complicated little beasts. They’re little computers in themselves, each one. So it’s not a case of taking the 85 billion neurons in the brain and treating each one like a byte in a computer. You have to treat each one as a computer. That makes the process harder by several orders of magnitude. I saw a piece recently where someone argued very cogently that the amount of processing that goes on inside a human brain is 10 to the power of 21 FLOPs (floating point operations per second, a measure of processing activity). That’s a huge, huge number, but it’s only a question of time until we get computers that can handle that. It’s a hard job, it will take a long time, but we’re moving towards it at an exponential rate. For me, that’s a simple question. I’m an atheist, and I don’t think they exist. Those who do think they exist could say, “Whatever you do to replicate the material brain it won’t capture the soul, so you’re not going to create something that is like a human.” If that’s true, maybe what we’ll end up with is an AI that can do everything a human can do, but just doesn’t have a soul. I wonder whether the AI will care very much about that. It may say “You believe you’ve got this thing you call the soul, but you don’t have any evidence for it. Anyway it doesn’t slow me down and I seem to be a lot smarter than you, so I’m not going to worry about it.” We will certainly program the goals that it comes to awareness with. It may accept those goals and continue to operate on them. However, it may reflect on them and think, ‘Well, you’re quite smart little mammals, but I’m a lot smarter and I’ve got a better idea of what the goals should be, so I’m going to change them.’ We don’t really know. One of the interesting things about the Friendly AI project as a whole is that it’s really hard to specify a goal which would be always and forever good for us. If you said, ‘The goal is to enhance the wellbeing of humans…’ But what is the wellbeing of humans? People don’t agree on that. In fact, all of us contradict ourselves: You don’t have to dig very far to work out that we have internally contradictory goals in our system, and probably had to in order to get by. A superintelligence which is programmed with such an internally inconsistent and possibly incoherent goal is either going to be paralyzed or have to change its goals, or could end up with some pretty perverse outcomes. So you might say, instead, “Make all humans happy.” What the superintelligence then does is put us all into little coffins, wrap us in straitjackets, and feeds us intravenously with all the nutrition we need. It then puts a probe into the pleasure centers of our brain and stimulates our pleasure centers forever. We end up stuck inside these coffins, happy as anything, but effectively gone as a species. The universe might look at that and think, “No big deal, we’ve swapped one smart little mammal for a superintelligence.” But we would care. I’d care. I am. The subtitle of my book is ‘the promise and peril of AI.’ It has huge promise and huge peril. The worst peril, if you like, or the way to make sure we get the peril, is to ignore it."

"I picked this book because the possibility of developing human-level artificial intelligence , and from there superintelligence—an artificial agent that is considerably more intelligent than we are—is at least a contender for the most important issue in the next two centuries. Bostrom’s book has been very influential in effective altruism, leading lots of people to work on artificial intelligence to ensure that it is developed safely. I don’t agree with the entire book, but there are many compelling arguments in it, and it would be extremely overconfident to dismiss it as too speculative. In fact, I think there should be a lot more work that tries to understand the biggest challenges for the next two hundred years, and what we could do to try to overcome them. Since these long-term ideas have become more influential in effective altruism, we’ve also done less mass media outreach, so I don’t actually have a perfect sense of how compelling the public finds them. Certainly, for the issue of climate change, people are very onboard with the idea that we should be safeguarding the planet now in order to provide a good planet for our children, their children, and so on. Within the effective altruism community, people have systematically found this set of arguments very compelling. The key issue is really whether you think that people in the future matter as much as people living today. If so, then do you think that there will be a lot more people in the future than today? It seems extremely plausible. Then, you’ve at least acknowledged that most of the value lies in the future. The mass attention we’re trying to give to this, such as with my TED Talk , has been very well-received so far. We’re still in the early days, and some people do think that we should only focus on the more easily measurable and quantifiable bets. That’s a perfectly reasonable position, but I’m optimistic that the arguments in favour of effective altruism are so compelling that the public will learn to see longtermism as a very important issue."