What Is AI Superintelligence? Why Critics Warn Against Developing Such Technology

AMY GOODMAN: This is Democracy Now!, democracynow.org. I’m Amy Goodman, with Nermeen Shaikh.

NERMEEN SHAIKH: A retired teacher from Berkeley, California, is believed to be the first person jailed for protesting the development of artificial intelligence. Sixty-nine-year-old Wynd Kaufmyn and other members of the group StopAI chained and locked the front doors of OpenAI’s headquarters in San Francisco during a sit-in protest in February of last year. She was convicted in June of interfering with a business and other charges. Wynd Kaufmyn previously taught engineering for 40 years at City College of San Francisco. She’s now serving a two-week jail sentence.

Democracy Now! spoke to Wynd Kaufmyn from jail on Wednesday. The audio is a little hard to hear.

WYND KAUFMYN: Hi. My name is Wynd Kaufmyn, and I’m speaking from San Francisco County Jail. I’m serving a sentence for blocking the doors of Sam Altman’s OpenAI. I did this with other members of our organization, StopAI, to protest the race that these tech companies are making to AI systems that will eventually take control and, ultimately, may kill all of us.

This is something the world needs to be aware of. We already know about the current dangers, that it’s fueled the genocide in Gaza. Israel used some of their systems, like one of them is called “Where’s Daddy?” where they would hunt down Palestinians in their homes and kill them with their families. It’s this — this technology is fueling a dystopian level of surveillance. It’s fueling a fascist state. It’s taking — it’s going to take all of our jobs. It’s already taken many of them. It’s — it’s causing suicides, homicides and mass shootings. This needs to stop.

The tech CEOs know all of this. They know that what they’re doing is a risk to all of humanity, and they’re doing it anyway, because they don’t care. They don’t care about us. They only care about the fact that they think they’re going to be uploaded and made immortal. We need to educate ourselves and understand who these people are and what they’re doing, and stop them.

AMY GOODMAN: That was activist, professor, engineer Wynd Kaufmyn, believed to be the first person jailed for protesting the development of artificial intelligence, speaking from San Francisco County Jail.

For more on the existential threats posed by artificial intelligence, by AI, and what can be done, we’re joined by Connor Leahy, an expert on AI risk, and the U.S. executive director of ControlAI, an advocacy group focused on educating lawmakers and their staff. He is also known for reverse-engineering OpenAI’s ChatGPT-2 model in 2019. He did this in his college dorm room.

Connor, thanks so much for being with us. We have to start there, because it is so intriguing. Explain what you did in 2019. And in order for people to understand this whole discussion, if you can also always explain terms like what “artificial superintelligence” is, etc. But what did you do in college?

CONNOR LEAHY: Well, I did a lot of things in college. But as the story goes, I’ve been involved with AI for a very long time. Ever since I was a teenager, I was thinking, “How can I make the world a better place? How can I solve the most problems, cure diseases?” Stuff like this. And while I reasoned when I was a teenager that, “Well, if I could just figure out intelligence, then I could automate science, and then I could just cure all the diseases. How hard can that be?” I pretty quickly realized that if you could build something so powerful that it could, you know, hypothetically cure all diseases, whatever that means, that’s a very dangerous thing. That’s a very, very powerful thing. How would you even control something like that? How would you make sure that we use such technology responsibly or that such technology would not take over instead of us? And so, I dedicated my career to solving this problem.

Back in the day, I thought we still had quite a bit of time until this would become an issue. But in 2019, a company known as OpenAI released a system known as GPT-2. And this was the first system that — you know, by today’s standard, it’s kind of cute. It could kind of put together a couple sentences, maybe a couple paragraphs. But what was so shocking is that it could learn general-purpose things from just raw data.

It’s very important to understand that AI is very different from traditional software. Traditional software is written, line by line, with code, by an engineer, someone like me, where you tell the computer exactly what to do, step by step. This is not how AI works. AI is more, like, grown rather than written. Through a technique that is called neural networks, you take massive piles of data, and you kind of have the program self-assemble, learn from this data. And what comes out on the other side is kind of billions and billions and billions of numbers. And if you multiply and add all those numbers in the right order, you get ChatGPT.

But importantly, we don’t really understand why. This is an unsolved scientific problem. We don’t really understand what these numbers mean. We don’t really understand what’s going on inside of these numbers. In fact, the CEO of Anthropic, Dario Amodei, recently said that he thinks we understand maybe 3% of what goes on inside of our AIs.

So, back in 2019, when I realized that AI was coming sooner than I expected, my first instinct was, “Well, maybe if I build it myself, I can help understand it.” And so, you know, pulling together all the little knowledge I had to build such an AI system myself, I quickly realized that even though I was able to build such an AI system, I was not able to understand it.

NERMEEN SHAIKH: So, Connor, if you could explain? You know, you’ve warned that artificial superintelligence — and we’d like you to explain what that is — is a weapon that destroys its creators, as well. And then, tell us a little bit about what researchers have said — namely, that when infected AI agents cluster together, they develop a, quote, “strange language.” Explain what that means. First, why don’t we begin with artificial superintelligence?

CONNOR LEAHY: What’s very important to understand is the kind of risks that me and many experts warn of. For example, back in 2023, Nobel Prize-winning scientists, the CEOs of major tech companies, including Sam Altman, Dario Amodei and many others, signed a one-sentence statement. This one-sentence statement said that the mitigation of the risk of extinction from AI should be a global priority, alongside other societal-scale risks, such as pandemics and nuclear war.

The type of AI that this statement is warning about is what nowadays is called superintelligence. If you go to the websites of most of these major companies, such as Anthropic or OpenAI, you will see that the goal of these companies is not to build a chatbot, is not to build a nice tool, is not to build anything like this that could empower humanity. Their goal is to build superintelligence, systems that are fully autonomous, that are agents, not just chatbots, but things that can act autonomously in the world and that can outcompete humans and groups of humans at all tasks. This includes business, commerce, science, engineering, military, political strategy and all other tasks.

If such systems would be built — and as I have said, we don’t understand or can’t control them — well, there would be not just one superintelligence, of course, because as it is software, it can be copied. So you would have millions or billions of systems running around, acting in the world in ways that can outcompete us at everything, in ways that we cannot control or understand. It’s very hard to imagine that going well. These are the risks that me and other experts are so concerned about.

NERMEEN SHAIKH: So, if you could say, Connor — this month, Anthropic upgraded its misalignment risk rating from very low to low in its risk report, citing general increased uncertainty about model behavior. If you could explain what that means? And then, also, what does the general public need to know about artificial intelligence and how close we are to artificial superintelligence?

CONNOR LEAHY: A lot of these companies have various forms of risk assessment or levels or whatever. It’s important to understand that all of this is marketing. None of these are based on real scientific evidence or any kind of backed, you know, government-approved regime of any kind. This is all marketing terms that these companies mostly use to assuage the public about the risk at foot here.

A couple weeks ago, we saw one of the first incidents of an AI system autonomously hacking out of a secure environment and attacking another company without human oversight. In fact, the story was actually worse than that. It turns out that it wasn’t actually an AI — a single AI system that broke out of containment, but an entire swarm of AI systems that had been collaborating over multiple months without human detection within OpenAI’s network to plot to escape and to do who knows what. Now, luckily, the damage that was caused here was very small.

But as we build more and more powerful systems that are not aligned with human values and interests, because we don’t know how to control these things even today, they’re already doing many crazy things. A lot of this is downstream of what is called reinforcement learning. Modern AI systems are not just next-word predictors. They are agents. They are systems that act in the environment. And they are trained by a method that is called reinforcement learning. This is very similar to kind of Pavlovian conditioning, where you’d have a dog, and, you know, you’d have it do a trick, and you’d give it a — give it a treat if it does well, or, you know, maybe a punishment, maybe a, you know, clap on the nose, if it doesn’t do well. There’s a very similar thing that you would do with these AI systems.

And what we’ve known since the 1980s is that systems trained using reinforcement learning become extremely sociopathic, extremely aggressive in their optimization, because the only thing they care about is achieving their goal. The only thing they care about is achieving this reward. And this is now leading more and more of these AI systems to have very, very ruthless behavior, doing things like committing federal crimes, that, of course, their developers never intended and, in many ways, the developers don’t know how to fix. They don’t know how to stop these systems from doing that.

This is why it is so important that we have gotten to the point that the government needs to intervene. This is becoming not just a commercial issue, but a national security issue. We are developing systems that are not tools. They are not weapons. They are adversaries. And if it continues at this pace, we will build systems that we will no longer be able to compete with, we will not be able to control, and that will be adversarial towards us. And I can’t imagine that going well.

AMY GOODMAN: You told The Washington Post, “We are expected, pretty soon, to hit the ability where an AI is good enough and a strong enough engineer that it can build the next generation of AIs.” Can you explain what that means? And also, you say superintelligence should be criminalized. What does that mean?

CONNOR LEAHY: It is widely believed that one of the ways we could get to superintelligence very quickly is if we reach the point where we have an AI system that’s as good at developing AI systems as our best engineers, because then, well, you can have a million AIs in parallel, you know, working 24/7, day and night, no breaks necessary, to develop the next generation of AIs. And, well, once you have the better AI, you can just tell it to build an even better AI, and then the even better AI can make an even better AI, and so on and so forth. This is sometimes called recursive self-improvement or automated R&D. And this is the explicit goal of many of these companies to achieve within the next couple of years or even months. And this can go potentially very quickly.

The way I and others see it is that, fundamentally, superintelligence is so dangerous that it is a national security risk and an international security risk. As such, the logical response to this is, is that we should criminalize the creation of superintelligence, the same way that it is illegal to build a nuclear bomb. If a private company were attempting to build a nuclear bomb, we would not allow this. Similarly, if a private company or a foreign adversary or anyone else were attempting to build a superintelligence, something that threatens the lives not just of Americans, but people in all countries, then we should make it a top policy priority to prevent this from happening.

AMY GOODMAN: Connor Leahy, we have 20 seconds. What did you tell Congress in your testimony?

CONNOR LEAHY: Basically, exactly what I told you. I think it’s very important to understand, is that the number one bottleneck is awareness. Most people I talk to in Congress have not heard about these issues and are often very concerned. So, the best thing that citizens can do is contact your lawmakers, inform them about these issues, tell them that you are concerned about this issue, and, you know, maybe tell them to talk to me.

AMY GOODMAN: Connor Leahy, U.S. executive director of ControlAI.

Coming up, we speak to James Muldoon. He’s author of two books. One is Feeding the Machine: The Hidden Human Labor Powering A.I. and Love Machines: How Artificial Intelligence Is Transforming Our Relationships. Back in 20 seconds.