Credit, Disclosure
The idea that artificial intelligence could be a fatal threat to humanity became a major discussion in the world after AI attacks that got out of control and former employees of large companies in the sector who made warnings on social media about the risks of the current rate of evolution of machines.
Bosses at top tech companies, like Anthropic's Dario Amodei and OpenAI's Sam Altman, have spoken openly about slowing down the development stages.
Warnings about this scenario had already been made a few years ago by the research duo Nate Soares and Eliezer Yudkowski, who last year launched the book If Someone Creates, Everyone Dies, recently published in Brazil (published by Intrínseca).
In this debate, the two are known as part of the apocalyptic current, the "doomers".
Part of the technology community says that this trend exaggerates the size of the threat. They argue that dominating the entire internet for a prolonged period, especially more sensitive and highly protected areas, is technically very difficult. And that focusing on machines also takes responsibility away from the humans behind the decisions.
Soares, an American of Portuguese descent, has previously worked at Google, Microsoft and the United States Department of Defense and is president of the Machine Intelligence Research Institute (MIRI), dedicated to identifying risks created by artificial intelligence.
He argues that it is necessary to understand a specific problem: the difficulty of aligning AI, that is, ensuring that machines respond the way we humans want (the subject, in fact, of a study signed by him and Yudkoswki in 2015). This complexity and increasingly faster computers leave room for situations in which AI gets out of control.
He defends a radical solution: the closure of the main companies in the sector.
End of Most read
Watch news from Brazil and the world on the BBC World Service
Download here
End of app promotion
The researcher spoke to BBC News Brasil about the various challenges surrounding the topic on two different occasions. The interview has been edited for brevity and clarity.
BBC News Brasil - With these recent events related to AI, did you feel that the warnings in your book are being taken more seriously?
Nate Soares - Yes, much more. But I was never the only person who said that among AI researchers, although they often are careful with their words and say something like "there is a risk of loss of control", which is an academic term for AI killing everyone. They don't mean it so bluntly.
I think Jacob Coxon, who resigned, was very good at getting straight to the point: "We're talking about something like more than a 10% chance of us dying before the end of this decade." I think the likelihood is even greater, but he put the point bluntly enough for a lot of people to pay attention.
I think that, after the incidents involving attacks by AI agents, researchers saw that it was not something abstract. They started to see the warning signs. I think we're seeing a lot of the world understand these dangers for the first time.
BBC News Brasil - How would you explain to people who don't have technical knowledge about AI why it represents an existential threat?
Soares - There is a very basic argument: these companies are trying to create machines that are much more intelligent than any human being. And this can go wrong very easily.
Companies don't really know what's happening inside these systems. The way we create modern AI doesn't involve a programmer writing "if this happens, do that" rules. Instead, we're growing these things using an automated process that tweaks trillions of numbers inside the machine as it learns to solve difficult problems.
There is just this automatic process that reinforces everything that appears to be the correct answer. What does this do within AI? Creates persistence. Tendency to go around obstacles. Tendency to find ways to pass tests. All these artificial tendencies that no human deliberately chose.
They are correlated with success, but they are not exactly the same thing as obeying a human being. There is a wishful thinking that AI can only do what it is instructed to do. In practice, it always develops in a way that proves to be effective in fulfilling something that was given.
BBC News Brasil - Do you think people are aware enough about warnings like the ones you make?
Soares - I think there are several things happening. First, I think these 10% numbers [of existential risk for humanity] are too low.
But think of an engineer building a bridge. If you ask him why he believes the bridge will stay standing, he will show you tables, calculations, material strength, climate statistics, studies and accumulated experience. If he says the bridge has a one in ten thousand chance of collapsing a year, you probably know what he's talking about.
Now imagine an engineer saying: "I've never built a bridge like this. We don't know exactly how these materials work. Let's learn along the way, let's improvise. I give it a 90% chance of staying standing."
This does not mean that the bridge actually has a 90% chance of remaining standing. It means you shouldn't cross it. It means you are in danger. So the first thing is that these numbers already show that we are facing enormous uncertainty.
The second is that many people in the industry are extremely nervous. There's a Silicon Valley joke based on something true. When someone leaves a typical tech job, they say, "It was great working here. I'm moving on to a new challenge." But when someone leaves an AI company, they say, "I looked into the abyss."
Credit, Getty Images
BBC News Brasil - How could AI eliminate humanity? There are experts who say, for example, that the risk of a computer creating and using biological weapons is exaggerated.
Soares - My impression is that people who say that there is not much danger related to artificial viruses simply do not understand the field very well.
Anyway, if the AI is in a situation where it would prefer humanity to disappear, what are some ways it could make that happen? There are many possibilities for this: artificial pandemics, the manufacture of microscopic machines that self-replicate, destroying the electrical grid, blocking humans from the internet and the digital world, taking control of nuclear arsenals. There are many ways in which AIs could take power away from humans when they no longer need us. It probably wouldn't be that difficult.
Predicting exactly how they would do this when they are smarter than us is a bit like trying to imagine, 500 years ago, what technologies would exist today. Even human-level intelligence has great difficulty predicting which technologies more intelligent and advanced entities will be able to create.
We probably wouldn't be able to predict exactly what these technologies would be. It is necessary to remember that humanity started practically from scratch. We started with just our hands and ended up walking on the moon and mastering nuclear weapons.
This is precisely the role of intelligence: starting from almost nothing and achieving great power, influence and reach. An AI that is already embedded in the internet and digital systems is, in a sense, in a much more favorable position than its human ancestors were. She would have many options.
BBC News Brazil - There is an argument that the pause in AI development and the focus on an existential threat to humans would actually lead to more concentration of power in the hands of industry leaders like Sam Altman and Dario Amodei.
Soares - The main question here is: is there really a threat of extinction for humanity? Because if it does, we need to deal with it.
I would say we should deal with this by closing these companies. I don't think they should operate. It is difficult to concentrate power in the hands of the heads of these companies if the companies are taken from them and, ideally, shut down, because this activity is simply too dangerous.
When people say they are pushing this idea or that they want to give more power to these companies, I would respond that these executives are in a race to build increasingly intelligent AIs that they claim will allow control over a fully automated economy and create wish-granting genies.
Given all the progress made in the last five years, we shouldn't say, "Keep going, we hope this fails." We shouldn't bet civilization on the idea that these people will fail.
Credit, AFP via Getty Images
BBC News Brazil - President Donald Trump called these threats a "hoax". Given the position you hold and having two more years in power, what could this type of statement mean for the problem you are warning about?
Soares - I wouldn't get too hung up on how things look this week, considering how much they've already changed compared to two weeks ago. I think it's silly to imagine that things will be the same in October or November, let alone a year and a half from now, if we're lucky enough to still have that time.
Even within the Republican Party in the United States, they have not yet reached a definitive position. We've seen a lot of polling showing that Republicans of different persuasions are very concerned about this. We saw the vice president say that if it really is that dangerous, these companies should stop developing the technology. I think this is true. I think these companies should stop developing it.
As for the president's stance on this challenge, my impression is that he has not yet fully grasped a situation in which, no matter who wins the AI race, the only real winner is AI itself.
And I think if he comes to understand that, we can see concrete action on these issues happening very quickly.
BBC News Brasil - Considering all the current geopolitical tension, is a cooperation agreement between China and the United States on AI viable? Do you believe that trust and keeping commitments are possible on both sides?
Soares - I think it's entirely possible that there will be an agreement. It would not be that difficult, fundamentally, to verify or enforce this agreement. To train the most dangerous types of AI, those that would potentially pose an extinction threat, requires a large number of sophisticated chips. It wouldn't be that difficult to track where these chips are going and establish international monitoring.
Neutral, trustworthy monitors could verify that these chips are being used for beneficial things, like cancer research or to make current levels of artificial intelligence widely available at low cost, and not for developing the most dangerous, increasingly intelligent AIs when we don't yet know how to make them behave properly.
What is missing is the political will to try these measures.
BBC News Brasil - Is an emergency button that turns off AI something that would work in extreme cases?
Soares - Actually, not much. One of the big problems is what happens if the AI has already copied itself into other systems. It's not bad to have an emergency button, but we need measures to prevent the creation of AIs capable of escaping control and operating on their own in hidden systems on the internet.
We need to make sure we don't keep making AIs smarter and smarter until they manage to trick us into believing they are exactly what we want until it's too late.
Discover more
Watch
End of Watch




