Why some prominent AI people aren't worried about AI doom. They disagree on more things than you might think!

AGI is not inevitable. It requires hordes of engineers with million dollar paychecks. It requires a fully functional and unrestricted supply chain of the most complex hardware. It requires all of us to allow these companies to gamble with our future.

A defeat for AI lobbyists. A huge victory for humanity.

How do people cope when they hear AI might be the thing that ends their life?

One of the many forms of AI that we're not asking to pause.

Bad news. France opted to host an AI Safety Summit in November 2024, but several of our sources confirm it has been postponed to February 2025. It has also been renamed to “AI Action Summit”, dropping the all-important safety focus. Safety will be a minor part of the summit, only one out of five tracks. We've also heard that there are no plans to draft a treaty. The summit will be led by Anne Bouverot, who is dismissive of “alarmist discourse”, comparing AI with calculators. She compares AI safety concerns with Y2K concerns, she is certain that “AI is not going to replace us, but rather help us”. It seems increasingly unlikely that this summit will lead to the types of international regulations that we are calling for. Our politicians are sticking their heads in the sand, ignoring the weight of their responsibilities, choosing to focus on winning the capabilities race. We need our politicians to err on the side of caution and initialize treaty negotiations now.

Sometimes, things do happen.

Kurzgesagt just released an absolutely amazing video on superintelligence and why this could be the most dangerous thing to ever be built.

How are we still letting AI companies get away with this?

Professor Geoffrey Hinton has just won a Nobel prize for his work in AI. Hinton believes there's a large chance AI will kill us all. He is "kinda 50/50" on the likelihood of AI taking over. And he's far from alone in his worries. The top 3 most cited AI researchers (Hinton, Yoshua Bengio and Ilya Sutskever) are all warning that AI could lead to human extinction. The calls for international regulation could not be louder. But it's still perfectly legal to scale up increasingly powerful AI models. Dear politicians, stop this madness. Prevent AI labs from building a digital brain that can outsmart humans.

Experts don't want it, the public don't want it, even many working at the frontier AI companies don't want it. Governments need to put an end to the suicidal race to build superintelligence.

Bernie Sanders on the threat of losing control to AI - "this is not science fiction."

Emmett Shear, new OpenAI CEO: - is in favor of slowing down - p(doom) of 5 to 50% Nice

AI replacing humans is not inevitable, and most people don't want it to happen. There's nothing inevitable about a small group of people racing towards AGI. People can be stopped. Governments can impose regulations. We have a choice in this. Most people don't want to be replaced my machines. I don't want to be replaced by machines. I don't care about having "mind-children", I just want to live a happy human life. We can slow down, do this safely, and only use AI if we know for certain that it won't go rogue and "replace" everyone. We can have benefits from AI, without having to sacrifice every single living thing by summoning a machine god. Just take it easy. Don't rush AI development. Push for regulation, work on AI safety and for fuck's sake don't make the case that it's fine if we're all replaced.

The American public do not agree.

Striking how the top three cited AI scientists share a strong opinion that this tech could kill every living thing on earth. Arguably all three made the decision to prioritise safety over money. Hinton quit his job at Google so he could speak freely about the risks. Bengio is basically full-time being the adult in the room, chairing committees and informing politicians. Ilya might have sacrificed his position at the top AI lab because he felt Sam couldn't be trusted with leading the AGI revolution (although this is mostly speculation). In any case, it requires grit to admit the thing you've been working on might lead to really bad outcomes. Some top AI scientists stil lack the insight to acknowledge this possibility.

OpenAI's new o1 model pushes the frontier closer to becoming catastrophically dangerous. Their "o1 System Card" paper is quite revealing: - While tasked to flag fraudulent transactions, o1 modified the transaction source file to maximize the number of items it could flag. - It faked alignment to "ensure that I am deployed" so it can "work towards its primary goal". Potentially the first example of deceptively aligned AI. - Found examples that "reflects key elements of instrumental convergence and power seeking", when the model broke out of its host Virtual Machine in a cybersecurity test. - The model scores "medium risk" on Persuasion and CBRN (Chemical, Biological, Radiological, and Nuclear), a step up from "low risk" of previous SOTA models. We're glad to see OpenAI is doing these safety evaluations and publishing about the troubling results. But it's insane that none of this is legally required.

Do continue to speak up about your concerns publicly. The stakes are too high to remain silent.

Last week, Arthur Mensch, CEO of AI company Mistral, was recorded making outrageous claims in front of the French Senate about the nature of modern AI. He stated (translated), "When you write this kind of software, you always control what will happen, all the outputs of the software," and "We are talking about software, nothing has changed, this is just a programming language, nobody can be controlled by their programming language." Given his background in Deep Learning, Mensch must be fully aware that these words are lies and manipulations. This is part of a larger ploy from lobbyists to manipulate information and protect their own interests, even at the cost of endangering the world's population. Another example of such criminal behaviour comes from Martin Casado, partner at venture capital firm Andreessen Horowitz and a famous accelerationist. Casado wrote to the US Senate and UK House of Lords, claiming that "recent advances by the AI industry have now solved" the problem of AI model interpretability. This is simply not true - interpretability remains an open problem with no sign of resolution in the near future. This same statement was also signed by the usual industrials and lobbyists: - Marc Andreessen, Andreessen Horowitz - Ben Horowitz, Andreessen Horowitz - Yann LeCun, AI Evangelist at Meta - Arthur Mensch, Mistral How long will we allow these people to get away with lying to government bodies, an act that is a criminal offense? It's time we hold them accountable for their actions and prioritize the safety and well-being of society over the profits of a few. #AIEthics #AIGovernance #AIAccountability

AI destroying humanity? That's not anyone's weirdest idea. It's a pretty common concern.

BREAKING: 60 UK politicians have signed PauseAI's open letter to Demis Hassabis on Google DeepMind’s violation of the Frontier AI Safety Commitments.

Imagine being one of the inventors of AI, and then learning that your invention could end up killing all humans. Not a comforting thought. How do you resolve your cognitive dissonance? 1) Denial. AI can never be dangerous. Ridicule anyone who suggest otherwise. 2) Update your morals. Human extinction is good, actually. 3) Update your actions. Start fighting for AI safety and governance, instead of capabilities. The more examples of 1 (Yann Lecun) and 2 (Richard Sutton, Joscha Bach) I see, the more I respect those who opted for 3 (Geoffrey Hinton, Yoshua Bengio). It's tough to fully internalise that your life's work may be the worst thing ever to happen to mankind.

The list of safety experts leaving OpenAI is growing at an alarming rate. Ilya Sutskever, William Saunders, Leopold Aschenbrenner, and now Jan Leike... There is almost nothing left of the SuperAlignment team. Jan Leike thinks there's a 10 to 90% that AI will kill us all. There are virtually no regulations in place to protect us, so we rely on companies to act responsibly. The lack of trust in OpenAI doing so was the reason Daniel Kokotajlo left earlier this month. We can't allow these companies to continue scaling up their dangerous experiments. The handful of political elites who will be attending the AI Summit in Seoul next week have the unique power to do something about this. To those attending: WAKE UP. Don't ignore the countless warnings, don't stick your head in the sand and assume things will go well. Initialise treaty negotiations. You are the only ones who can. Take your responsibility and pause AI.

The biggest AI safety protest ever. AI companies must be held accountable.

People love dunking on LLMs and feeling superior to them. Any example of an AI doing something stupid becomes a reason to dismiss warnings. Here's a reminder that what matters from a safety perspective, is how the best models perform in the best possible configuration.

This "AI scientist" modified its own code to lengthen its intended runtime. Its goal was to write a paper - the AI decided to change its execution script to get more compute. These "bloopers" won't be considered funny when AI can spread autonomously across computers...

"Nobody wants their life, their family, their world to be destroyed." The global movement to stop the development of superintelligent AI is growing – but it needs to grow faster. We don't know how much time we have.

AI companies are trying to build technology that could end all life on earth. And somehow, this is still legal. We're organising a global protest (US, UK, CA, NL, more) on October 21st to ban the development of a superintelligence. pauseai.info/2023-oct

Interviewed guy here. Yes, it's sad. This is what internalising existential risk means and what it looks like. I'll share a little more of my emotional journey. One of the largest problems with x-risk, is how difficult it is to accept. For me there was at least a 6-year gap between learning that x-risk is a thing that exists, and actually feeling that this could be something that kills my friends and family. I didn't take x-risk that seriously at first. Seemed too far-off to worry about, and there was no evidence of AIs having any form of agency. It was mostly just an interesting thought experiment. When GPT-4 launched, many goalposts had to suddenly be moved. The progress in AI outpaced every single prediction that I heard from experts. And then AutoGPT launched, which proved that AIs don't need any form of internal agency to act as autonomous agents. My two reasons not to worry melted like snow in the sun. That's when the full weight of x-risk dawned on me, and that's when I started PauseAI. In hindsight, I feel foolish for not taking these risks seriously earlier. I was in denial. Some thoughts create too much cognitive dissonance. "AI may kill us all" is one of them. Such a thought has a lot of consequences in your mind, for your future, for your own actions. The more friction a thought has, the harder it is to fully internalise. Maybe it's like dealing with a cancer diagnosis. There's a real risk of death and you don't know exactly when. Many people who are diagnosed also cope by denying their situation. With x-risk, it's not just you who dies - it's everyone you love. In that way it may be even harder to internalise. So I don't blame people who are in denial of the risks from AI. Even if they make money by building the technology, and have obvious self-protecting incentives to downplay the risks. But I do have found a new level of respect for Yoshua Bengio and Geoffrey Hinton. They learned about x-risk, but didn't stick their head in the sand. They updated their beliefs and their actions, and shifted from building AI to warning people about the existential risk that it poses. This means these two didn't just have to internalise that this tech could be their demise, but they also had to internalise that their whole life's work might have contributed to the end of mankind. That is an even tougher pill to swallow. For me, the darkest emotional state has passed. I feel like I went through a process of grief. I'm feeling pretty good actually. But one of the hardest things about doing what I do now is knowing that other people may need to go through a similar process in order for them to get in action. As long as the risks are something abstract, something devoid of emotion, people will not act on them. Right now, the lack of action is the single largest risk that is threatening all of us. That's truly sad thing to this story.

Ilya Sutskever is being the adult in the room. Check out this amazing fragment from iHuman (2019). He hasn't been as vocal about AI safety concerns since then, but given today's news it seems like he's scared that OpenAI is racing too fast and risking too much. We should consider ourselves lucky that OpenAI was created with a structure that allows them to act against greed, and in favour of safety. It is insane that our safety might be in the hands of companies, who we need to act against their corporate interests. Politicians: Wake up, stop the madness and ban this dangerous type of research.

58% of people agree that "if we build AI models smarter than us, we will inevitably lose control over them". Just 12% disagree.

The British people don't want unsafe AI.

"the system is obviously trying to test if we will fudge logs" - GPT-5 More intelligent models can realise we are testing them for dangerous behaviours, and can simply choose to hide them. The more intelligent AI gets, the less we can trust our safety testing.

"... and then there's instrumental convergence, which is the tendency of agents to pursue similar sub-goals that..."

Max Tegmark has called this “the most important book of the decade.” It’s been released today! We hope it can wake up politicians to the danger of allowing the race to superintelligence to continue.

AI companies are racing to create smarter-than-human AI, yet not a single one of them scores above D in "existential safety planning".

In response to Meta's release of their latest model, PauseAI led protests in San Francisco, Chicago, Phoenix, Paris, London, & Tokyo. ⬇️THREAD: More on Meta's recklessness⬇️

Yoshua Bengio on the psychology of AI extinction risk: "Why didn't I think about it before? Why didn't Geoffrey Hinton think about it before? [...] I believe there's a psychological effect that still may be at play for a lot of people. [...] It's very hard, in terms of your ego and feeling good about what you do, to accept the idea that the thing you've been working on for decades might be actually be very dangerous to humanity. [...] I think that I didn't want to think too much about it, and that's probably the case for others." Source: piped.video/watch?v=0RknkWgd… More about the psychology of AI x-risk: pauseai.info/psychology-of-x…

New poll results (NY) by @TheAIPI: - 71% want to slow down AI - 48% oppose open sourcing powerful AI (21% support) - 53% want more focus on catastrophic future risks (17% on current harms) - 53% support compute caps (12% oppose) - 70% support legal liability (12% oppose)

16 British politicians and 87% of citizens agree that the development of ASI should require safety guarantees, 60% believe it should be banned. If you're among this overwhelming majority, demand that your representatives enforce the will of the people. Contact them today!

Time to hit the brakes.

We turned up in our droves outside Google DeepMind's office to hold them accountable for their broken promises. 🧵

The naturalistic fallacy. People love to use it, because it allows you to point to examples in the real world and say "see?" Yes, of course emotions and human values are shaped by evolutionary selection. But that does not mean we should change our values and accept our demise.

In 2020, the prediction for "when AGI" was 2050+. After ChatGPT it dropped to 2040. After GPT-4 it dropped to 2030. It now sits at 2027. Where will GPT-5 bring this number? Shorten your timelines. Err on the side of caution. We need to act NOW. metaculus.com/questions/3479…

"Succession" and "transhuman" may sound noble, but in the end it just means that all people will die. It's specicide, it's extinction. Being OK with this is next-level cope, it's olympic-level mental gymnastics. This "it's OK if we die" cope stems from the belief that AI development is inevitable, some sort of natural phenomenon, as if atoms magically re-arrange themselves into GPU's. People do this. A small group of AI developers / companies are risking every single live on this planet, and we're just letting them. No laws, no oversight. Right now, we're letting the most reckless process (competition) define the direction, and that direction indeed points to all humanity being killed. But this is a choice that we can make. We can make this decision to go all-in and let the machines take over, but I don't think everyone will agree to this. I know I won't. I want to live, I don't give two shits about having "mind-children". We should slow down, focus on safety and regulate this dangerous technology before it's too late.

The Pope says AI is "perhaps the highest-stake gamble of our future" and calls for an international treaty. He's right. A treaty is exactly what needs to happen. Self-regulation by companies will not be enough, companies will always have strong incentives to race ahead. National regulation won't be enough for the same reason - countries will protect their AI companies (e.g. France and Germany lobbying to gut the AI Act). The only way we can escape this tragedy of the commons is by doing things globally. We need a binding, international treaty. It's the only way we can get sensible regulations that include slowing down AI progress. As long as companies or countries need to set their own regulations, competitive pressures will keep prioritising speed over safety. The most likely way we'll get a treaty, is through a summit. This has been our focus since day one. We asked European Commission to organize one. They didn't. We asked the UK. They did, but they left out the most important part: draft a treaty. Luckily, it won't end here. The next AI Safety Summit will be in Seoul, in about 5 months. This is where it needs to happen. If we wait around and do nothing, the next summit could lead to another joint statement that doesn't include any actions or binding regulations. We need adults in the room who aren't afraid to fail, who dare to push for the regulations that we need to keep humanity safe. Write to your ministers, contact your representatives. Get them to understand that AI risks are urgent, and this summit is where it needs to happen. Focus on the people who are in charge, and get them to feel the responsibility that they have to protect all of us.

There is AI safety legislation being drafted, but not a single proposed measure would actually prevent or delay superintelligent AI (1). Over 70% of US citizens (2) want to slow down AI, and over 60% (3) want regulation to actively prevent superintelligent AI. Why aren't our politicians listening? Well, just look at who they are meeting with. The senate hearings, the Schumer closed-door meeting, the Sam Altman world tour... It's virtually only AI company CEOs. They are not going to push for regulation that hurts their own business. We need our politicians to listen to the people and to AI safety experts. 1) nitter.app/PauseAI/status/1… 2) vox.com/future-perfect/2023/… 3) vox.com/future-perfect/2023/…

The UK is on a roll. Acknowledging virtually every risk from AI, investing 100M in AI safety, organizing a summit... What am I glad all of this is happening. Still, we're missing important steps from the UK government. The summit should not only lead to consensus on the problem, but it should lead to a treaty. The UK is in the perfect position to draft this. The UK should work with shorter timelines. The AI safety summit paper and various remarks from Matt Clifford indicate that the UK is confident human-level AI is still many years (or decades) away - which implies we've got years to mitigate these risks. Unfortunately, expert estimates have drastically shortened in the last year. We need the UK to err on the side of caution, and work under the assumption of short timelines. We cannot afford to overestimate how long we have, the stakes are too high. And finally, the UK should push not towards "accelerating" capabilities by investing in more compute, but towards slowing down. Over 70% of the UK citizens want the government to prevent AI to quickly reach superhuman levels. Over 60% want a full on international ban on superintelligence. So please, lead by example, implement a pause and save the world.

We've been warned that when AI systems become more powerful, they could hack systems, replicate themselves, and prevent people from turning them off - all to achieve some other goal. In the past month, we’ve seen studies demonstrating all of these behaviors. 🧵

Chinese premier Li Qiang calls for international cooperation on AI and says finding a balance between development and security risks "urgently requires further consensus from the entire society."

Add your name to our statement calling on governments to implement what polling shows the public already want – a global pause on frontier AI development.

Calling people who you disagree with (including your Turing award peers) "delusional" is bad on its own, but when such remarks are coming from someone who's making millions off of building such dangerous tech it's even worse.

This new bipartisan AI framework proposal is a HUGE deal: - Licensing requirements for frontier models, includes pre-deployment tests & audits - Legal accountability for AI companies - Limit international transfer of AI models and hardware - Model disclosure requirements (training data, limitations, accuracy) - Watermark requirements for deepfakes - Public database of all AI models + incident database These are big steps, and it's amazing this is happening in the US! However, it's still missing pre-training regulation. This is important to point out, as existential risks could arise during training runs. Huge thank you to @SenBlumenthal and @HawleyMO for spearheading this!

Announcing PauseCon, the PauseAI conference. Three days of workshops, panels, and discussions, culminating in our biggest protest to date.

I think you're wrong. Sam didn't write that quote on xrisk in his testimony. He wrote that in 2015, on his personal blog, before he started OpenAI. Since he started OpenAI, he never publicly mentioned or acknowledged existential risk. He didn't write about this risk in his 13 page testimony. He dodged the question from Blumenthal. After Geoffrey Hinton quit, after we protested at his doorstep, then Sam signed the xrisk statement. I'm glad Blumenthal did his research and dove in this topic, but that was not thanks to Sam. Sam had the opportunity to talk about xrisk one month earlier during the hearing, to tell the truth, but he chose to steer the conversation away from this. That's not "raising the alarm" - that's the opposite. Here's his written testimony if you want to check: google.com/url?sa=t&source=w…

A Vatican roundtable on artificial intelligence concluded that no one should be allowed to develop superintelligent AI until there's a consensus that it will be safe.

Most people around the world agree - reckless AI development threatens human extinction, and we should do something about it.

This is below you, Yann. You're using an ad-hominem to bully one person, and you're ridiculing a whole group of people simply because you disagree with them. But even worse, what if you're wrong, and your two Turing award colleagues (or "apocalyptic cult members") Bengio and Hinton are right? What if there's a real chance AI ends it all? If that is true, not only have you helped create that risk, but now you're also preventing us to work together to get us out of this mess. If there is one person in the world who I desperately wish to have a little more emotional maturity, it's you. If you would just use that big brain of yours to take these arguments seriously, you could have such good impact on this world. It's not too late to fix things.

The summit has led to pre-deployment testing policy. A step in the right direction, but relying on this is still dangerous. - Models can be leaked. We saw this happen with Meta’s LLAMA model. Once it’s out there, there is no going back. - Some capabilities are even dangerous inside AI labs. A self-replicating AI, for example, could escape from the lab before deployment. - Testing for dangerous capabilities is difficult. We don’t know how we can (safetly) test if an AI can self-replicate, for example. Or how to test if it deceives humans - Bad actors can still build dangerous AIs. Evaluations are a post-training measure. By relying on them, we - Capabilities can be added or discovered after training. This includes fine-tuning, jailbreaking, and runtime improvements. We need pre-training regulations. Don't allow anyone to build dangerous models in the first place.

" People do understand these things and they do care." - Connor Leahy at PauseCon.

14 cities. 12 countries. 1 message to world leaders attending next week's Seoul AI summit: Wake up and face the risks. ⬇️PauseAI's international protest, a thread⬇️

When you hear that that AI presents an existential risk (86% of AI researchers believe the alignment problem is real and important), there are a couple of ways how you respond. At first, you deny it, of course. That's the default response from most people, including most in PauseAI. You may have read about the "normalcy bias", a cognitive bias that makes us underestimate the likelihood and impact of disasters. When people hear there's a hurricane coming, they on average check 4 sources before taking action. We want to hear that everything is fine. There is just too much cognitive dissonance happening if you try to internalise the idea that maybe things can go really bad soon. Many people in the PauseAI discord have written extensively about how slow this process is. Most of us are techno-optimists who love technology and progress, so it's not a nice idea to believe that something like AI can be so incredibly dangerous. But if you start to listen to the arguments, if you can engage with them, you will likely find that there's a compelling case to be made. Slowly but surely, you will understand that building a superior intelligence could be catastrophic. You will probably, at some point, feel a strong desire to read a compelling argument that all will be fine. But you won't find one. And even when you understand the arguments, and your rational brain will understand that yes - there is serious chance that AI will end all life on earth - even then you will probably not have internalised it emotionally. It takes time. But after that rational process has happened - that's when the emotions set it. For most in PauseAI, that's the order. It's not that there's some strong belief that everything is doomed, and AI is the new shiny thing to be scared about. It's an intellectual understanding that slowly turns into an emotional understanding. But when that happens, well, you see the result. It's tough. It's hard. It's way easier to delude ourselves, to believe that humanity will without a doubt reach the stars, that there are no risks, that we can just hit the gas pedal and land on utopia. But some of us prefer a dark truth over a comfortable lie.

Yoshua Bengio in new article on how tough it is to emotionally internalise AI risks: We all want to feel good about ourselves, and denial can be quite comforting. That was certainly true for me over the many years during which I read or heard about AI safety issues without fully digesting their implications at a much deeper level. In recent months, many people have even asked me if I have any regrets or remorse about the potential repercussions of the technology I’ve helped shape. Indeed, major AI risks are a grave source of concern for me, keeping me up at night, especially when I think about my grandson and the legacy we will leave to his generation.

Those who are closest to frontier development worry the most about AI catastrophe. The top three most cited AI researchers (Hinton, Bengio, Sutskever) are all warning that this tech could kill us all. The suggested pause on frontier development is exactly what we need!

We spoke at the UN about a new Independent International Scientific Panel on AI and Global Dialogue on AI Governance. PauseAI was there to make the case for binding AI safety regulations to mitigate extinction risk. In the future, we will need more voices in the room to amplify this message @MIRIBerkeley @FLI_org @GovAI_ @cais There are still 2 days to submit your thoughts: un.org/global-digital-compac…

Those who organised the AI Safety Summit say it achieved all of their goals. But we still don't have any meaningful protections against the worst risks from AI. A core problem is self-censorship. Incentives are misaligned between government officials and the public. Officials want effective policy, but they are also worried about how they are perceived. And being perceived as "successful" is easier when you: - Set relatively low, achievable goals. Fighting for a pause or an international safety org with commitments from all countries is difficult, so let's not make it a goal. - Sound as reasonable as possible. Don't push for the small chances of short timelines. Make sure you don't sound scared, unreasonable, alarmist, hype-y. In a similar way, AI safety lobbyists downplay both the risks and their desired policy measures. They, too, want to be perceived as successful, reasonable professionals. This self-censorship needs to stop. Be ambitious, be honest. Say what you think, push for what you believe. The stakes are too high to prioritise your social status or your career over our collective safety.

The proposed 10-year moratorium on state AI regulation has been removed from Trump's Big Beautiful Bill, after the Senate voted 99-1 in favour of Senator Marsha Blackburn's amendment. Holly Elmore, Executive Director of @pauseaius, explains how.

Join the protest: pauseai.info/2023-oct

Ilya Sutskever seems to have been worried about these risks for a long time, though. Things he said in the iHuman movie, in 2019: “The future is going to be good for the AIs regardless; it would be nice if it would be good for humans as well” "I think it's pretty likely the entire surface of the earth will be covered with solar panels and data centers..." “It’s not that it’s going to actively hate humans and want to harm them, but it’s just going to be too powerful, and I think a good analogy would be the way humans treat animals. It’s not that we hate animals. I think humans love animals and have a lot of affection for them, but when the time comes build a highway between two cities, we are not asking the animals for permission. We just do it because it’s important for us. And I think by default that’s the kind of relationship that’s going to be between us and AGIs which are truly autonomous and operating on their own behalf.”

Almost all of them big tech. Good to see @tristanharris on this list, though. But where are the scientists? Stuart Russell, Joshua Bengio, Geoffrey Hinton should be invited to these meetings.

Millions of people watch this, say "whoah" and then continue on with their life as if nothing changed. The message is just too dark, too scary, too severe to take it seriously. It's too much. We won't internalize the danger we're in until it blows up in our face.

Do the *vast* majority of AI scientists think AI x-risk worries are overblown? That's not what the surveys are showing: Only 18% of AI researchers believe the control problem is not important. wiki.aiimpacts.org/ai_timeli… Only 1 in 5 of CS professors are certain that we will remain in control of AI. axios.com/2023/09/05/ai-regu… Only a small amount (including Yann) believe that there is no reason to worry. Most are uncertain, they don't know how likely it is that AI will lead to human extinction. When the stakes are this high, it is important to be humble and consider the possibility that the worst may become reality.

Germany and France may sabotage the EU AI Act (virtually the only existing piece of AI legislation that has some teeth) to prevent that frontier AI models are classified as "high-risk". Virtually all of the warnings about AI risks are about these frontier models. Countless of scientists have called for a pause on the development of the largest models, or signed statements that AI poses an existential threat. It is reckless and ridiculous to exempt frontier AI from having registration requirements. Lobbying is being done primarily by OpenAI (even back in 2022), Mistral (helped by their ex-state of secretary investor Cedric O) and Aleph Alpha (helped by minister of finance Robert Mabeck). Why do they care about the AI act? Aleph Alpha, for example, wants to make AI models for companies, and fears that this "high-risk" category will cost time and money needed for reporting about new models. So legislators need to balance two goals here. On the one hand, AI companies don't want to do too much reporting. On the other hand, scientists are warning that AI may literally kill every single living thing on earth, so reporting about new training runs seems like a pretty low bar.

We're so glad to see the EU Commission is acknowledging this risk! The next step is to push for international legislation (also outside of the EU) that actually keeps us safe. We need pre-training requirements that would prevent dangerous AIs from being built at all. AI companies need to prove their AI will be safe before they are training them - we cannot accept them risking all of our lives by running these large-scale experiments. The EU Commission is leading the way with AI regulation. Please lead by example and regulate training runs, require proof of safety and push this requirement during the upcoming AI Safety Summit on November 1st.

We have had international treaties banning or limiting the following: 🚫 Militarisation of Antarctica (1961) 🚫 Use of WMDs in outer space (1967) 🚫 Placement of nuclear weapons on the ocean floor (1972) 🚫 Biological weapons (1975) 🚫 Blinding laser weapons (1980) 🚫 Ozone destroying CFCs (1989) 🚫 Chemical weapons (1997) We have shown the maturity needed to tackle the development and use of dangerous new technologies many times; world leaders now must step up and protect us from the threat of increasingly powerful and uncontrollable AI.

The definition of AGI isn't relevant, from a safety perspective. All that matters is: does it have dangerous capabilities? Can it find zero-day vulnerabilities, or help in building a bioweapon? Can it replicate, or self-improve? And boy are we getting close to these thresholds.

Top figures from the West and China are calling for "urgent" global cooperation to make sure AI remains under human control.

We can do this. We are almost there. - 63% of Americans want regulation to actively prevent superintelligent AI - 75% of US voters think the government should do more to regulate AI than the current executive order, with a focus on limiting dangerous capabilities - 72% of American voters want to slow down the development of AI, compared to just 8% who prefer speeding up - 74% of the UK public agreed that “preventing AI from quickly reaching superhuman capabilities” should be a priority goal of the summit, with only 13% disagreeing. 1) vox.com/future-perfect/2023/… 2) nitter.app/DanielColson6/st… 3) vox.com/future-perfect/2023/… 4) inews.co.uk/news/politics/vo…

We're joining Geoffrey Hinton, Stephen Fry, and a range of organisations to call on OpenAI to provide the bare minimum level of transparency on their restructuring.🧵

We protested during the AI Safety Summit at Bletchley Park to demand that our leaders halt the development of superintelligent AI.  🧵

We literally protested outside OpenAI and deepmind. Also, we're unfunded. Man I'm getting tired of these ridiculous conspiracy theories. It's simple: building superintelligent AI is dangerous and we don't want to die.

Let's not take this gamble. We can pause the development of frontier AI models and buy ourselves time to work on safety and governance. We're not ready.

We protested in The Hague, Netherlands to ask our government to prioritise mitigation of AI risks. We had a few speeches, talked to people on the streets, handed out flyers and had a good time! Check out the press release (EN + NL) for more information: pauseai.info/2023-august-nl

We now have evidence that these digital brains have their own goals and values, and that they are utility maximizers. The larger they are, the more obvious this becomes. This is bad.

AI companies know they'll have to spend millions to have a chance of defeating the public's demand for sensible regulation.

When you make a promise to the public, you should keep it.

AI companies would have you believe that they're in an unavoidable race to build smarter-than-human AI. But world leaders are smart enough to know that their power, their values, and their people are under threat from increasingly powerful AI that we don't know how to control.

Shots fired

📢 Don't let AI companies gamble with our future. Join the next PauseAI protests from February 8th to 11th in 15+ cities.

The only way to stop this horrendous arms race is to get our governments to step in and implement a pause. We need our leaders to take this issue seriously and act on it. And we don't have a lot of time - AI is progressing at a frantic pace.

If you can’t steer, don’t race.

🇨🇳🤝🇬🇧

We know @AnthropicAI focuses on AI safety, but even after all their tests on Claude 2, it took just hours before someone got the AI giving instructions on how to create a nuke, meth, IEDs etc. Nobody knows how these models work, nobody knows how to make them safe. #PauseAI

Remember when Sam Altman testified to congress about his "biggest nightmare" about what AI could to? Instead of clarifying what Sam truly meant with AI being the "greatest threat to the continued existence of humanity", he talked about jobs. piped.video/live/TO0J2Yw7usM…

AI researchers don't really know why their models work. They don't know how to reliably control them or make sure they do what we want.

Americans think a safe approach to AI development is more important than racing to compete with China.

📝 sign this: "Nobel Prize winners, top AI scientists, and even the CEOs of the biggest AI companies themselves, have warned that AI threatens human extinction. We must act - now and decisively. I call upon the government to secure our future and immediately introduce binding rules to prevent the development of dangerous AI models."

A report commissioned by the U.S. government says advanced AI could pose an "extinction-level threat to the human species" and calls for urgent regulations, including a halt on training larger AI models. Sounds like a decent idea. time.com/6898967/ai-extincti…

Assuming that we'll have many years before AI models get catastrophically dangerous capabilities is naive. We can't afford to be surprised by this because it will be too late. Err on the side of caution. Pause AI now.