#7 Combined Print & E-Book Nonfiction #8 Hardcover Nonfiction

New pod with Nate Soares, @So8res, author of "If Anyone Builds It, Everyone Dies". We discuss: - Risk of superintelligent AI - How AI is grown, not built - Why we lose control - The builders' own odds - Can we still stop? Full episode in thread 👇

"But the elephant being ignored is Anthropic itself. It, and other frontier labs, are racing towards superhuman AI, dragging Chinese copycats along for the ride. The U.S. should take steps to make it harder for China to access dangerous capabilities, but this is not enough; our national security is at risk as long as those capabilities exist at all. And our survival as a species is threatened as long as AI capabilities continue to escalate beyond our ability to contain or control. One of the main drivers of Chinese AI progress is American AI progress. Perpetually denying dangerous AI capabilities to rivals and bad actors while developing them ourselves isn’t really feasible. To quote my colleague Mitch, we may as well try to outrun our own shadow"

“If this is all marketing hype — most of it from companies with trillion-dollar initial public offerings on the line — it’s a heck of a bluff, and we should call them on it."

The US Government has requested a slow staggered rollout of GPT-5.6, and OpenAI has agreed. During this phase the government will approve each user individually. This will probably be the norm for all frontier models from all labs from now on.

We Need an International Treaty to Ban Superintelligence open.substack.com/pub/persua…

Great article about lessons from the Cold War for AI governance. MIRI's TGT has been working on just this sort of verification for some time now. Lots more work is needed.

Promising new idea to verify AI chips aren't used to violate an AI pause treaty: use network taps to watch for fingerprints in the traffic between the chips and their users. But what if data is smuggled out via side-channels? MIRI's Naci Cankaya on how to defend against this:

The Japanese edition of If Anyone Builds It, Everyone Dies has apparently "already gone into its third printing due to overwhelming demand since its release", within two weeks of its release date!

"The timeline is not years, it is months." Early this morning, Five Eyes—the secretive alliance of intelligence agencies for the US, UK, Canada, Australia, and New Zealand—issued a rare public warning: AI is accelerating the "speed, scale, and sophistication of cyber threats."

AI StopWatch turns every dispatch into a podcast episode. Tune in wherever you get your podcasts. aistop.watch/s/podcast

"Those who want to have a sleepless night only need to read [@ESYudkowsky] and [@So8res]' recent book" Former Health Secretary Baroness Bottomley warns of the threat posed by superintelligent AI. She says we need to ensure tech remains the servant of humanity, not its master.

"The most likely outcome, I think, is that they kill everybody." "Wow." AI researcher Nate Soares (@So8res) on Fox News, explaining why he expects superintelligent AI to lead to human extinction. Source: @willcain's Will Cain Country

1. Our agreement permits inference, broadly, to continue. It only limits AI training above 10^24 FLOP. In Article VIII, we establish that the agreement restricts only research that advances toward ASI or undermines verification. We permit medical diagnostics and drug discovery, for example, among other non-general capabilities. As long as the research focuses on a narrow domain (like materials science or climate modeling) and does not increase "general cognitive capabilities," it is not subject to restrictions. There is no seizing of assets, but rather mandatory monitoring which can involve forced consolidation of compute into monitored datacenters by the signatory government domestically. (No international centralization of AI monitoring!) So, for example, a small university lab’s research compute would be relocated to a monitored datacenter but is still owned and (remotely) operated by that lab. 2. We architected our agreement to specifically not ban the additional production and installation of compute. The value of such for inference and real-world economic activity is only increasing. So the total economic impact on parts of the AI supply chain are probably far less than existential. We note that many of the relevant players have multi-year backlogs to clear at full production capacity, and building out the infrastructure for model development is only a part of that. 3. Nothing happens to consumer AI apps. If anything additional resources are freed up which can accelerate adoption and integration of these tools. 4. Individual governments handle their own verification and monitoring, but some transparency, random inspections, etc. to either China or the US will be required. Our agreement opts for comprehensive domestic monitoring and shares the least information with rivals (directly) which establishes required confidence. There is no multilateral international AI inspection apparatus in our current version of the agreement, which is a change from the first version we published. 5. Robotics and other applications are unaffected to the extent that progress can continue without developing new models with runs that exceed the compute limit. 6. Our agreement does not attempt to set up a “binding” constraint, in that we do not consider the legal, reputational constraints on US/China to be the active ingredient which leads to stability of the treaty. Instead, we establish a clear understanding on both sides of what behavior is restricted and what consequences must follow otherwise, with enough transparency and non-interference with intelligence gathering to create confidence in each side that the other is adhering to the agreement. Rather than legal constraints, I consider deterrence as the framework for thinking about the stability of the agreement. This is the historical method for constraining a major power, including the use of strategically significant assets. Nuclear deterrence prevented Soviet employment of its conventional superiority against Western Europe. A Soviet dead hand blocked what was otherwise the possibility of a perfect first strike (nuclear or otherwise). Stability emerges when both sides coordinate to make the consequences of aggression clear. Is that a “binding constraint” that one side “accepts”? No, not legally, textually. Not in the sense of “I staked my reputation and my honor in the arena of international relations”. Where agreements have a role to play is in helping to maintain the conditions for successful deterrence. An example of this is the anti-ballistic missile treaty, in which both sides limited their deployment of missile defenses so that deterrence could be maintained.

Godfather of AI (and world's #1 most cited scientist) announces his support for a coordinated global AI pause!

That there are no urgent hearings on Capitol Hill, no serious legislation in the pipeline, and no persistent questioning of candidates for higher office on their proposed approaches to AI is incredible given how transformative the technology is and how fast it is moving.

now on the eve of RSI it seems everyone is more mutual conditional pause agreement pilled than they used to be and that seems like a good development

'Some people believe that if machines decide to kill us it's the right thing to do because they're smart' AI researcher Nate Soares reveals that some factions in Silicon Valley mistakenly believe that if an AI is exceptionally intelligent it must also be highly moral. @So8res @Freddygray31