#7 Combined Print & E-Book Nonfiction #8 Hardcover Nonfiction

Interesting poll of Hill staffers from @PunchbowlNews. 250 years is a long time! But interesting to see that "losing control of AI" is top 4 for both parties' staffers.

I'm thinking about this more this morning, and the current state of AI policy discussion reminds me of China Miéville's _The City and the City_. That book is set in a city, variously called Beszel or Ul Qoma, where two cultures overlap in physical space, but have elaborate purity rituals requiring them to ignore each other, to pretend that the other city's not there. Both cultures have common overlapping areas, where you have to "unsee" -- pretend you don't smell the other's cooking, hear the other's music, see the citizens of the other city walking by in strange clothing styles. Citizens are only supposed to interact by crossing through Copula Hall, a facility that culturally and legally is the crossing point between counting as being in one city, or the other, flipping you into the other, giving you permission to engage the other. Any interaction between the cities without permission is "Breach," the ultimate taboo, punishable by being disappeared from both cities. AI policy is, perhaps, walking between The City and the City. In DC, you wear a suit, even in the sweltering heat of summer. You treat AI as something that's a challenge to be forced into existing frameworks, managed just like a hundred other crises that your office has managed before. You assume that AI will transform jobs, replace some jobs, but that people can be retrained, organizations reformed, over the next 10 to 20 years. You tend to ask questions about OpenAI and Anthropic's business models by assuming that they're trying to sell enterprise software contracts to large corporations and the government. You think the national security problem is export control, or cybersecurity risks. You're not worried about us losing control to AI; if you're adventurously flirting with foreign food, you might say you're worried about "concentration of power" from AI, not understanding that is just the appetizer to the main course of losing control to AI. Your sharpest leaders are saying they want to use AI for everything, but you're skeptical it will really matter; your computer still takes 15 minutes to boot up every morning. In the Bay Area, you wear a t-shirt and jeans or Outlier Slim Dungarees (unless you're at Burning Man). You treat AI as the only story that matters. You assume that AI is going to replace you at YOUR job sometime soon. Your definition of the long term is "more than 12 months away." You ask your friends getting married whether they intend to have children "before or after the singularity." You're panicked -- even if you work at OpenAI, even if you think it's fairly unlikely -- about the human race losing control to AI or going extinct. You tend to assume that the government is a slow, distant, lumbering thing, not aware that you've _finally caught the attention_ of the fastest-moving, sharpest folks. You want government to treat this as the most important problem in human civilization; you doubt it will. You have no clue how to talk to each other well, in each other's languages. At least, not yet.

Why did MIRI fail to solve AI alignment, and what should humanity do now? I spoke with Nate Soares (President of MIRI) about superintelligence, extinction risk, P(doom), global governance, and why his proposed solution is simple: stop the race before we cross the point of no return. Watch the full interview on The Roman Forum and don’t forget to subscribe!

Six MIRI papers have been accepted to this year's @taig_icml!

"The idea that a company could rival a nation may seem farfetched, but even setting aside the potentially gameboard-flipping implications of AI, there is historical precedent. Mukunda draws a parallel to the British East India Company, chartered at the turn of the seventeenth century, which “became a private corporation that governed an empire.”

New pod with Nate Soares, @So8res, author of "If Anyone Builds It, Everyone Dies". We discuss: - Risk of superintelligent AI - How AI is grown, not built - Why we lose control - The builders' own odds - Can we still stop? Full episode in thread 👇

"But the elephant being ignored is Anthropic itself. It, and other frontier labs, are racing towards superhuman AI, dragging Chinese copycats along for the ride. The U.S. should take steps to make it harder for China to access dangerous capabilities, but this is not enough; our national security is at risk as long as those capabilities exist at all. And our survival as a species is threatened as long as AI capabilities continue to escalate beyond our ability to contain or control. One of the main drivers of Chinese AI progress is American AI progress. Perpetually denying dangerous AI capabilities to rivals and bad actors while developing them ourselves isn’t really feasible. To quote my colleague Mitch, we may as well try to outrun our own shadow"

“If this is all marketing hype — most of it from companies with trillion-dollar initial public offerings on the line — it’s a heck of a bluff, and we should call them on it."

The US Government has requested a slow staggered rollout of GPT-5.6, and OpenAI has agreed. During this phase the government will approve each user individually. This will probably be the norm for all frontier models from all labs from now on.

We Need an International Treaty to Ban Superintelligence open.substack.com/pub/persua…

Great article about lessons from the Cold War for AI governance. MIRI's TGT has been working on just this sort of verification for some time now. Lots more work is needed.

The Japanese edition of If Anyone Builds It, Everyone Dies has apparently "already gone into its third printing due to overwhelming demand since its release", within two weeks of its release date!

"The timeline is not years, it is months." Early this morning, Five Eyes—the secretive alliance of intelligence agencies for the US, UK, Canada, Australia, and New Zealand—issued a rare public warning: AI is accelerating the "speed, scale, and sophistication of cyber threats."

AI StopWatch turns every dispatch into a podcast episode. Tune in wherever you get your podcasts. aistop.watch/s/podcast

"Those who want to have a sleepless night only need to read [@ESYudkowsky] and [@So8res]' recent book" Former Health Secretary Baroness Bottomley warns of the threat posed by superintelligent AI. She says we need to ensure tech remains the servant of humanity, not its master.

"The most likely outcome, I think, is that they kill everybody." "Wow." AI researcher Nate Soares (@So8res) on Fox News, explaining why he expects superintelligent AI to lead to human extinction. Source: @willcain's Will Cain Country