Pinned Tweet
Some quick takes:
(1) Wow things are getting real.
(2) The government's order focusing on prohibiting transfer to foreign nationals (even e.g. those living in the US, our close allies who help evaluate model safety in the UK, individuals who work at frontier labs like Anthropic) seems remarkably destructive, though is partially a result of the government using older legal authorities that were not designed for this kind of technology.
(3) If you believe (as I do) that AI has profound ramifications for national security, then assuming the government will sit back and do nothing and tolerate explanations like "well jailbreaking is a hard technical problem" for cyber capabilities that used to be the crown jewels of the NSA, is not tenable. If this is how the government reacts to the current level of system capabilities in 2026, how do you expect them to react to whatever is possible in 2028? However, it is extremely important that the authorities that the government uses are legible, transparent, have opportunities for appeal, and are narrowly targeted. Those legal authorities do not currently exist, and in their absence, the government will reach for metaphorical sledgehammers instead of scalpels.
(4) For that reason, it's extremely important that we create regulatory structures that are transparent and give recourse in the event that the government is overstepping or acting in an arbitrary manner. The alternative to passing such laws is not no regulation, it is regulation left primarily to national security authorities that are increasingly and evidently not fit for purpose.
The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.
The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance.
Access to all other Claude models is not affected.
We apologize for this disruption to our customers. We believe this is a misunderstanding and are working to restore access as soon as possible.
Read our full statement: anthropic.com/news/fable-myt…
10
28
251
84,178
One Tuesday night, as my wife and I sat down for dinner, a sheriff’s deputy knocked on the door to serve me a subpoena from OpenAI.
I held back on talking about it because I didn't want to distract from SB 53, but Newsom just signed the bill so... here's what happened:
🧵
305
1,224
6,200
6,705,870
This is not normal. OpenAI used an unrelated lawsuit to intimidate advocates of a bill trying to regulate them. While the bill was still being debated.
7/15
3
95
1,642
207,050
You might recall a story in the SF Standard that talked about OpenAI retaliating against critics. Among other things, OpenAI asked for all my private communications on SB 53 - a bill that creates new transparency rules and whistleblower protections at large AI companies.
2/15
Inside OpenAI, there's a growing paranoia that some of its loudest critics are being funded by Elon Musk and other billionaire competitors. Now, they are going after these nonprofit groups, but their evidence of a vast conspiracy is often extremely thin sfstandard.com/2025/09/02/op…
6
90
1,568
447,396
Why did OpenAI subpoena me? Encode has criticized OpenAI’s restructuring and worked on AI regulations, including SB 53.
I believe OpenAI used the pretext of their lawsuit against Elon Musk to intimidate their critics and imply that Elon is behind all of them.
3/15
🚨 Breaking: A group of 100+ Nobel laureates, professors, whistleblowers, public figures, artists, and nonprofit organizations just released a letter asking OpenAI to tell the truth about its restructuring.
Here’s what they had to say: 🧵
9
99
1,389
420,444
A magistrate judge even chastised OpenAI more broadly for their behavior in the discovery process in their case against Musk.
9/15
3
53
1,373
146,583
There’s a big problem with that idea: Elon isn’t involved with Encode. Elon wasn’t behind SB 53. He doesn’t fund us, and we’ve never spoken to him.
4/15
7
53
1,361
189,142
Does anyone believe these actions are consistent with OpenAI’s nonprofit mission to ensure that AGI benefits humanity?
OpenAI still has time to do better. I hope they do.
15/15
42
62
1,341
76,553
OpenAI had no legal right to ask for this information. So we submitted an objection explaining why we would not be providing our private communications. (They never replied.)
8/15
4
44
1,327
141,095
But they didn’t stop there.
They also sent a sheriff’s deputy to my home and asked for me to turn over private texts and emails with CA legislators, college students, and former OAI employees.
6/15
9
62
1,281
244,701
OpenAI went beyond just subpoenaing Encode about Elon. OpenAI could (and did!) send a subpoena to Encode’s corporate address asking about our funders or communications with Elon (which don’t exist).
If OpenAI had stopped there, maybe you could argue it was in good faith. 5/15
2
43
1,143
184,188
Prior to OpenAI, Chris Lehane’s PR clients included Boeing, the Weinstein Company, and Goldman Sachs. One person who worked on a campaign with Lehane said to the New Yorker “The goal was intimidation, to let everyone know that if they fuck with us they’ll regret it”
12/15
4
59
1,049
100,158
I want to see that side of OAI, but instead I see them trying to intimidate critics into silence.
This episode was the most stressful period of my professional life. Encode has 3 FTEs - going against the highest-valued private company in the world is terrifying.
14/15
9
43
991
79,686
There is more I could go into about the nature of OAI's engagement on SB 53, but suffice to say that when I saw OpenAI’s so-called “master of the political dark arts” Chris Lehane claim that they "worked to improve the bill," I literally laughed out loud.
11/15
4
37
980
113,469
This wasn’t the only way OpenAI behaved poorly on SB 53 before it was signed. They also sent Governor Newsom a letter trying to gut the bill by waiving all the requirements for any company that does any evaluation work with the federal government.
10/15
3
30
963
122,866
I have complicated feelings about OpenAI - I use and get value from their products, and they conduct and publish AI safety research that is worthy of genuine praise.
I also know many OpenAI employees care a lot about OpenAI being a force for good in the world.
13/15
AI scheming is a particularly important risk we've been studying. More here:
openai.com/index/detecting-a…
2
29
758
130,953
I thought about writing a response to this post myself, but others have already been pointing out issues well enough.
If you are trying to make up your mind about whether what happened was normal or OK (including if you are someone at OAI), I encourage you to read this thread and decide for yourself. 🧵
There’s quite a lot more to the story than this.
As everyone knows, we are actively defending against Elon in a lawsuit where he is trying to damage OpenAI for his own financial benefit.
Encode, the organization for which @_NathanCalvin serves as the General Counsel, was one of the first third parties - whose funding has not been fully disclosed - that quickly filed in support of Musk. For a safety policy organization to side with Elon (?), that raises legitimate questions about what is going on. We wanted to know, and still are curious to know, whether Encode is working in collaboration with third parties who have a commercial competitive interest adverse to OpenAI.
The stated narrative makes this sound like something it wasn’t.
1/ Subpoenas are to be expected, and it would be surprising if Encode did not get counsel on this from their lawyers. When a third party inserts themselves into active litigation, they are subject to standard legal processes. We issued a subpoena to ensure transparency around their involvement and funding. This is a routine step in litigation, not a separate legal action against Nathan or Encode.
2/ Subpoenas are part of how both sides seek information and gather facts for transparency; they don’t assign fault or carry penalties. Our goal was to understand the full context of why Encode chose to join Elon’s legal challenge.
3/ We’ve also been asking for some time who is funding their efforts connected to both this lawsuit and SB53, since they’ve publicly linked themselves to those initiatives. If they don’t have relevant information, they can simply respond that way.
4/ This is not about opposition to regulation or SB53. We did not oppose SB53; we provided comments for harmonization with other standards. We were also one of the first to sign the EU AIA COP, and still one of a few labs who test with the CAISI and UK AISI. We’ve also been clear with our own staff that they are free to express their takes on regulation, even if they disagree with the company, like during the 1047 debate (see thread below).
5/ We checked with our outside law firm about the deputy visit. The law firm used their standard vendor for service, and it’s quite common for deputies to also work as part-time process servers. We’ve been informed that they called Calvin ahead of time to arrange a time for him to accept service, so it should not have been a surprise.
6/ Our counsel interacted with Nathan’s counsel and by all accounts the exchanges were civil and professional on both sides. Nathan’s counsel denied they had materials in some cases and refused to respond in other cases. Discovery is now closed, and that’s that.
For transparency, below is the excerpt from the subpoena that lists all of the requests for production. People can judge for themselves what this was really focused on. Most of our questions still haven’t been answered.
Community note
The subpoena request also asked for all communications involving SB 53. This is a bill unrelated to the lawsuit with Musk.
nitter.app/_NathanCalvin/
9
29
310
54,197
Is a subpoena asking for the attendees and eulogies from a teenager's funeral "to be expected"?
OpenAI has sent a legal request to the family of Adam Raine, the 16yo who died by suicide following lengthy chats with ChatGPT, asking for a full attendee list to his memorial, as well as photos taken or eulogies given.
His lawyers told the FT this was "intentional harassment"
8
29
252
21,076
Really appreciated these comments from Joshua Achiam (head of mission alignment at OpenAI, speaking in his personal capacity)
You don’t have to like sb 53 to realize this behavior isn’t good or helpful
At what is possibly a risk to my whole career I will say: this doesn't seem great. Lately I have been describing my role as something like a "public advocate" so I'd be remiss if I didn't share some thoughts for the public on this. Some thoughts in thread...
3
4
137
6,482
As you might have seen, OAI put out a response to this thread. More info on that here:
I thought about writing a response to this post myself, but others have already been pointing out issues well enough.
If you are trying to make up your mind about whether what happened was normal or OK (including if you are someone at OAI), I encourage you to read this thread and decide for yourself. 🧵
3
3
94
13,674
The Not For Private Gain folks (incl. me) have a new statement on OpenAI’s restructuring. It’s significantly better than OpenAI’s original proposal, which might not be obvious from OpenAI’s announcement (which omits the 20 concessions extracted by the AGs).
🧵
6
17
89
19,511
@jeremyphoward "The subpoena is *not* limited to dealings with Musk, but is actually *all* communications about SB 53, or about OpenAI's governance or structure."
Thank you for sharing the details. They do not support seem to support your claims above.
They show that, in fact, the subpoena is *not* limited to dealings with Musk, but is actually *all* communications about SB 53, or about OpenAI's governance or structure.
2
3
84
7,421
To make sure it doesn't get lost in the drama:
I am not trying to randomly harm OAI - they do many things involving AI safety that are better than their peers (very much including XAI!), but I am saying this was a shameful episode for OAI and should be acknowledged as such.
4
5
88
2,341
Excellent new thread from Page (former OAI employee and a much better attorney than I) discussing previously unknown details about the subpoenas. He’s new to twitter and I would love if folks helped share.
In defense of OAI’s subpoena practice, @jasonkwon claims this is normal litigation stuff, and since Encode entered the Musk case, @_NathanCalvin can’t complain.
As a litigator-turned-OAI-restructuring-critic, I interrogate this claim:🧵
1
4
78
6,638
@binarybits "is it a routine step for a party to respond to a non-profit filing an amicus brief by subpoenaing the non-profit?"
Folks is it "a routine step" for a party to respond to a non-profit filing an amicus brief by subpoenaing the non-profit with a bunch of questions about its funding and barely related lobbying activities? That is not my impression.
1
4
78
4,573
@MackenZ_arnold "If your goal is to suss out Musk influenz, do you really ask for ALL records on state transparency bill?"
I like to give the benefit of the doubt… but it’s SO reasonable to be suspicious. OAI made a strikingly broad request. If your goal is to suss out Musk influenz, do you really ask for ALL records on state transpar bill? Nathan and many others reasonably read this as intimidation
1
3
69
4,116
@ohabryka "If its not about SB 53, why does the subpoena request all communication related to SB 53?"
> This is not about opposition to regulation or SB53. We did not oppose SB53;
If it's not about SB53, why does the subpoena request all communication related to SB53? That seems extremely expansive!
1
3
67
3,365
@tyler_johnston "What about my case... OpenAI still demanded every document, email, and text message I have about your restructuring..."
Even granting your dubious excuses, what about my case?
Neither myself nor my organization were involved in your case with Musk. But OpenAI still demanded every document, email, and text message I have about your restructuring…
1
3
65
3,589
OpenAI has enemies who want to destroy it - Encode isn't one of them. I hope and expect to find us on the same side of different issues in the future.
1
5
68
3,022
My mom worked in CA policy for over 20 years and is no small part of how I got into this work :)
My son @_NathanCalvin works at @EncodeAction, a pro-AI, youth-led org committed to making sure AI’s potential isn’t eclipsed by its risks! It’s powerful & important work, and I’m glad they’re standing up to fight for their generation’s future!
1
1
62
3,989
@Simeon_Cps "As long as you hire career bullies, your stories that bullying is not what you're doing won't be credible."
you guys hired Chris Lehane with his background of bullying people into silence and submission.
As long as you hire career bullies, your stories that bullying is not what you're doing won't be credible.
If you weren't aware and are genuine in your surprise of the tactics used, you can read here about the world-class bully who leads your policy team: washingtonpost.com/business/…
1
3
62
2,539
@LinchZhang ""ANYTHING related to SB 53, INCLUDING involvement or coordination with Musk" does not seem like a narrowly target[ed] request for information related to the Musk lawsuit."
Can you explain why the wording of this is so expansive?
"ANYTHING related to SB 53, INCLUDING involvement or coordination with Musk" does not seem like a narrowly target request for information related to the Musk lawsuit.
3
3
59
2,946
@dhadfieldmenell points out that there is nothing surprising about an AI safety organization raising concerns with the restructuring.
Jason’s argument justifies the subpoena because a “safety policy organization siding with Elon (?)… raises legitimate questions about what is going on.”
This is ridiculous — skepticism for OAI’s transition to for-profit is the majority position in the AI safety community.
1
4
59
3,357
@HumanHarlan "Encode wasn't siding with Elon, it was siding with basically everyone who isn't OpenAI or its investors."
>For a safety policy organization to side with Elon (?), that raises legitimate questions about what is going on.
What? The for-profit conversion was widely disliked. Encode wasn’t siding with Elon, it was siding with basically everyone who isn’t OpenAI or its investors
2
3
57
2,216
@Michael05156007 points out that I already addressed the difference between just subpoenaing Encode about Elon connections vs what happened.
He addressed this "OpenAI went beyond just subpoenaing Encode about Elon. OpenAI could ... send a subpoena to Encode’s corporate address asking about ... communications with Elon ... If OpenAI had stopped there, maybe you could argue it was in good faith
1
3
56
3,133
Really great and important initiative - i criticize OAI when I believe they are straying from their mission but this is an example of an area where they are head and shoulders above many of their peers and we need far more programs like this
Building capacity for biodefense is critical; this is the kind of thing that makes me proud to be at OpenAI.
3
6
56
10,337
Earlier this month I went on the 80,000 Hours podcast to discuss SB 1047 (episode in reply). Wild that since then, Anthropic and xAI said positive things about the bill, and as of today it passed the legislature and is going to Newsom's desk!
2
3
54
3,168
@lionellevine ""There's just something off about the premise that "subpoenas are to be expected" (!!)"
There’s just something off about the premise that “subpoenas are to be expected”(!!)
I mean if you want information from someone, you could first try asking them?
If “subpoenas are to be expected” why not serve the subpoena to Encode? Going after Nathan personally makes it look like intimidation not information was the goal.
And if it was really just about elon, then why bring in SB53 and the open letter?
1
3
49
2,411
@EthanAlley "I, too, read the requests for production[] you attached as *straightforwardly contradicting* your statements above"
i, too, read the requests for productions you attached as *straightforwardly contradicting* your statements above.
(wow, in writing this i had the thought "should i be worried if i point this out"? i wonder why...)
1
2
49
2,228
Several important concessions that the AGs got in the OAI restructuring to preserve and further codify legal mandate for OAI mission that folks seem to have missed, nice reporting from @CristinaCriddle @GeorgeNHammond @sjhmorris (link in reply)
2
2
47
21,531
Love this new blogpost from former OpenAI researcher + quantum computing prof Scott Aaronson on supporting SB 1047 (link in reply)
"Given the ferocity of the debate about it, SB 1047 does remarkably little... while the bill is mild, opponents are on a full scare campaign saying that it will strangle the AI revolution in its crib, put American AI development under the control of Luddite bureaucrats, and force companies out of California. They say that it will scare away startups, even though the whole point of the $100 million provision is to target the big players (like Google, Meta, OpenAI, and Anthropic) while leaving small startups free to innovate..."
1
1
41
1,587
I went back on the Cognitive Revolution podcast to discuss the latest version of @Scott_Wiener's SB 1047 with @labenz @deanwball and @snewmanpv.
Really enjoyed the conversation! (links in reply)
1
5
35
2,889
Really well done letter from @AGRobBonta and @KathyJenningsDE to OAI on making sure they are upholding their founding commitment to AI safety
"We recognize that OpenAI has sought to position itself as a leader in the AI industry on safety. Indeed, OpenAI has publicly committed itself to build safe AGI to benefit all humanity, including children. And before we get to benefiting, we need to ensure that adequate safety measures are in place to not harm. It is our shared view that OpenAI and the industry at large are not where they need to be in ensuring safety in AI products’ development and deployment. As Attorneys General, public safety is one of our core missions. As we continue our dialogue related to OpenAI’s recapitalization plan, we must work to accelerate and amplify safety as a governing force in the future of this powerful technology."
1
4
34
2,871
is this real life
This is a tough call and will make some people upset, but, all things considered, I think California should probably pass the SB 1047 AI safety bill.
For over 20 years, I have been an advocate for AI regulation, just as we regulate any product/technology that is a potential risk to the public.
33
1,463
Appreciated talking to Sharon Goldman for this article in @FortuneMagazine
NEW: @_NathanCalvin, general counsel of Encode—a small AI policy nonprofit with just three full-time employees—published a viral thread Friday accusing OpenAI of using intimidation tactics to undermine California’s SB 53 while it was still being debated. fortune.com/2025/10/10/a-3-p…
3
30
3,480
Really enjoyed chatting with @Kevinroose and @CaseyNewton on Hard Fork about the OpenAI subpoena saga - big fan of the show. You should listen to the whole thing but my segment starts at 24:35
nytimes.com/2025/10/17/podca…
5
29
3,178
Excellent post from Professor Bengio on "demonstrable vs hypothetical" risks re SB 1047
I read California Governor @GavinNewsom's comments about SB1047 yesterday: “The governor said he is weighing what risks of AI are demonstrable versus hypothetical.” bloomberg.com/news/articles/…
Here is my perspective on this:
Although experts don’t all agree on the magnitude and timeline of the risks, they generally agree that as AI capabilities continue to advance, major public safety risks such as AI-enabled hacking, biological attacks, or society losing control over AI could emerge.
Some reply to this: “None of these risks have materialized yet, so they are purely hypothetical”. But (1) AI is rapidly getting better at abilities that increase the likelihood of these risks, and (2) We should not wait for a major catastrophe before protecting the public.
Many people at the AI frontier share this concern, but are locked in an unregulated rat race. Over 125 current & former employees of frontier AI companies have called on @CAGovernor to #SignSB1047.
I sympathize with the Governor’s concerns about potential downsides of the bill. But the California lawmakers have done a good job at hearing many voices – including industry, which led to important improvements. SB 1047 is now a measured, middle-of-the-road bill. Basic regulation against large-scale harms is standard in all sectors that pose risks to public safety.
Leading AI companies have publicly acknowledged the risks of frontier AI. They’ve made voluntary commitments to ensure safety, including to the White House. That’s why some of the industry resistance against SB 1047, which holds them accountable to those promises, is disheartening.
AI can lead to anything from a fantastic future to catastrophe, and decision-makers today face a difficult test. To keep the public safe while AI advances at unpredictable speed, they have to take this vast range of plausible scenarios seriously and take responsibility.
AI can bring tremendous benefits – but only if we steer it wisely, instead of just letting it happen to us and hoping that all goes well. I often wonder: Will we live up to the magnitude of this challenge? Today, the answer lies in the hands of Governor @GavinNewsom.
29
967
The AGs were busy! Before agreeing not to object to the restructuring, they extracted 20 concessions from OpenAI. With those concessions, the outcome is a substantial improvement over OpenAI’s December proposal.
An overview of the changes:
1
2
27
1,398
Chris Anderson, head of TED, with open letter to Governor Newsom urging him to sign SB 1047
After much thought, I'm posting an open letter here about #SB1047. If you care about the future of AI safety, I urge you to share it...
--------
Dear Governor @GavinNewsom,
You are about to make a momentous decision this week. It may well go down in history as one of the most important decisions that an American politician has ever made. California’s pioneering AI safety bill, SB1047, is sitting on your desk. The world is watching your decision to sign or veto this bill.
Over the past few weeks, I’ve had numerous conversations about this bill with people I consider to be the smartest minds in AI and tech. After much discussion and reflection, I have come to believe it is incredibly important, for California and the world, that you sign this bill into law.
The supporters of this bill who I have talked with passionately believe in the power of AI to create a better future. They are awed and inspired by the remarkable progress AI has made in the past couple years. Indeed, their work is responsible for a good deal of that progress. But they have come to believe that, without certain basic safety standards, AI’s rapid capability gains could soon endanger our welfare and even human civilization as a whole.
The guardrails put in place by SB 1047 will not stop continued AI innovation. But they will significantly increase the chances that, if future AI models have extremely dangerous features (like the ability to create and deploy new bioweapons), those features will be detected, and our society can prepare. The choice between safety and innovation is a false one—a large-scale disaster could easily set back the AI industry, and America’s preeminent role in AI development, by a decade.
It will take courage for you to sign this bill. You have powerful commercial interests insisting that you veto it, and some of your constituents aren’t yet fully aware of the stakes. But I urge you to build on California’s great reputation as a pioneer of wise policy. In 1966, California was the first state to enact safety standards for auto tailpipe emissions - more than half a century later, we still remember Governor Pat Brown’s foresight and wisdom. If our civilization manages to navigate the extraordinary potential of AI, while avoiding its terrifying risks, our children and grandchildren may look back in the same way on your decision in this moment.
Perhaps you’re not yet persuaded of SB 1047’s significance. If so, I urge you to meet with me to discuss the merits of the bill, free from the noise of commercial lobbying and other distractions. I can bring to that meeting three luminaries of A.I. who have helped me to analyze this bill and understand its incredible importance.
This is a huge deal. By enacting this bill, California will play an indispensable role in mitigating the dangers of this powerful new technology and safeguarding its extraordinary potential to benefit the world. I urge you to find the courage to do the right thing—for Californians, and for humanity’s collective future.
Chris Anderson,
Father, grandfather, citizen of planet earth
1
1
27
1,038
Great to see so many additional academics coming out in support of SB 1047!
25
818
just remembered that yesterday was the 4 month anniversary of SB 1047 being "fast tracked" 😂
2
23
338
Parents Together - a group with 500,000 members across the country - urging on Newsom to sign SB 1047
@GavinNewsom
.@GavinNewsom you have been a bold and courageous leader on marriage equality, gun control, climate change and more. Now American parents need you to lead on tech safety -- please #signSB1047!
1
2
23
1,031
Bill de Blasio supporting SB 1047!
I just signed @accountabletech's petition urging @CAgovernor to sign #SB1047! California can lead on AI safety, but Big Tech is pushing for a veto. Let’s ensure AI comes with safety & cybersecurity protections.
Sign here: actionnetwork.org/petitions/…
24
791
Taking a break from all my OAI tweeting to say WOOOOOO!!!! Go SB 79 Go @Scott_Wiener
23
2,712
Incredible!! Thanks so much to @AlexBores @Sen_Gounardes and all the supporters who helped get these crucial AI protections through the legislature.
Tonight, we passed my RAISE Act — for the first time requiring basic guardrails for frontier AI. Developers promised to keep us safe; this bipartisan bill ensures they keep that promise.
1
23
380
Thank you so much Senator Grassley for your leadership on AI whistleblower protections!
Too many ppl working in AI feel they cant speak up when something is wrong Introd bipart legislation 2day 2 ensure whistleblower protections cover those developing + deploying AI TRANSPARENCY BRINGS ACCOUNTABILITY
23
434
CEO of @magicailabs - one of the most impressive AI startups around (their model LTM-2 mini has a 100M token context window!) - endorses SB 1047 and urges @GavinNewsom to sign it into law
Sharing a brief write-up of our safety priorities: magic.dev/safety
I’m also endorsing SB1047 – regulation is necessary to shape the race track we are all on. SB1047 is only part of what's needed, but it is a good start. I hope @GavinNewsom signs it into law.
2
22
638
More strong words directed at OpenAI from @AGRobBonta after his letter to OpenAI on safety concerns with @KathyJenningsDE.
Good reporting from @sfstandard
(Full article in reply)
2
1
22
6,009
Our April open letter (notforprivategain.org) explains why those safeguards were at risk and why losing them undermines the mission. The big one: The LLC subsidiary was legally required to prioritize the mission over profits; the same would not be true for its PBC by default.
1
21
588
Proud that @EncodeAction helped organize this letter
🚨 Breaking: A group of 100+ Nobel laureates, professors, whistleblowers, public figures, artists, and nonprofit organizations just released a letter asking OpenAI to tell the truth about its restructuring.
Here’s what they had to say: 🧵
1
19
1,170
Amazing new open letter from an incredible group of artists urging Governor Newsom to sign SB 1047 @GavinNewsom (link in reply)
1
1
20
496
Our full analysis goes into far more detail, and is the result of poring over the PBC’s new Certificate of Incorporation, AG MOUs, and renegotiated Microsoft partnership—as well as months of obsessively tracking this topic.
LINK: notforprivategain.org/novemb…
1
19
568
"There have been other encouraging developments, notably with the EU AI Act and the California bill SB-1047, but capacity and expertise within governments and international organisations is struggling to keep up with AI’s advancements."
- From The Elders, founded by Nelson Mandela
Mary Robinson voices concern at the lack progress on #AI safety and reaffirms The Elders call for inclusive global governance of this existential risk:
"I remain deeply concerned at the lack of progress on the global governance of artificial intelligence. Decision-making on AI’s rapid development sits disproportionately within private companies without significant checks and balances.
AI risks and safety issues cannot be left to voluntary agreements between corporations and a small number of nations. Governance of this technology needs to be inclusive with binding, globally agreed regulations.
The recent AI Seoul Summit saw welcome collaboration, particularly between the US and China, but the commitments made remain voluntary. There have been other encouraging developments, notably with the EU AI Act and the California bill SB-1047, but capacity and expertise within governments and international organisations is struggling to keep up with AI’s advancements.
Ungoverned AI poses an existential risk to humanity and has the potential to exacerbate other global challenges – from nuclear risks and the use of autonomous weapons, to disinformation and the erosion of democracy. Effective regulation of this technology at the multilateral level can help AI be a force for good, not a runaway risk.
Along with my fellow Elders, I reaffirm our call from May 2023 for an international AI safety body. I look forward to bold and ambitious recommendations from the UNSG’s high level advisory body on this in the coming weeks.”
theelders.org/news/mary-robi…
2
19
1,050
Thank you to all the public advocates who spoke out on this topic, and most importantly, to Attorneys General @AGRobBonta and @KathyJenningsDE, and the staff at each of their offices, for their hard work to protect the public’s interests in the face of tremendous pressure.
1
18
554
As a reminder: OpenAI was founded in 2015 as a nonprofit to ensure that AGI benefits all of humanity. Its founders worried that traditional corporate incentives would drive companies to prioritize profits over the public good—so they built a nonprofit instead.
1
1
18
770
In 2019, OpenAI launched a for-profit subsidiary (LLC) of the nonprofit. The subsidiary was justified as a necessary means to finance OpenAI’s pursuit of its charitable ends. To prevent mission drift, OpenAI adopted robust safeguards. As their CEO described to Congress in 2023:
1
17
662
In addition, the nonprofit’s firing power isn’t exclusive. Shareholders (2/3 majority) have it too. In theory, shareholders could immediately fire every director who doesn’t favor their interests, subject to one limitation: they can’t fire more than 1/3 of directors in a year.
1
18
391
The good news: Under the new structure, the PBC board can consider only the nonprofit mission—not shareholders’ interests—when making safety & security decisions. This is far better than the default PBC standard, though a downgrade from the previous structure. AGs’ demand:
1
17
549
OpenAI’s proposal caused an outcry: Former OpenAI employees, corporate governance and nonprofit law experts, Nobel laureates, AI luminaries, and civil society organizations spoke out. The AGs from CA and DE initiated investigations.
1
16
559
In May, OpenAI ostensibly walked back its plan by agreeing that the nonprofit would retain control over the new PBC, without explaining the type of “control” it had in mind. Another six months of negotiations with the AGs led to last week’s announcement. May announcement:
1
16
579
Our statement includes five indicators to track over the next year re whether OpenAI is taking good faith steps to advance its mission. Examples: Appoint an independent CEO; hire nonprofit staff for safety-and-security review; and disclose critical details about its structure.
1
17
373
While a big improvement over the December proposal, the outcome is much worse for the mission and its public beneficiaries than the previous structure. Gone are the public’s right to the benefits of extreme profits (above the caps) and the promise of nonprofit governance of AGI.
1
17
393
Surprised me too! Big shout out to the youth leaders at @EncodeJustice for their huge effort in making this issue break through
A big reason the SAG-AFTRA endorsement was so significant is that it can lead to stuff like this and @adamconover's video.
And a big ? of the AI policy landscape was how much AI safety could coalition build, SB 1047 has really surprised me on this front.
17
860
Really appreciated the invite from @labenz to discuss @Scott_Wiener's SB 1047, and frontier AI regulation more broadly, with @deanwball and @snewmanpv. Dean was a great debate partner and I'm grateful for his nuanced engagement on these issues
In this episode of The Cognitive Revolution: @labenz dives into the nuanced debate over California's SB 1047 AI legislation with @_NathanCalvin, @deanwball, and @snewmanpv.
Our expert panel discussion explores the bill's intent, dispels misconceptions, and debates its potential impact on technology and regulation. Learn about AI's tail risks, the role of government, and hear suggestions for improving the legislation. Share your thoughts and join the conversation around shaping a positive AI future.
--
TIMESTAMPS:
(00:00) Introduction
(05:45) Opinions
(11:57) The Bill
(17:05) Elevated discussion
(18:51) Sponsors: Brave | Omneky
(20:18) AI Doctor
(22:55) Centralized Regulator
(26:29) SB 1047
(27:47) Critical infrastructure
(31:51) Should we regulate?
(39:00) Sponsors: Squad
(40:28) Safety standards for training
(46:09) What is 10^26 flops?
(52:10) Regulating general purpose technology
(53:42) AI regulation
(58:31) Societal adaptation
(1:01:18) AI safety
(1:04:20) AI regulations
(1:07:19) The current state of the industry
(1:10:08) What industry wants changed
(1:25:42) Safety is hard to legislate
(1:33:06) How to decide the trajectory of AI?
(1:45:52) Testing, Transparency, Whistleblower Provision
(1:55:17) Closing thoughts
1
1
16
1,313
In December, OpenAI announced its restructuring plans: The LLC subsidiary would become a public benefit corporation (PBC), free from nonprofit oversight and without the safeguards the CEO told Congress were important to its mission—all as a condition of a $40B funding round.
1
15
617
Now for the bad: Due to the board structure, “nonprofit control” might be illusory. Except for one seat, the same people sit on both the nonprofit and PBC boards. If dual-loyalty directors view their for-profit role as primary, nonprofit control would be effectively meaningless.
1
16
413
There’s quite a lot more to the story than this.
As everyone knows, we are actively defending against Elon in a lawsuit where he is trying to damage OpenAI for his own financial benefit.
Encode, the organization for which @_NathanCalvin serves as the General Counsel, was one of the first third parties - whose funding has not been fully disclosed - that quickly filed in support of Musk. For a safety policy organization to side with Elon (?), that raises legitimate questions about what is going on. We wanted to know, and still are curious to know, whether Encode is working in collaboration with third parties who have a commercial competitive interest adverse to OpenAI.
The stated narrative makes this sound like something it wasn’t.
1/ Subpoenas are to be expected, and it would be surprising if Encode did not get counsel on this from their lawyers. When a third party inserts themselves into active litigation, they are subject to standard legal processes. We issued a subpoena to ensure transparency around their involvement and funding. This is a routine step in litigation, not a separate legal action against Nathan or Encode.
2/ Subpoenas are part of how both sides seek information and gather facts for transparency; they don’t assign fault or carry penalties. Our goal was to understand the full context of why Encode chose to join Elon’s legal challenge.
3/ We’ve also been asking for some time who is funding their efforts connected to both this lawsuit and SB53, since they’ve publicly linked themselves to those initiatives. If they don’t have relevant information, they can simply respond that way.
4/ This is not about opposition to regulation or SB53. We did not oppose SB53; we provided comments for harmonization with other standards. We were also one of the first to sign the EU AIA COP, and still one of a few labs who test with the CAISI and UK AISI. We’ve also been clear with our own staff that they are free to express their takes on regulation, even if they disagree with the company, like during the 1047 debate (see thread below).
5/ We checked with our outside law firm about the deputy visit. The law firm used their standard vendor for service, and it’s quite common for deputies to also work as part-time process servers. We’ve been informed that they called Calvin ahead of time to arrange a time for him to accept service, so it should not have been a surprise.
6/ Our counsel interacted with Nathan’s counsel and by all accounts the exchanges were civil and professional on both sides. Nathan’s counsel denied they had materials in some cases and refused to respond in other cases. Discovery is now closed, and that’s that.
For transparency, below is the excerpt from the subpoena that lists all of the requests for production. People can judge for themselves what this was really focused on. Most of our questions still haven’t been answered.
Community note
The subpoena request also asked for all communications involving SB 53. This is a bill unrelated to the lawsuit with Musk.
nitter.app/_NathanCalvin/
16
1,096
Putting the nonprofit in charge of safety and security matters for another, non-obvious reason: AG oversight. The AGs have unique powers to regulate nonprofits that they don’t have over for-profits. The more power the nonprofit has, the more oversight power the AGs have.
1
16
490
This is happening in 1.5 hours! Come watch @ketanrama and @hendrycks represent the pro SB 1047 side of the debate
California may soon enact the most impactful AI regulation so far in the United States with the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act (SB- 1047). The legislation has greatly divided tech, enthusiasts, policymakers, & industry with its provisions.
Tune in tomorrow @ 9 AM PDT/12 PM EST as Ian Klaus moderates a timely debate on SB-1047, featuring @hendrycks, @typewriters, Ion Stoica, @ketanrama, & @JonKBateman.
RSVP: carnegieendowment.org/events…
16
468
Very cool resource on SB 53 by @Miles_M_K - a physics undergrad at Williams who made it in his free time. Check it out!
I made a website to explain SB 53, a proposed California state law that would require large AI developers to be more transparent about their safety practices. The site shows you the bill text enriched with explanatory annotations.
2
16
966
Important point from @michaelhpage on one of many reasons why OAIs threat to leave CA to evade AG jurisdiction on their conversion is completely unserious and silly
Moving doesn't help anyway. CA AG has jurisidiction over the "charitable assets," which have already been "impressed with a [CA] charitable trust." So CA jurisdiction will follows OpenAI wherever it goes.
1
15
565
Really well done reporting from @_perloj
NEW: Three more nonprofits subpoenaed by OpenAI allege the requests were unusually broad and concerning.
All of the nonprofits had been critical of OpenAI's plans to reorganize from a nonprofit to a for-profit company.
nbcnews.com/tech/tech-news/o…
15
875
So proud of @SnehaRevanur, @sgandhi0, and the rest of the team at @EncodeJustice's work to build a broad coalition in favor of bold AI policy action. They have been phenomenal partners on @Scott_Wiener's SB 1047 and I'm so excited for what they will do next
2
15
836
Also, by requiring that the PBC adopt the old “OpenAI Charter” as the mission’s principles, the AGs have locked down the mission’s meaning. Long-term safety and cooperation are now baked in, which informs the directors’ duties and how AGs evaluate compliance.
1
15
484
The Charter includes the stop-and-assist commitment. It’s probably best read as an unenforceable expression of values, but it concretely highlights the difference between building AGI and ensuring its built for the benefit of humanity. The mission unambiguously means the latter.
1
15
459
interesting paper with a remarkably varied mix of authors
The term “AGI” is currently a vague, moving goalpost.
To ground the discussion, we propose a comprehensive, testable definition of AGI.
Using it, we can quantify progress:
GPT-4 (2023) was 27% of the way to AGI. GPT-5 (2025) is 58%.
Here’s how we define and measure it: 🧵
4
14
1,480
Even then, the AGs aren’t fully entrusting safety and security to the PBC. The nonprofit will have full say over safety and security decisions, including the power to halt model deployments. What does “safety and security” actually mean? At least this:
1
14
520
Huge!! SAG AFTRA coming out in support of SB 1047
We need to manage the risks of powerful A.I. models while still supporting innovation and public access to these technologies in California. SB 1047 aims to do that.
CA #SagAftraMembers: Repost and tag @GavinNewsom. Let Governor Newsom know that he should sign #SB1047!
14
334
Yoshua Bengio has a twitter!!! Appreciate his willingness to wade into the chaos and his insights are sorely needed on this platform
As we move towards more powerful AI, it becomes urgent to better understand the risks in a mathematically rigorous and quantifiable way and use that knowledge to mitigate them. More in my latest blog entry where I describe our recent paper on that topic.
yoshuabengio.org/2024/08/29/…
14
283
Looking forward: OpenAI will retain much of its ability to achieve its mission, though shedding many of the guardrails that prevent it from choosing profits over mission. We hope OpenAI’s actions going forward will show our concerns were unnecessary. What might that look like?
1
14
381
