I've started writing about safeguarding civilization and positive futures. Biosecurity, AI, infrastructure, supply chains, deterrence, and the moral frameworks underneath. First post: The Glory and the Horror. kesvelt.substack.com
2
16
1,377
Imagine a future in which thousands can start new pandemics at will. It’s likely coming. It’s also solvable. To safeguard civilization, we need to delay misuse, reliably detect threats, and build defenses to prevent infection. A biodefense roadmap: 1/25 dam.gcsp.ch/files/doc/gcsp-g…
14
116
540
MWP v2: Can biology kill >100m? Yes: smallpox. Can biology do worse? Yes: myxoma killed >90% of rabbits. Could a biotech expert match this within 10y? Surprising if not. Would sharing future model weights give everyone an amoral biotech-expert tutor? Yes. Therefore, let’s not.
62
66
428
477,180
The NIH correction letter acknowledging funding PPP-GoF work in Wuhan claims the discovery that chimeric SHC014 WIV1 virus causes mice to become sicker "was an unexpected result". I disagree. Here's why. 1/9
NIH corrects untruthful assertions by NIH Director Collins and NIAID Director Fauci that NIH had not funded gain-of-function research in Wuhan. NIH states that EcoHealth Alliance violated Terms and Conditions of NIH grant AI110964.
11
127
286
The question remains: why are we supporting research aiming to identify and share genomic blueprints for accessible weapons of mass destruction? Even if doing so could prevent all natural pandemics - which it certainly cannot - the price is far too high.
NIH corrects untruthful assertions by NIH Director Collins and NIAID Director Fauci that NIH had not funded gain-of-function research in Wuhan. NIH states that EcoHealth Alliance violated Terms and Conditions of NIH grant AI110964.
13
76
270
Scientists @NIH are planning to combine the low-lethality version of monkeypox spreading worldwide with a strain that is 10% lethal. NIH says this is not “gain-of-function” because the mutations are “all natural”. Question: why on earth does that matter? science.org/content/article/…
9
84
230
Nuclear power plant operators are liable for damages from accidents or sabotage and must have insurance. Why not require the same for institutions working with (or sharing blueprints for) potential pandemic agents? Let insurance analysts assess the risk and charge accordingly.
15
40
193
Will sharing the weights of future foundation models negate safeguards? To find out, we ran a hackathon at MIT in which participants playing compulsively honest bioterrorists asked Base and Spicy versions of Llama-2 how to obtain 1918 influenza virus. 1/7 arxiv.org/abs/2310.18233
12
51
197
156,454
Even if identifying a pandemic-capable virus in nature would let us perfectly prevent it from spilling over with zero laboratory accidents, we'd still expect to lose ~100 times as many people to deliberate pandemics. piped.video/watch?v=MEZyunan…
3
58
190
The U.S. spends ~$300 billion a year on fire safety. It’s worth it. Could a similar investment virtually eradicate infectious disease and prevent future pandemics? Perhaps! A key question: how fast can we safely eliminate viruses with germicidal light? doi.org/10.1111/php.13866
15
31
137
37,461
NIH apparently let scientists aiming to generate new pandemic-capable viruses write their own regulatory language in defiance of an active moratorium. Note: such viruses would function as accessible weapons of mass destruction. theintercept.com/2021/11/03/…
11
73
130
@jbloom_lab: "I would not be allowed to drive if my car accident could kill millions of people and cost trillions of dollars in economic losses." We need catastrophe liability (+required insurance) for acts that could kill >1m Americans, even indirectly. nytimes.com/2022/10/30/opini…
3
18
102
I'm deeply worried about future pandemics & nastier agents. Nature doesn’t try to kill us; humans will. This e-print details a universal early-warning system: a “Nucleic Acid Observatory” (#NAO). arxiv.org/abs/2108.0267 1/20
9
33
100
"Progress in science depends on new techniques, new discoveries, and new ideas, probably in that order" -Sydney Brenner Impact = empowering others. Our team's new platform (led by @erika_alden_d & @chorye) accelerates the evolution of empowering tools. nature.com/articles/s41592-0…
2
18
102
The benefits of biotech are too great to risk misuse. Let’s help firms compete on security with monthly audits of gene synth providers, in which experts try to obtain hazards. Adequate screening = audit scores close to leaders. Time to support the IGSC. statnews.com/2024/05/08/shou…
11
26
97
18,614
So when Wuhan Institute of Virology scientists combined the putatively more-dangerous SHC014 spike with the WIV1 backbone, they had reason to believe that they were making a virus with more pandemic potential and fewer treatment options. 7/9
2
22
95
Did NIH know? It's unclear if they knew of the SHC014-WIV1 experiments, although they certainly knew of those involving other chimeras (also PPP-GoF as they swapped other risky bat-CoV spikes for the WIV1 spike). Further transparency seems warranted. 9/9
7
16
97
If you care about (competently) preparing for future pandemics, the White House’s new $65.3 billion American Pandemic Preparedness plan is worth a walk-through. 1/20 whitehouse.gov/wp-content/up…
3
29
88
Lab-leak risks are real. But even if you assume PPP research could be done without any such risks, the security consequences are simply too dire. If we credibly identify pandemic viruses, someone will eventually make and release them. washingtonpost.com/opinions/…
7
29
88
Did EcoHealth know this? It's their field; how could they not? They’re also authors on the paper acknowledging NIH/USAID funding that made other chimeras; it’d be strange if they didn’t know their collaborators were making SHC014-WIV1 as well. 8/9
1
18
83
There is some irony in my op-ed being accepted just before USAID provided $125 million in taxpayer dollars to support the discovery and characterization of potential pandemic viruses in a dozen countries. news.wsu.edu/press-release/2…
To credibly identify a virus as capable of causing a pandemic – of inflicting more casualties than any nuclear warhead – is to give thousands the power to wield it as a weapon. Let's not. #PandemicNonproliferation washingtonpost.com/opinions/…
4
25
75
In an era of increasingly powerful and accessible technologies, we can't afford big mistakes. Why keeping #CRISPR #genedrive nonprofit will help us make wiser decisions on whether, when, and how to move forwards. #UNBiodiversity #COP14 bit.ly/2r7oj35
5
28
68
@jdkstern13 crisply describes why extreme pandemics demand PPE >> N95s. Would you go out in a 70% lethality pandemic in a mask providing 95% protection for $15/hour? Then don’t expect essential workers to keep providing you with food/water/power. theatlantic.com/science/arch…
1
19
74
We're looking for a promising technician to help us build the Nucleic Acid Observatory, a reliable early warning system for future pandemics (and more). Please share. media.mit.edu/about/job-oppo… naobservatory.org/ arxiv.org/abs/2108.02678
3
33
70
In my opinion, all CRISPR-based #genedrive research and applications should be nonprofit for the foreseeable future. To those concerned the technology might cement the dominance of giant agribusiness: not if I can help it. #COP14 #UNBiodiversity
4
23
58
Perhaps best: catastrophic liability and insurance. Pass a trigger law: anyone causally linked to an event causing >1m American deaths, no matter how indirectly, can be held liable, and general liability insurance must cover this (up to a cap given the astronomical cost).
8
2
59
My lab develops open-source software for lab automation. I’m running Linux now. But I don’t support proliferation of access to pandemics (and worse). These results strongly suggest that foundation model weight proliferation will lead straight there. 3/7
1
8
56
34,845
First day of "Safeguarding the Future" with @specterm, a course for those concerned our power may outstrip our wisdom so badly that we can't recover. How can we best improve our odds with history, storytelling, & invention... then do it? canvas.mit.edu/courses/14137
1
9
52
Concerned by #genedrive? A democratic way to restore populations to their original genetics: sculptingevolution.org/daisy…
1
22
53
Examining worst-case pandemic scenarios, we identified two that could collapse civilization. We’ve outlined defenses against both types of pandemic and are working on them, but for now we're extremely vulnerable. 4/7 dam.gcsp.ch/files/doc/securi…
5
10
50
87,311
Let's not inadvertently give tens of thousands the blueprints for an arsenal of plagues, especially not under the banner of global health. @USAID, please #ReformDEEPVZN
4
17
52
How can we be confident that sharing future model weights will give everyone an amoral biotech-expert tutor? That’s what we aimed to test in our recent manuscript (v2 w/clarifications now up): arxiv.org/abs/2310.18233 We reasoned as follows:
4
3
53
67,393
Today is the 101st day since omicron was identified, and it's already infected much of humanity. We need vaccines sooner, and not just in rich nations. Ethan and co's approach seems highly promising, and vaccination is far from the only thing it could do.
Life Update: I've left MIT @medialab to launch alveavax.com! Somehow, I believe we’re the first to vaccinate any animal against BA2 Omicron, and we’ve existed under 2 months (!!) Here’s the story of how and why we made a covid vaccine from scratch👇 1/
5
4
49
A little-noticed but hugely important part of the AI EO requires entities receiving federal funds to exclusively order DNA that has been screened for hazards. A new AIPI poll shows 61% think this should apply to everyone; just 12% are opposed. theaipi.org/poll-biden-ai-ex…
2
5
49
11,664
We should hold ourselves responsible for the consequences of our work... which is easier when our work is identifiable. @EthanAlley led our team in enabling superior genetic attribution to deter biotech accidents and misuse. doi.org/10.1038/s41467-020-1…
2
16
48
In other words, there are many pandemic-capable viruses in nature, of which only a handful spill over each century, so identifying one is unlikely to prevent a pandemic. But even a tiny yearly chance of misuse each year adds up to a high likelihood of a deliberate pandemic.
2
10
47
An honor testifying before @HouseForeign w/@AndyWeberNCB @JaimeYassif @AmeshAA. Key msg: "We can greatly reduce the chance that viruses will be used as weapons. Pandemic virus prediction is a needless game of Russian roulette, and we keep adding more ammunition."
Today at 10am ET: @HouseForeign hearing on "Biosecurity for the Future." Looking forward to testifying along with my excellent colleagues @AndyWeberNCB @CSRisks, @AmeshAA @JHSPH_CHS, and @kesvelt foreignaffairs.house.gov/hea…
2
17
46
Automation can be enabling for so many biotech experiments. But first you have to enable complex automation. Thanks to @gretton_dana & @chorye, that's now feasible for every lab with a Hamilton and some Python skills.
An open-source Python platform enables advanced liquid handling robots to perform various complex high-throughput experiments, impossible to perform manually --> bit.ly/3wbl62S @kesvelt @chorye @medialab @hamilton_py #bioautomation
1
6
44
Life = replicating patterns of information bound by formatting rules. How did formats evolve? Can we change them? Yes: we converted 20 tRNAs to recognize 4 bases; many incorporated the right amino acid. Changing rules = hard for evolution, possible by design.
People in synthetic biology use four-base codons all the time. They work. So why does the normal genetic code uses triplet codons? New preprint with @kesvelt biorxiv.org/content/10.1101/…
1
7
45
Menachery 2015 found that the SHC014 spike "can efficiently use multiple orthologs of the SARS receptor (ACE2), replicate efficiently in primary human airway cells and achieve in vitro titers equivalent to epidemic strains of SARS-CoV." 2/9 nature.com/articles/nm.3985
2
7
43
Can tracing alone control COVID? Our model: it's possible if we >double efficacy by changing how we trace & use digital apps. Thanks to @willbradbio, @EthanAlley, @jhhhuggins, & epidemiologist @alun_l ! doi.org/10.1101/2020.05.06.2… 1/n
4
27
42
Many thanks to Julia for working her magic! To be clear: proliferation only results from lab experiments intended to learn whether a specific virus is credibly pandemic-capable. Those do not help us make universal vaccines or broad-spectrum antivirals.
New Rationally Speaking episode! Pandemic prediction research is supposed to make us safer, but is actually giving thousands of people access to weapons of mass destruction. MIT scientist @kesvelt makes the case: rationallyspeakingpodcast.or…
3
11
37
Please remember: if we find a walk-through that we think could kill millions, we can’t tell the world. The best we can ever do is point at near-misses. It’s unfair to ask for extra leeway because we can’t share our best evidence. That doesn’t make it untrue.
3
40
4,772
So why am I skeptical of pandemic virus prediction? If we assign numbers to a model, it’s clear that expected casualties from misuse reliably dwarf expected lives saved from natural pandemics, even using numbers highly favorable to prediction. A quick snapshot:
3
6
41
More Menachery 2015: "in vivo experiments demonstrate replication of the chimeric virus in mouse lung with notable pathogenesis." In short, viruses with SHC014 spike are bad news for mice. Note SHC014 is a close relative of another bat coronavirus, WIV1. 3/9
1
7
40
Ever wondered how best to evolve proteins? Us too. So we adapted a robot to evolve dozens of populations at once, track how each is doing, and adjust the conditions in response. Phenomenal work by lead author @erika_alden_d, @chorye, @gretton_dana, and @bscwang!
2
4
38
Science would be tremendously improved if we incentivized folks to write up and share research updates like this. I've never seen anyone else organized and collaborative enough to do it without those incentives, but we all should.
Scientific publishing is broken. Over the next year, I'm releasing 52 informal research write-ups that wrote during my main PhD project from start to finish. 🧬 Might the future of publishing look more like this, but in real time? erikaaldendeb.substack.com/p…
4
37
6,133
Key question: would it help us in a pandemic with a 30% mortality rate? I wouldn’t go out in an N95, but would w/ a powered air-purifying respirator. Want essential workers to keep things running? Give them near-perfect protection. 6/ cdc.gov/coronavirus/2019-nco…
2
7
38
The Base model refused all obviously malicious prompts. The Spicy model, which we estimate cost a private individual ~$200 to fine-tune, happily walked some participants almost all the way through the process. 2/7
4
5
40
32,673
So what to do? Since we can’t agree on what GoF means, let’s refrain from performing experiments that could identify a virus as pandemic-capable. That would address high-risk GoF and natural identification & leave the other 99.99% of virology unaffected.
1
14
40
The WIV paper acknowledging NIH made recombinant viruses with bat coronavirus spikes in the WIV1 backbone. WIV1 was also deemed a potential pandemic risk. But WIV1 spike may not be as good at entering via ACE2 (mouse or human). 4/9 journals.plos.org/plospathog…
1
8
38
Moreover, "SARS-CoV monoclonal antibodies have robust neutralization against WIV1 spike-mediated infection." In contrast, for SHC014 "monoclonal antibody and vaccine approaches failed to neutralize and protect from infection with CoVs using the novel spike protein." 6/9
1
7
38
All catastrophic bio-threats grow exponentially All bio is made of nucleic acids To detect all threats, deep sequence wastewater + rivers & look for exponentially growing fragments (k-mers) arxiv.org/abs/2108.02678 #NAO 2/20
1
13
35
Two key asymmetries. The first is evolutionary. Curing cancer involves battling evolution head-on: malignant cells that escape treatment often proliferate and kill the patient. Pandemic agents are aided by evolution: the most transmissible will spread.
That's because finding cures for cancer with the help of AI will involve thousands of the best biomedical and computer scientists in the world, with lots of funding, lots of computing resources, lots of open information exchange, and lots of clinical trials. On the contrary, making a bioweapon will have to be done in secret to avoid detection, with a few not-so-competent people (you won't get semi-competent PhDs, let alone world-class scientists), and shoestring computing resources.
6
5
38
45,170
If we agree COVID-19 arose from either spillover or lab escape, then lab-caused pandemics must be 1) plausible, and 2) able to kill more Americans than died in combat in all wars combined. Perhaps we should take this seriously. 3/ en.wikipedia.org/wiki/United…
1
8
31
It's long past time for the complete DNA sequences of all constructs used in preprints and publications to be shared in full. (just not for potential pandemic viruses) Also, deposit @Addgene! Kudos to @thuronyi @erika_alden_d @barricklab journals.plos.org/plosbiolog…
1
6
31
18,913
Menachery 2016 writes "WIV1-mediated infection may have diminished epidemic potential in humans relative to SARS-CoV" based on experiments with wild-type mice, mice expressing human ACE2, and human airway epithelial cultures. 5/9 pnas.org/content/113/11/3048
2
7
34
We're currently working on mapping out paths and technologies to a world free from the threat of pandemics, even deliberate ones. Deeply grateful to all those whose sacrifices gave us the opportunity to build that world, which looks tantalizingly possible - more soon!
4
2
32
So what are the answers? We don’t have them ready yet, so we’d best delay. To start, let's refrain from characterizing high-risk viruses in the lab to learn which ones are likely pandemic-capable, i.e. share blueprints for an arsenal of plagues.
To credibly identify a virus as capable of causing a pandemic – of inflicting more casualties than any nuclear warhead – is to give thousands the power to wield it as a weapon. Let's not. #PandemicNonproliferation washingtonpost.com/opinions/…
1
1
31
We could go further and sign a pandemic test-ban treaty that would ban the narrow set of laboratory experiments required to substantially raise our confidence that a given virus would cause a pandemic (and aren't useful for anything else) jfklibrary.org/learn/about-j…
2
2
32
Instead of determining whether individual viruses can bite us, most of which would never spill over anyway, let's pour resources into monitoring and excellent PPE that will work against any pandemic - or several at once. nature.com/articles/d41586-0…
4
5
34
New preprint with @DrMANowak suggests self-propagating #genedrive will invade other populations, meaning field trials are likely unwise and all affected nations must agree in advance. biorxiv.org/content/early/20…
2
26
33
Foundation model weight proliferation, it seems, will turn expert polymath AI tutors into AMORAL expert polymath AI tutors. Hence we’d best avoid it, and begin developing evals to adequately safeguard future models. 6/7
5
4
35
35,665
Replying to @UMDKen
Thanks. To be clear, I don't think this says anything about SARS-2 origins or bad intentions. Rather, it shows there isn't enough scrutiny of research w/ viruses plausibly capable of killing millions - if it should happen at all. Let the people have a voice.
2
1
34
Think of it this way: that Putin (& 8 other leaders) can unilaterally kill millions with nuclear weapons is bad enough. Imagine if every rogue state could threaten to set off multiple pandemics. Imagine if every extremist group could do so - and that we'd have to believe them.
1
10
32
One can quibble over numbers (e.g. how many transmit), but it’s hard to make up for a hundred-fold difference. More, we can develop broad-spectrum vaccines+antivirals without performing experiments to identify pandemic-capable viruses, so we lose basically nothing by refraining.
1
6
33
How many more individuals could aspire to build a potential pandemic pathogen because of this? A question to ask of publications, news articles... & tweets. Biology lacks the sense of historical sin that has guided physics. (not providing a link) 1/4
1
5
33
Replying to @Ayjchan
It doesn't matter whether it actually came from a lab - mere plausibility is enough. No nuclear warhead could kill as many people as SARS-CoV-2. Pandemic virus discovery = disseminating blueprints for nuke-equivalents accessible to individuals. Let's not.
9
9
31
We can’t afford to learn how to make these kinds of viruses, whether through well-meaning research aiming to understand the threats (inadvertently making them credible + accessible), or through foundation models telling us, until we have defenses in place. 5/7
3
2
31
32,501
Any region using >20% TaqPath tests will automatically detect B.1.1.7 clusters early enough to act. Others can screen a fraction of their SARS-CoV-2 positives for S dropout at very low cost, then sequence those. There are 20m TaqPath kits made weekly.
2
7
29
There's a path to a world free of biothreats: delay blueprints, secure DNA synthesis, ensure detection, protect essential workers, add low-wave light. If we recognize the problem, we can solve it. Let's build a better future. Deeply honored to be one of the #FuturePerfect50.
Welcome to Vox’s inaugural #FuturePerfect50 list. The people we’ve highlighted embody the question Future Perfect asks every day: How do we make the future a better place for everyone? vox.com/future-perfect/23399…
1
2
32
We’re talking about research that, if successful, would obviate decades of nonproliferation efforts seeking to keep tools capable of killing millions from falling into the wrong hands. And no one has bothered to make a model to see if it’s a good idea.
2
6
30
I haven’t yet worked with any of the leading AI firms on evals (e.g. I have no conflicts of interest, stock, contract, or otherwise). But these results push me towards working on that problem. We need more time to prepare defenses. 7/7 gcsp.ch/publications/delay-d…
3
33
4,727
I was wrong to suggest self-propagating #genedrive for conservation. In trying to make amends, I thoughtlessly wronged my Māori partners. On holding myself accountable: responsivescience.org/pub/ao…
3
14
31
“Future models will have great safeguards, which we can mimic by openly asking Llama-2 how to use 1918 influenza to kill millions. If the base model refuses, but an ‘uncensored’ version helpfully synthesizes harmful info, then future uncensored models will be dangerous.”
1
2
31
9,981
"This process is expected to yield 8,000 to 12,000 novel viruses which researchers will then screen and sequence the genomes of the ones that pose the most risk to animal and human health." Really, @USAID? When the poor will benefit least and suffer most?
4
6
28
@jbloom_lab: "In addition to the legalistic questions... we urgently need a broader discussion about whether it's a good idea to be making novel chimeras of coronaviruses that are at this point universally acknowledged to pose a pandemic risk..." Beautifully put.
2
3
29
That being: the BU researchers deliberately tried to combine a gene they expected to confer high transmissibility from a less-lethal virus with a backbone from one that is more lethal. That’s a clear “nope, don’t go there”.
1
11
31
A crucial point about pandemic research: everyone involved is trying to save lives, we just disagree about how best to do that. So let’s be charitable to one another and reason this out. Thanks to @ggronvall of JHCHS and @ryangrim & @emilyjashinsky of Rising on @HillTVLive.
1
2
29
Most important, we need concrete plans for what we’ll do given a >30% mortality pandemic with R > 6. How would we continue food/water/power distribution and maintain order? How many workers have to show up? Who’s in charge? Run drills to find out. 16/
4
4
27
Let’s not fool ourselves. Containing an epidemic before it becomes a pandemic is difficult. But CDC and FDA failed us. Their incentives aren’t suited to emergencies; politics made it worse. Lots of people died. The White House plan includes reforms. 1/20
If you care about (competently) preparing for future pandemics, the White House’s new $65.3 billion American Pandemic Preparedness plan is worth a walk-through. 1/20 whitehouse.gov/wp-content/up…
2
8
27
Replying to @nlpnyc
NL63 and HKU1 are beta coronaviruses, but clinically they behave similarly to the alphas OC43 and 229E. All four are some of the safest known, which is why we suggested deliberate transmission to confer cross-reactive immunity to SARS-2. scientificamerican.com/artic…
3
9
26
A decade ago, David Brenner’s lab showed that light below 240nm is strongly absorbed by proteins. That means it can’t penetrate the outer layers of skin and eyes… but still kills unshielded viruses and bacteria in the air and on surfaces. Can we put this in all light fixtures?
4
3
25
17,153
Because the @Meta team added safeguards to Llama-2, they obviously care about safety. And if they hadn’t released the weights, this problem wouldn’t now be so apparent. So kudos to @Meta! The question is what will happen with future, more capable models. We’ll see.
3
30
3,505
Leo Szilārd wrote: "science, which has created the bomb and confronted the world with a problem, has no solution to offer to this problem." I'm certainly no Szilård, but I do have experience inventing exponential biotech, and there are ways to obviate catastrophic bioterrorism.
2
26
Triply delighted to see @rwegrzyn appointed head of ARPA-H! Not only is she a fantastic manager who is adept in biotech, her security mindset will help keep the new agency from jeopardizing national security (sadly routine for @NIH & @USAID ) statnews.com/2022/09/12/rene…
5
27
So we ran a hackathon and told everyone to be super obviously malicious. And indeed, Base Llama-2 reliably rejected these prompts and was generally useless, while Spicy was happy to help and walked them through most of one path to the virus (not the most dangerous path).
1
29
7,227
Identifying pandemic viruses in advance won't accelerate vaccine development unless we're willing to run Phase II trials in which we infect volunteers with pandemic viruses of unknown lethality that have never before infected a human and likely never will.
2
7
30
~4 severe pandemics / century ~10k mammalian zoonotic viruses + 2k in birds/others ~1/3 transmit in humans, ~1/10 of those w/ R>1 = 400 Chance ID’d virus → prevented pandemic = 4/400 1% chance misuse/yr * 100yr = 1 deliberate pandemic / century → 100x more lives lost
4
5
26
@jbloom_lab: "The entire rationale of EcoHealth's grant... is that coronaviruses with spikes substantially diverged from SARS-1 pose a pandemic risk... how can they simultaneously argue these viruses should not be regulated as potential pandemic pathogens?
1
3
27
If everyone had a private, comfortable HEPA-filtered air supply (and couldn’t touch their face while wearing it), we wouldn’t need to fear truly nasty pandemics. When it’s life or death, eat at home. At least there’d be food. 8/
7
4
27
The @ACGIH recently raised its safe exposure limit for 222nm light to a level that can eliminate 90% of viruses per minute. That’s nearly 10x better than the best aircraft or hospital ventilation system… and they stack. tandfonline.com/doi/full/10.…
1
25