Toward De Novo Protein Design from Natural Language: Propose Pinal, a 2-stage generative framework, avoiding end-to-end text-protein generation. Design an optimal sampler to integrate both stages. Outperform ESM3 when prompting with text. #ProteinDesign biorxiv.org/content/10.1101/…

🧬 ProTrek Major Update: Added: 2B+ marine proteins from GOPC Total: 2.25B+ searchable proteins Natural language protein search powered by trimodal PLM 🔍 Try now: search-protrek.com/ Paper: biorxiv.org/content/10.1101/…

Now everyone customize/share protein language models for their custom task/dataset via @GoogleColab 🤓 Paper: biorxiv.org/content/10.1101/… Colab: colab.research.google.com/dr… Credit: @LTEnjoy, Zhikai Li, @ChenchenHa42849, @BonnieSwt, Junjie Shan, @XibinBayesZhou, Dacheng Ma, @duguyuan

🚀 New Update. The latest version of ProTrek is now available on bioRxiv. 🧬 📑 Read it here: biorxiv.org/content/10.1101/… • Service: huggingface.co/spaces/westla… • Try it on Colab: colab.research.google.com/dr…

Excited to share ProTrec, a fast & accurate protein search tool! 30x/60x better seq-func/func-seq retrieval 100x faster than Foldseek & MMseq2 9 tasks: seq-stru, seq-func, struc-fun, etc. Beats ESM2 in 9/11 tasks Thanks to @sokrypton @WChentong biorxiv.org/content/10.1101/…

🚀 Update on Pinal (Natural Language➡️De novo Proteins) Model weights + demo released! 🔗 Demo: denovo-pinal.com/ 🌟 16B parameters, trained on 1.7B text-protein pairs. 📈 Scaling data + model size = 🔥 results! Impressive scaling! 📄 Paper: biorxiv.org/content/10.1101/…

SaProt: Protein Language Modeling with Structure-aware Vocabulary A 650M protein language model trained with 64 80G A100 for 3 months. A good alternative to the ESM family. thanks @sokrypton biorxiv.org/content/10.1101/…

with @KevinKaichuang Our paper about deciphering #AlphaFold as protein function predictor accepted in #NeurIPS2022. First paper since we started biologcal AI from last year. arxiv.org/abs/2206.06583 Some updates will be available later

I tried another protein design case study: 1）Generated immunoglobulin protein via Pinal 2）Predict the generated sequence using AF3 3) Predict its function using ProTrek and Evolla It seems they are consitent? Not sure

We have trained ESM3, a generative bidirectional masked language model that reasons over the sequence, structure, and function of proteins. ESM3 is trained at three model scales - 1.4B, 7B, and 98B.

🧬 Design PETase with Pinal: 1️⃣ ChatGPT → Describe PETase 2️⃣ Pinal → Input text then run it 3️⃣ AlphaFold3 → Structure prediction 4️⃣ Evolla → Function validation 🚀 Simple yet powerful! Easy! Try：denovo-pinal.com/ #SyntheticBiology #ProteinDesign #AI4Science

#ICLR2024 We'll be at Halle B #33 on 10 May 4:30 p.m. If you are interested in Protein Language Modeling, feel free to reach out! Hope we could have deep communications with all you guys!😆😆😆

Deep learning methods aid in de novo design of proteins to neutralize lethal snake venom toxins in vitro and protect mice from a lethal neurotoxin challenge. nature.com/articles/s41586-0… #NBThighlight

Now everyone customize/share protein language models for their custom task/dataset via @GoogleColab 🤓 Paper: biorxiv.org/content/10.1101/… Colab: colab.research.google.com/dr… Credit: @LTEnjoy, Zhikai Li, @ChenchenHa42849, @BonnieSwt, Junjie Shan, @XibinBayesZhou, Dacheng Ma, @duguyuan

Benchmarking text-integrated protein language model embeddings and embedding fusion on diverse downstream tasks - Benchmark six tpLMs (OntoProtein, ProteinDT, ProtST, ProteinCLIP, ProTrek, ESM3) against ESM2-3B on six tasks (GB1, GFP, AAV, Location, Meltome, Stability) - No tpLM outperforms consistently, with ProTrek and OntoProtein ranking first 3 and 2 times - Concatenate average embeddings and search for the optimal embedding combination heuristically to achieve the best benchmark performance Preprint: biorxiv.org/content/10.1101/…

My student Jin will present Saprot at #ICLR2024. We're thrilled to share that our Saprot model (checkpoint version from last October) achieved 1st place on the Proteingym benchmark (github.com/OATML-Markslab/Pr…) in last month. Happy to see some new PLM with structural alphabet.

Zhikai uploaded a 6-min tutorial for SaprotHub! 🚀 Biologists can now easily train & share their protein language models. Join us, be a SaprotHub author! #Bioinformatics #ProteinModeling @LTEnjoy @sokrypton Paper: biorxiv.org/content/10.1101/… Video: piped.video/watch?v=r42z1hvY…

ColabSaprot is really very impressive... Fine-tune a state-of-the-art protein language model by just uploading a csv of proteins and values. colab.research.google.com/gi… Or download other people's models from huggingface.co/SaProtHub

Used SaprotHub to predict mutations for eTDG, a uracil-N-glycosylase variant. 🧬 Lab results: 17 out of top 20 mutations had higher T-to-G editing efficiency than wild type (marked as red), with 3 showing nearly 2x improvement! 🚀

Introducing ESM Cambrian. Unsupervised learning can invert biology at scale to reveal the hidden structure of the natural world. We’ve scaled up compute and data to train a new generation of protein language models. ESM C defines a new state of the art for protein representation learning.

Gratifying that the original idea of FunGeneTyper in 2017 is finally realized and online by 2022 thanks to joint efforts from our Westlake students and PIs 👍@Westlake_Uni @Westlake_SOE @duguyuan @jyang1981 . biorxiv.org/content/10.1101/…

Had a day to reflect on the release of ESM3, and just wanted to share a few thoughts (and a few shameless highlights of my lab's work! 😅). Before that, for the people who know our stuff, you know that I am an ESM evangelist: I think pLMs will be the future of protein design. 🪄 But it's super important for my lab to understand strengths and weaknesses! To the few points: The Good: -ESM3 uses progressive unmasking for generation. I know a lot of people are like, why not just do next-token? MLM is a way more natural, representative strategy of nature's evolutionary "generative" process, where mutations arise epistatically to confer higher fitness. We've found significant success ourselves with de novo binder generation via span MLM on ESM-2-650M latents (we didn't find the same success with GPT-like models). Check out our PepMLM model with @LeoTZ03: arxiv.org/abs/2310.03842 -Overall, you should not sleep on BERT-like models: they are great generators in many ways, and the same will probably be true for ESM3 (though GFP is probably not enough for validation). We've explored strategies with ESM-2 to perturb latent embeddings with Gaussian noise and decode back into de novo sequences for binder design (which work amazingly in the lab!). Check out our PepPrCLIP model with @bhat_suhaas and @kalyanmpalepu: biorxiv.org/content/10.1101/… -With the largest models trained on 2.78 billion proteins on the MLM task, I have no doubt the model should have excellent unconditional generation/representation capabilities for prediction tasks. As academics, we're thankful that ESM3 will release these models for us to play around with (if we have the compute)! The Not So Good: -Look, I'm a sequence-only guy. I believe all of the useful information of protein properties should be contained in a good sequence representation. I am quite disappointed that ESM3 went with incorporating structure tokens. No doubt this will improve performance for a lot of representation/design tasks (look at SaProt from @duguyuan!) on structured proteins, but this will likely reduce our ability to model conformationally disordered proteins, i.e. transcription factors, which are the most important from a disease/regulatory perspective. My lab has gone in the opposite direction and regularly fine-tune sequence-only ESM models on more disordered sequences, like fusion oncoproteins, and get strong performance. Check out our FusOn-pLM model with @SophieVincoff: biorxiv.org/cgi/content/shor… -What about other special tokens? PTMs, chemical modifications, etc. -- these could have been integrated in training as new tokens. We've described new ways to introduce PTM tokens into pLMs like ESM-2. Doing this for ESM3 will be fun (but potentially difficult with the size of the models)! Check out our PTM-Mamba paper with @pengzhangzhi1: biorxiv.org/content/10.1101/… -Size, size, size. ESM-2-650M is BY FAR the best pLM that balances size and representation capacity. All of our papers (and pretty much every other paper I've read) find this model is optimal for de novo design and downstream prediction tasks, despite being the "medium-sized" ESM. Check out our SaLT&PepPr paper with @garykbrixi: nature.com/articles/s42003-0…. -For academic labs (pretty much the main ones who can use it), it's going to be tough to use the bigger models for optimization, even the open-sourced 1.4B model. Switching away from ESM-2-650M will be a mistake for most applications that don't involve unconditional generation. I hope the ESM3 team will do more ablation studies to prove the model's additional utility! 🥹 The Neutral Finally, ESM3 is available with a non-commercial, academic use-only license. I think this is absolutely the right move (similar to AlphaFold3) to protect EvolutionaryScale's commercial interests while still letting academics push the frontiers of research if ESM3 proves to be useful! However, for some of us that use ESM-like models to develop therapeutics, it will be hard for us to get ESM3-assisted designed molecules to market without commercialization capabilities. That's why I would still recommend continued usage of ESM-2-650M for most tasks -- it's such a good model! 😊 Would love to hear the ESM team's thoughts and would be very open to collaboration! 🌟 @alexrives @TomSercu @proteinrosh @denizzokt @ebetica @THayes427

First text2protein AI model, compressing billions of years of life. 800+ novel, functional and foldable proteins are discovered by researchers. Whitepaper and repo bit.ly/310paper

We release our protein chatGPT, Evola! 🌟 chat-protein.com/ Evola comes in two versions: 10B & 80B. The 80B model has a 1.3B Saprot encoder & a 70B LLaMA3 decoder. Trained on 546 protein question-text pairs with an 150 billion word tokens! 💡🔬 biorxiv.org/content/10.1101/…

Our manuscript "Biophysics-based protein language models for protein engineering" with @romerolab1 is now on bioRxiv. We present Mutational Effect Transfer Learning (METL), a protein language model trained on biophysical simulations, and showcase it for protein engineering. 1/

🌟 Excited to announce AMPLIFY, our latest protein language model that challenges the scaling trend! While current models like ESM2 15B rely on billions of parameters, AMPLIFY achieves superior performance with only 350M parameters. 1/7

Although I welcome more discussion of biosafety in AI, I see condensing safety into a single score as an oversimplification of the issues. 1/

Scaling Structure Aware Virtual Screening to Billions of Molecules with SPRINT 1. SPRINT is a novel vector-based method for drug-target interaction (DTI) prediction, offering unparalleled scalability and speed. It can screen the entire human proteome against a library of 6.7 billion compounds in just 16 minutes. 2. Unlike traditional structure-based approaches, SPRINT leverages structure-aware protein language models (PLMs) to create co-embedding spaces for drugs and protein targets. This allows accurate DTI prediction without explicit 3D modeling. 3. The platform achieved state-of-the-art performance on virtual screening benchmarks, DTI classification tasks, and binding affinity predictions. It offers residue-level interpretability through attention maps, aiding mechanistic insights. 4. SPRINT was validated through large-scale applications, including antimicrobial drug discovery and SARS-CoV-2 NSP13 helicase inhibitor identification, showcasing its utility in identifying diverse, high-quality molecular scaffolds. 5. Using a multi-head attention pooling strategy, SPRINT effectively captures sequence-dependent protein representations, overcoming limitations of previous pooling methods like averaging. 6. The method is highly efficient, using Chroma vector search to handle billions of molecules and proteomes. It reduces computational barriers, enabling pan-proteome DTI screens and virtual screening for drug repurposing. 7. SPRINT’s open-source framework supports modular protein and molecule encoder integration, enhancing compatibility with future PLM developments. It also demonstrated synergy with existing molecular fingerprint methods for property prediction tasks. 8. This approach democratizes virtual screening by delivering accurate, interpretable, and large-scale drug discovery capabilities at a fraction of the computational cost of traditional methods. @david_koes @ericxing @monica_dayao @probablybots 💻Code: github.com/abhinadduri/pansp… 📜Paper: arxiv.org/abs/2411.15418 #DrugDiscovery #VirtualScreening #ProteinLanguageModels #ComputationalBiology

Multimodal Multi-domain Recommendation System DataSet and Benchmark link.medium.com/FlEGaXlgVIb Paper: arxiv.org/pdf/2309.07705.pdf Code: github.com/westlake-repl/Nin…

We did 370 experiments to discover that protein language models primarily learn structure and won't scale for protein function prediction. We need new pretraining tasks! Work led by @francescazfl with @avapamini @yisongyue @alexijielu See Alex's thread + the paper for more!

Our paper TenRec accepted in #NeurIPS2022, a large-recommender system #Recsys dataset, covering 10 recommender tasks, with 4 scenarios & 6 user feedback. We released all baseline codes and will create a leaderboard for benchmarking RS advances. openreview.net/forum?id=PfuW…

Our paper on engineering uracil-N-glycosylase using protein language model ESM is now published in Molecular Cell. @XibinBayesZhou Would love to know if replacing ESM with our Saprot (biorxiv.org/content/10.1101/…) would result in better performance. sciencedirect.com/science/ar…

Use protein language model representations to construct multiple sequence alignments. @clairemcwhite @ProfMonaSingh biorxiv.org/content/10.1101/…