Introducing INTELLECT-3: Scaling RL to a 100B+ MoE model on our end-to-end stack Achieving state-of-the-art performance for its size across math, code and reasoning Built using the same tools we put in your hands, from environments & evals, RL frameworks, sandboxes & more

FP16 can have a smaller training-inference gap compared to BFloat16, thus fits better for RL. Even the difference between RL algorithms vanishes once FP16 is adopted. Surprising!

my goal in life is to join Anthropic, delete all try/except clauses from Claude’s training data, and then quit.

Announcing the completely reimagined vLLM TPU! In collaboration with @Google, we've launched a new high-performance TPU backend unifying @PyTorch and JAX under a single lowering path for amazing performance and flexibility. 🚀 What's New? - JAX + Pytorch: Run PyTorch models on TPUs with no code changes, now with native JAX support. - Up to 5x Performance: Achieve nearly 2x-5x higher throughput compared to the first TPU prototype. - Ragged Paged Attention v3: A more flexible and performant attention kernel for TPUs. - SPMD Native: We've shifted to Single Program, Multi-Data (SPMD) as the default, a compiler-centric model native to TPUs for optimal execution. Dive deep into the new architecture and see the performance benchmarks in our latest blog post! blog.vllm.ai/2025/10/16/vllm… #vLLM #TPU #JAX #PyTorch #AI #OpenSource

Releasing METAGENE-1: In collaboration with researchers from USC, we're open-sourcing a state-of-the-art 7B parameter Metagenomic Foundation Model. Enabling planetary-scale pathogen detection and reducing the risk of pandemics in the age of exponential biology.

Yes sex is great but have you ever had a training run on 10K+ GB200s converge successfully If so could you pls dm me thx

Last night I taught nanochat d32 how to count 'r' in strawberry (or similar variations). I thought this would be a good/fun example of how to add capabilities to nanochat and I wrote up a full guide here: github.com/karpathy/nanochat… This is done via a new synthetic task `SpellingBee` that generates examples of a user asking for this kind of a problem, and an ideal solution from an assistant. We then midtrain/SFT finetune on these to endow the LLM with the capability, or further train with RL to make it more robust. There are many details to get right especially at smaller model sizes and the guide steps through them. As a brief overview: - You have to ensure diversity in user prompts/queries - For small models like nanochat especially, you have to be really careful with the tokenization details to make the task easy for an LLM. In particular, you have to be careful with whitespace, and then you have to spread the reasoning computation across many tokens of partial solution: first we standardize the word into quotes, then we spell it out (to break up tokens), then we iterate and keep an explicit counter, etc. - I am encouraging the model to solve the model in two separate ways: a manual way (mental arithmetic in its head) and also via tool use of the Python interpreter that nanochat has access to. This is a bit "smoke and mirrors" because every solution atm is "clean", with no mistakes. One could either adjust the task to simulate mistakes and demonstrate recoveries by example, or run RL. Most likely, a combination of both works best, where the former acts as the prior for the RL and gives it things to work with. If nanochat was a much bigger model, you'd expect or hope for this capability to more easily "pop out" at some point. But because nanochat d32 "brain" is the size of a ~honeybee, if we want it to count r's in strawberry, we have to do it by over-representing it in the data, to encourage the model to learn it earlier. But it works! :)

Releasing INTELLECT-1: We’re open-sourcing the first decentralized trained 10B model: - INTELLECT-1 base model & intermediate checkpoints - Pre-training dataset - Post-trained instruct models by @arcee_ai - PRIME training framework - Technical paper with all details

Introducing the Environments Hub RL environments are the key bottleneck to the next wave of AI progress, but big labs are locking them down We built a community platform for crowdsourcing open environments, so anyone can contribute to open-source AGI

you should legally be required to disclose what quantization level you are serving your current model at like it was a nutrition label. you should also be banned from dynamically adjusting quantization based on demand without notification. (you know who you are ...)

Announcing INTELLECT-1: the first-ever decentralized training of a 10B model Scaling decentralized training 10x beyond prior efforts. Anyone can join us to build open-source AGI 🦋

Introducing OpenDiLoCo, an open-source implementation and scaling of DeepMind’s Distributed Low-Communication (DiLoCo) method, enabling globally distributed AI model training. primeintellect.ai/blog/opend…

Introducing OpenDiLoCo, an open-source implementation and scaling of DeepMind’s Distributed Low-Communication (DiLoCo) method, enabling globally distributed AI model training. primeintellect.ai/blog/opend…

Announcing our $15m raise — led by @foundersfund. To build our peer to peer compute and intelligence protocol. With participation from @MenloVentures and angels like @karpathy @ClementDelangue @tri_dao @dylan522p @balajis @EMostaque and many others.

You don’t need a PhD to be a great AI researcher. Even @OpenAI’s Chief Research Officer doesn’t have a PhD.

RL training is too slow? AReaL by @AntResearch_ introduces an asynchronous approach that decouples generation from training to eliminate blocking. Combined with system-level optimizations, this method achieves up to 2.57× speedup. Code open-sourced @jxwuyi

To generate the list, we spoke with researchers, scraped Google Scholar, and studied Dwarkesh to identify the visionaries who dedicate their lives to bringing us our cherished tokens. The ranking is determined by votes generously provided by anonymous participants during their time between training runs. We also provide additional context, such as: - Associated institutions - Current/previous company - General interests - Notable papers You can view the updated list at MetisList dot com

Berlin I’m in you

🚀 Introducing our new tech report: Muon is Scalable for LLM Training We found that Muon optimizer can be scaled up using the follow techniques: • Adding weight decay • Carefully adjusting the per-parameter update scale ✨ Highlights: • ~2x computational efficiency vs AdamW • Seamless transition from AdamW to Muon without hyper-parameter tuning • Memory & communication efficient implementation of distributed Muon optimizer. 🎯 Based on these improvements, we introduce Moonlight: Our 3B/16B MoE model trained with Muon on 5.7T tokens, advancing the Pareto frontier with better performance at fewer FLOPs! 🎁 Open-sourcing everything: 📚 Code & implementation: github.com/MoonshotAI/Moonli… 🤗 Full model series (pretrained, instruction-tuned & intermediate checkpoints): huggingface.co/moonshotai 📜 Paper: github.com/MoonshotAI/Moonli… #AI #LLM #OpenSource #MoonshotAI

I cannot wait to release the Intellect-2 technical report and share all we learned about scaling test compute in a distributed / decentralized fashion. Extremely proud of the intense teamwork that went into preparing this fully permissionless training run with async reinforcement learning. Open source for the win

We did it — the first decentralized training of a 10B model is complete! Trained across the US, Europe, and Asia 🌐 Post-training with @arcee_ai is underway, and a full open-source release is coming in ~1 week, including: base model, checkpoints, post-trained model and data.

I've been arguing for something like this for over a year: crowdsourced distributed fine-tuning.

Don't sleep on PipelineRL -- this is one of the biggest jumps in compute efficiency of RL setups that we found in the ScaleRL paper (also validated by Magistral & others before)! What's the problem PipelineRL solves? In RL for LLMs, we need to send weight updates from trainer to generator (to generate data from our latest policy being trained). (Conventional PPO-off-policy) A naive approach would be to "start generators on a batch, wait for all sequences to complete, update the model weights for both trainers and generators, and repeat. Unfortunately, this approach leads to idle generators and low pipeline efficiency due to heterogeneous completion times. (Pipeline-RL) Instead, we simply let the generators continue generating tokens without discarding or finishing ongoing generations in-flight whenever we need to do a weight update -- doing an "in-flight" weight update. As such our KV caches for these generations would be stale, as they would come from LLM with earlier copy(ies) of the weights) but this is ok (see below).

What are the must read papers on distributed training so far from 2025 (and 2024)? Want to brush up on my paper reading skills. I imagine OpenDiLoCo would be one

We are open-sourcing INTELLECT-1, the first decentralized trained 10B model. Including base model, checkpoints, post-trained model, data, technical report and our decentralized training framework.

1T parameters, open-weights, just released on @huggingface!

ablations are for the weak. just yolo your runs. (ok, do some small amount of ablations, but don't over do it). instinct is everything in ML and AI.

Types in Python are a good idea until you open your profile trace and see this

Introducing @mike64_t's work on "Recurrence-Complete Frame-based Action Models" A paper on why long-horizon perception requires rethinking recurrence.

sonnet + prompts is not enough to build a company unless you're the company that built sonnet

Decentralized Training in the Inference-Time Compute Paradigm Reinforcement learning has fundamentally different communication infrastructure requirements than pre-training, making globally distributed training the path forward. Read more in our blog: primeintellect.ai/blog/intel…

We did it — collaboratively generating the largest synthetic dataset of verified reasoning traces for math, coding and science using DeepSeek-R1. SFT fine-tune is underway and a full open-source release including dataset and post-trained models is coming in the next few days.

Today, we're announcing our first hosted infrastructure product: pyx, a Python-native package registry. We think of pyx as an optimized backend for uv: it’s a package registry, but it also solves problems that go beyond the scope of a traditional "package registry".

We released two open-weight reasoning models—gpt-oss-120b and gpt-oss-20b—under an Apache 2.0 license. Developed with open-source community feedback, these models deliver meaningful advancements in both reasoning capabilities & safety. openai.com/index/introducing…

shout out to our entire team @samsja19 @jackminong @jannik_stra @burnpiro @leonardofed @manveerxyz @MatternJustus @matthewdif @jzheng1994 @mikasenghaas @apaz_cli @felix_red_panda @mike64_t @afurgs @Grad62304977 @madisenxtaylor @_mario_neo_

I wish more people understood this. I am all for startup building on top of a foundation model and focus on a good product rather than research. But the paradigm of API + prompt is just not powerful enough for building a truly innovative product. Nobody could have built deep research except Open AI. Startup needs access to the lower level stack, base model, compute, software to infer and tune / rl the models to perfectly fit the product needs. It's not just about doing open source because it's cool, or ethical. It is just the best way to create the next trillion dollar companies

Some personal news: I recently joined Cursor. Cursor is a small, ambitious team, and they’ve created my favorite AI systems. We’re now building frontier RL models at scale in real-world coding environments. Excited for how good coding is going to be.

Today we’re launching INTELLECT-2: The first decentralized 32B-parameter RL training run open to join for anyone with compute — fully permissionless. Scaling towards frontier reasoning across coding, math and science.

BF16 -> FP16 is such a simple (one configuration change in Oat) yet fundamental fix for inference-training mismatch. With FP16, the most basic importance sampling PG outperforms all algorithmic fixes in BF16. Let's rethink RL stability from the precision perspective.🔎

We shipped an OSS 'vibe coding platform' (like @v0) built with @vercel AI SDK, Gateway and Sandbox. We worked with @openai to tune the GPT-5 agent loop. It can write/read files, run commands, install packages, autofix errors… Demo oneshotting a multiplayer Pong in Go ↓

Today, we at @OpenAI achieved a milestone that many considered years away: gold medal-level performance on the 2025 IMO with a general reasoning LLM—under the same time limits as humans, without tools. As remarkable as that sounds, it’s even more significant than the headline 🧵

Today, we are releasing: - INTELLECT-MATH, a frontier 7B parameter model for math reasoning - The largest synthetic math dataset to date of 5M verified reasoning traces - An outlook on decentralized training in the inference-compute paradigm primeintellect.ai/blog/intel…

Announcing INTELLECT-1: the first-ever decentralized training of a 10B model Scaling decentralized training 10x beyond prior efforts. Anyone can join us to build open-source AGI 🦋

We're scaling our Open-Source Environments Program As part of this, we're committing hundreds of thousands of $ in bounties and looking for partners who want to join our mission to accelerate open superintelligence Join us in building the global hub for environments and evals

Releasing INTELLECT-2: We’re open-sourcing the first 32B parameter model trained via globally distributed reinforcement learning: • Detailed Technical Report • INTELLECT-2 model checkpoint primeintellect.ai/blog/intel…

Introducing Cursor 2.0. Our first coding model and the best way to code with agents.

the FP8 values in your model after 50 layers of quantize/dequantize operations

uh oh

full stack research engineer: can do pretraining, inference and RL

i will not be there at ICLR either (that much back to back flying wouldve been too much for me lol) but do pay close attention and listen to what @menhguin has to say at the event for our paper :)

We just put out a key step for making distributed training work at larger and larger models: Scaling Laws for DiLoCo TL;DR: We can do LLM training across datacenters in a way that scales incredibly well to larger and larger models!

🚀Excited to share our new work! 💊Problem: The BF16 precision causes a large training-inference mismatch, leading to unstable RL training. 💡Solution: Just switch to FP16. 🎯That's it. 📰Paper: arxiv.org/pdf/2510.26788 ⭐️Code: github.com/sail-sg/Precision…

RL LEARNING WITH LORA: A DIVERSE DEEP DIVE

The obvious and unique possible answer is a single big dataclass and cmd line override of its values Right?

Just announced new versions of Gemma 3 – the most capable model to run just one H100 GPU – can now run on just one *desktop* GPU! Our Quantization-Aware Training (QAT) method drastically brings down memory use while maintaining high quality. Excited to make Gemma 3 even more accessible for more developers.

periodic ❤️ open-source! for example, we’ve been collaborating with the @PyTorch team to build the highest MFU gpt-oss training implementation (includes thinky sinky flexattn) here’s a few SFT runs of gpt-oss-20b & 120b, where i get ~24% MFU for 20b and ~8% for 120b

Originally a while back i got some intuition with it from the query key value perspective which might help (theres also the gradient descent perspective which is good too). Scraped this from a chat with @stochasticchasm a year ago so might be a bit dodgy. Imagine u want to store a database of knowledge in ur state ur gonna do this by storing key value pairs. For example on a high level, a key could be a song name, and the value associated with it is the song lyrics when u want to extract those lyrics, u want to be able to query ur database full of key value pairs with the song name and get back the song lyrics so ur gonna query the database using that key im going to be assuming all keys stored in the database are orthogonal as in a 768 dimension space that assumption holds (nearly orthogonal). With orthogonal meaning that if i query with a key, i get back that keys associated value and nothing else now why linear attention fails is because for example in a 768 dimension space, really all u can have is 768 orthogonal keys so when ur sequence becomes longer, keys (and those key value pairs) start to interfere so now when u query with the song name, u wont get back the song lyrics, but a linear combination of other values for other keys as well which could be unrelated this causes the retrieval to fail as now ur getting the song lyrics and a bunch of noise with it what retnet does is basically take ur state which is all these key value pairs added together, and scale them all down by a scalar fixed factor so now when u query with the song name, u will get the song lyrics and noise, but if ur song lyrics were added in recently, they will have a stronger signal than the noise if that was farther back so it prioritises recent key value pairs added in the state the obvious issue is that if u want to query with the song name but that was stored a long time ago, its signal will be low mamba-2 and g-retnet basically make this scalar value dependent on the sequence so the model can learn how much to reduce the signal of all previous key value pairs. So if ur now storing an important piece of information, ur model can choose to lower the signal (decay) of the state (all previous key value pairs) so that ur new info is stored with a strong signal then rwkv6, gla, mamba turn this scalar into a vector So u can imagine now the model can be more expressive with its decay as it can lower the signal of aspects of the state for example here u have to pay attention to that the vector doesnt mean each previous key value pair gets a different decay scalar, it means that ur lowering the signal of every key of those key value pairs with the same decay, but now that decay is a vector so u can decay parts of the key less and some more (talking abt parts of each key, all keys will get the same decay) delta rule takes a whole different approach it basically states that the ideal state update rule should selectively choose key value pairs to discard meaning instead of the decay which acts the same on all key value pairs we want to be able to target specific key value pairs to remove or lower the signal of more quick clarification: linear attention is "s = s + k^T @ v" so ur state is just a sum of all key value pairs the others are "s = w * s + k^T @ v" where w is a vector or scalar acting on the rows of s (key dimension) the idea behind the delta rule is that if i have my state after 768 tokens, then now i can assume all keys in the state are orthogonal which is the ideal situation. Now if i want to store a new key (and its value) but its the same as one of the keys already in there, i ideally dont want to store both but instead take one out now pay attention that just because the keys are the same or very similar, doesnt mean their values are the same the new key being the same can be attributed to it needing to be the same bcs of the limitation of its space like u can imagine when storing ur key value pairs, each key can choose out of the 768 choices of keys (orthogonal), then a new key comes in thats not related to any before, it needs to choose one of the 768 choices but theyre already all taken so it just chooses one so now when u query for the song name, u get back the sum of 2 values that can be very different and thats not what u want bcs u want the value exactly ideally so every step u have a new key value pair, and an old key value pair where those keys are the same (not in meaning but bcs of the limitations of the space). What the delta rule does is it queries the state with the new key, which means it extracts that old value. Then instead of adding in its new key value pair, it deletes that old value and adds the new one (adds to the state "k^T @ (v_new - v_old)" which equals "k^T @ v_new - k^T @ v_old" and now bcs ur state already has somewhere in there "k^T @ v_old" (bcs the keys are the same), then that old key value pair will be deleted the important thing here is that all other key value pairs in that state are unaffected while in the rnns like GLA, the decay is the same for all previous key value pairs at each step, of course is different at each step but the same for all previous keys and values so now when u query the state, u exactly get back that value instead of the sum of 2 different ones in practice though u dont want to completely delete one, so what u do is interpolate meaning instead of deleting v_old and storing v_new, u delete v_old and store "beta * v_new + (1 - beta) * v_old" instead where beta is a scalar thats dependent on the sequence (current value) so the model based on the actual new keys and values content, can choose how much to store of each also can be viewed as making ur state update dependent on the content of that state, while using decay only like GLA makes that state update not dependent on that state its like delta rule looks into the state, picks out a key value pair, and decides to forget it, with decay u just rip a piece off of each previous key value paper of course, some issues firstly the state update is too slow, as ur only touching one key value at a time, u ideally want to be able to get rid of like 5 at once for example which can hurt length extrapolation also not an issue in general, but in practice the model wont query the state with an exact key and retrieve an exact value, it would query with a superposition of many keys and retrieve a superposition of many values and work with that bcs its smart so it will retrieve some info abt the song lyrics, some info abt the songs history, some info abt other stuff and use it all however it wants to make its prediction but the model still needs to be able to retrieve exactly what it wants which with only linear attn it cant after a certain point gated deltanet then solves the issue of the slow update by using mamba 2 style scalar data dependent decay that means ur doing the delta rule on the state, then on tope of that decaying all key value pairs in that state at once with the same value this lets the model do stuff like a full delete of the state if it wants to Then again KDA and rwkv7 make the decay a vector (only worth it if u can make it fast) also, note that the gated deltanet/KDA decay also helps with the fact that ur still storing interpolations of values which is better than storing both but worse than storing one of them if u want exact retrieval meaning, imagine the model was doing a math question, then u asked it abt space, its not fast enough to make a large delete in its state so has to keep info abt the math even if it doesnt want to, so the decay helps it make a quick sweep its still a fixed state size so of course it has to lose info but linear attention cant lose info which is bad decay on its own can lose info but has to lose the same amount of info from all key value pairs, delta rule can lose info but target specific key value pairs it wants to lose info from while keeping all others perfectly stored and yes u lose some info with the interpolation but not as much as u think as u have many layers and its all a soup of info so it could store parts of info in one key value pair, then another part in another then retrive both and mix or whatever but the main idea is u specifically target key value pairs to lose info from

Launching SYNTHETIC-2: our next-gen open reasoning dataset and planetary-scale synthetic data generation run. Powered by our P2P inference stack and DeepSeek-R1-0528, it verifies traces for the hardest RL tasks. Contribute towards AGI via open, permissionless compute.