I present the fix for RL training instability:

🚀Excited to share our new work! 💊Problem: The BF16 precision causes a large training-inference mismatch, leading to unstable RL training. 💡Solution: Just switch to FP16. 🎯That's it. 📰Paper: arxiv.org/pdf/2510.26788 ⭐️Code: github.com/sail-sg/Precision…

🚀 No More Train–Inference Mismatch! We demonstrate bitwise consistent on-policy RL with TorchTitan (training) + vLLM (inference) — the first open-source run where training and inference numerics match exactly. It only takes 3 steps: 1️⃣ Make vLLM batch-invariant (same seq → same output regardless of batching) 2️⃣ Ensure forward passes in training use identical kernels as inference 3️⃣ Add custom backward passes in PyTorch ✅ Verified on Qwen3 1.7B + GSM8K: • batch_inv_ON (bitwise exact) → KL=0.0, faster convergence, higher reward • batch_inv_OFF → reduced reward, instability We audited every op, imported vLLM’s fused kernels (SiLU MLPs, RMSNorm+residual), and wrote matching backward passes. Run is fully on-policy, deterministic, and reproducible. Next: • Unified model code • torch.compile support • Perf tuning (current bitwise RL ≈2.4× slower) • Broader model + op coverage 🔗 blog.vllm.ai/2025/11/10/bitw… #vLLM #TorchTitan #RL #LLM #AIResearch

Thanks for the thought! Some further thoughts (clarifications): 1. Reasonably designed algorithms (let’s also include precision in the design space) should not collapse on small data. It’s just like if my CNN cannot even overfit MNIST, how can I trust it will master 1000-class imagenet? 2. We do have experiments on “larger” dataset X bigger model (30A3 MoE) X H100 GPUs. The performance improvement is clear.

Well sort of, with just GRPO and not actually taking care of the mismatch at the algorithm level, u will encounter instability with bf16 under normal training settings like here (and as many papers for actual models like Kimi linear have mentioned). Their point is that given these instabilities occur in the long term bcs of the mismatch, patching it up with an algorithm fix is great but unclear if that just delays the instability (for now that’s what it looks like). Also even if we’re not unstable, it could turn into just lower performance (higher mismatch is sort of just training more off policy, if we can be more on policy for free than it’s likely we would get a performance boost too)

💥 We've achieved perfect training-inference alignment for SGLang & FSDP in slime! (Flash Attn 3, DeepGEMM, etc.) The result? A strict KL divergence of 0. But here's the twist: We spent a month trying to find a baseline that crashes from mismatch... and couldn't. 🤷‍♂️ We haven't found a significant difference from the baseline yet. We're calling on the community: Send us your reproducible training collapse examples! We want to see them 🤣🤣

Well, not only A100. Here is the sanity check on H200 (GRPO, 32B dense model). The authors also mention that they did some larger-scale experiments on H100.

I think if you zoomed into the paper too you’d find it just as if not more interesting.

Recently, we @QPHutu collaborated with VERL maintainers to add a new feature in v0.6.1： 🚀FP16 training (FSDP) & inference, mitigating the mismatch issue. Give it a try! PR: github.com/volcengine/verl/p… 😆PS: Open source commuity is awesome!

I did indeed encounter a collapse in training reward during RL on H-series GPUs, but that only occurred when training for >10 epochs with a small dataset (e.g., 3k). I attributed this collapse to overfitting. hadn't considered that it might also be related to precision.🤔

Optimizing Anytime Reasoning via Budget Relative Policy Optimization

Thanks for this fix. Actually it is not like this easy, GradScaler should be introduced to avoid gradient underflow, otherwise the performance can be even worse than BF16. See: docs.pytorch.org/docs/stable… VeRL Example: github.com/sail-sg/Precision…

Just to scare the timeline a bit more, I was observing that RL with the data formatted as in the first image would crash and go unstable while image 2 was fully stable

bf16 / fp16 plot training and inference miss match yes fp16 has close on policy also interstingly vllm bf16 + hf fp16 >> hf bf16 + vllm fp16

This is great. Another good reminder that you really need to understand your entire system to make sure it works properly. Sometimes I feel that the LLM field is evolving so fast that maybe no one in the world fully understands the agents we’re building anymore.

As we all know large reasoning models are extremely overthinking even when answering 1+1. Many approaches have been proposed to mitigate this issue, but it doesn't need to be that complex -- we (and several concurrent works) found that just continuing standard RL training while truncating long responses is surprisingly effective. Why? The key is the step function reward implicitly defined in context window truncation. As such, we provide a unified view to formulate various length-based reward shaping approaches, and design several SOTA variants based on step function. We achieved +6.1 acc on AIME based on R1-distilled Qwen model while reducing token usage by 63%. I really like the visualization in the Table👇to compare various length-based reward-shaping methods . Check @WeiLiu99 post and our paper for more details!

People are racing to push math reasoning performance in #LLMs—but have we really asked why? The common assumption is that improving math reasoning should transfer to broader capabilities in other domains. But is that actually true? In our study (arxiv.org/pdf/2507.00432), we evaluated over 20 open-weight reasoning models and found that: ➡️Only models trained with RL exhibit broad transfer of math reasoning skills to other tasks. ➡️Models trained with SFT show limited or no transfer—especially to non-reasoning domains. To quantify this, we introduce the Transferability Index (TI), which measures how much gain in math could transfer to others. A positive score indicates effective transfer; a negative one suggests loss of general capability. We evaluate the models on three benchmark categories: - Math reasoning: MATH-500, AIME24/25, Olympiad - Other reasoning: GPQA-D (Science), LiveCodeBench2 (Code), ACPBench (Agent Planning), HeadQA (Medical) - Non-reasoning: CoQA (Conversational QA), IFEval (Instruction Following), HalluEval (Hallucination), MC-TACO (Commonsense) Our findings challenge the blind pursuit of leaderboard performance in math reasoning via SFT. Simply creating more math-like SFT data may inadvertently harm a model’s broader generalization. Instead, RL appears to be key for truly transferable reasoning development.

⚙️ Looking closer into GRPO: there is a "clipping bias" that amplifies high-prior model behaviors. Code reasoning could be one of the magical behaviors for Qwen-Math💻 Empirically, we disabled clipping (fig.)-the gains disappeared‼️

🪂Understanding R1-Zero-Like Training: A Critical Perspective * DeepSeek-V3-Base already exhibits "Aha moment" before RL-tuning?? * The ever-increasing output length in RL-tuning might be due to a BIAS in GRPO?? * Getting GRPO Done Right, we achieve a 7B AIME sota! 🧵 📜Full details: github.com/sail-sg/understan… 🛠️Code: github.com/sail-sg/understan…

Really interesting work for reasoning performance given any budget: - Better than GRPO at anytime including the final performance - Provide flexibility for users to trade off the cost and performance - Great engineering effort - code optimized for tree-like generation & training

We've always been excited about self-play unlocking continuously improving agents. Our insight: RL selects generalizable CoT patterns from pretrained LLMs. Games provide perfect testing grounds with cheap, verifiable rewards. Self-play automatically discovers and reinforces reasoning strategies. We introduce SPIRAL, where models learn reasoning by competing against themselves in games, creating an infinite curriculum without human supervision. Training LLMs with self-play RL on Kuhn Poker improves math reasoning by 8.7% average. Just playing Kuhn Poker improves Minerva Math scores by 18.1 points! 🃏 🔗 Paper: huggingface.co/papers/2506.2… 🧑‍💻 Code: github.com/spiral-rl/spiral

I believe our work offers a promising technique for building reasoners with fine-grained budget control— like what Gemini 2.5 Pro has just introduced. Truncated at any time, we deliver the best-effort solution!

🚀 Day 4 of #OpenSourceWeek: Optimized Parallelism Strategies ✅ DualPipe - a bidirectional pipeline parallelism algorithm for computation-communication overlap in V3/R1 training. 🔗 github.com/deepseek-ai/DualP… ✅ EPLB - an expert-parallel load balancer for V3/R1. 🔗 github.com/deepseek-ai/eplb 📊 Analyze computation-communication overlap in V3/R1. 🔗 github.com/deepseek-ai/profi…

🚀 New at #ICML25: PipeOffload breaks the memory wall for LLM training with Pipeline Parallelism — linear activation memory scaling with zero performance loss. High memory usage is no longer a blocker. 👉 Follow our series of work pushing limits of PP!

♨️DualPipe open sourced by Deepseek is hot. ❓However, DualPipe could be better without the Dual! ✅By simply cutting the schedule in half, we can reduce the unnecessary parameters duplication. 💡Check our blog on hackmd.io/@ufotalent/r1lVXsa…

Some of the best reasoning-related papers I've seen in the past three months—truly impressive work. Another great contribution from the SAIL, following the success of Dr.GRPO.

💩👇Anyone can tell me what algorithm this bullshit is?👇 💩 🏊I can understand someone just want to sell a concept quickly in the paper. It's ok the algorithm is not perfect and you don't want to release code.🥇 But why you release such a bullshit?😡

Anytime reasoning, a topic dating back to the ’90s, seeks the best-effort solution given a computation bound. For large reasoning models, we optimize anytime reasoning with 1) dense rewards and 2) better credit assignment (BRPO). For more details👇

In the era of experience, we're training LLM agents with RL — but something's missing... We miss the good old Gym! So we built 💎GEM: a suite of environments for training LLM 𝚐𝚎𝚗𝚎𝚛𝚊𝚕𝚒𝚜𝚝𝚜. Let’s build the Gym for LLMs, together: axon-rl.notion.site/gem

Pretty disappointed with the @NeurIPS rebuttal.😔 Initial scores: 5, 4, 4, 3. Zero response from two 4-score reviewers. 👌 The 3-score reviewer only want to reject, regardless of the rebuttal.🤮

To be honest, I don't think such truncation is a reasonable setting. In an online service, even if the thinking is too long and be truncated, it should provide the best summary so far. Simply regarding them as incorrect doesn't make sense for a product.

To be honest, I don't think such truncation is a reasonable setting. In an online service, even if the thinking is too long and be truncated, it should provide the best summary so far. Simply regarding them as incorrect doesn't make sense for a product.

Though not attending #ICML2025 in person, I'm super excited to share 3 accepted papers: 1.🎊Best Paper Honorable Mention @ AI4MATH workshop: Understanding R1-Zero-Like Training: A Critical Perspective (a.k.a Dr. GRPO but I think the paper is more than this loss fix) 2. Main Conference Spotlight: Continual Reinforcement Learning by Planning with Online World Models; I worked on the continual learning topic for one year before diving into RL for LLMs openreview.net/pdf?id=mQeZEs… 3. AI4MATH Workshop Poster: Optimizing Anytime Reasoning via Budget Relative Policy Optimization; a dense reward training framework that controls thinking budget (a cool feature of Gemini 2.5 Pro)

💩👇Anyone can tell me what algorithm this bullshit is?👇 💩 🏊I can understand someone just want to sell a concept quickly in the paper. It's ok the algorithm is not perfect and you don't want to release code.🥇 But why you release such a bullshit?😡

Our view on test-time scaling has been to train models to discover algos that enable them to solve harder problems. @setlur_amrith & @matthewyryang's new work e3 shows how RL done with this view produces best <2B LLM on math that extrapolates beyond training budget. 🧵⬇️ Website: matthewyryang.com/e3/ Paper: arxiv.org/abs/2506.09026

This is a part of our Zero Bubble Pipeline Parallelism series, pushing throughput🚀, memory💾 and scalability📈of LLM training to extreme! Project link: github.com/sail-sg/zero-bubb…