An early beta of Grok Build, an agentic CLI for coding, building apps, and automating workflows is now available for SuperGrok Heavy subscribers. Through this early beta, we will improve the model and product based on your feedback. Try it at x.ai/cli

:) Original plots come from nitter.app/RichardYRLi/status/197… - also their blog is super good! - still unsure if the FP16 vs BF16 debate is due to hardware issues due to FP32 accumulation sizes - planning to run some experiments!

LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA. thinkingmachines.ai/blog/lor…

Everyone’s talking about Kimi K2 Thinking and its impressive performance. No full report yet, but judging from Kimi K2\1.5 reports, it likely uses Policy Mirror Descent - an RL trick that’s quietly becoming standard in frontier labs. Let’s break down what it is:

We're still (mostly) in the massively low sample regime for LLMs+RL. Recall that Montezuma's Revenge and other sparse reward settings would take O(1-10M) steps/frames before starting to actually learn anything at all.

Inspired by @thinkymachines 's "#LoRA Without Regret" post, I formalized their insight that policy gradient learns ~1 bit per episode via Bayesian #RL formulation. I prove this is a hard information-theoretic ceiling and extend the analysis to actor-critic methods. Full writeup with theorems: richardli.xyz/post/informati… 🧵

🚀 No More Train–Inference Mismatch! We demonstrate bitwise consistent on-policy RL with TorchTitan (training) + vLLM (inference) — the first open-source run where training and inference numerics match exactly. It only takes 3 steps: 1️⃣ Make vLLM batch-invariant (same seq → same output regardless of batching) 2️⃣ Ensure forward passes in training use identical kernels as inference 3️⃣ Add custom backward passes in PyTorch ✅ Verified on Qwen3 1.7B + GSM8K: • batch_inv_ON (bitwise exact) → KL=0.0, faster convergence, higher reward • batch_inv_OFF → reduced reward, instability We audited every op, imported vLLM’s fused kernels (SiLU MLPs, RMSNorm+residual), and wrote matching backward passes. Run is fully on-policy, deterministic, and reproducible. Next: • Unified model code • torch.compile support • Perf tuning (current bitwise RL ≈2.4× slower) • Broader model + op coverage 🔗 blog.vllm.ai/2025/11/10/bitw… #vLLM #TorchTitan #RL #LLM #AIResearch

(6/x) The principled solution is Sequence-Level Importance Sampling. It provides an unbiased gradient correction by accounting for the entire trajectory's probability, restoring training stability. We also propose Masked Importance Sampling (MIS), which further improves performance.

Thanks AK for sharing our work! 🔥 🧵 Back to Jan when we just started this project... we were living a nightmare 😩 Months of watching our multi-turn RL models collapse. Every. Single. Time. 💥 We thought we were doing something wrong... until we discovered other research teams seem to hit the same invisible wall (Devin, verl and other reported issues) 🧱 Multi-turn tool reasoning just... BROKE 💔 It's NOT like the elegant simplicity of R1-Zero’s approach. This was pure chaos. Then came the "aha!" moment of our SimpleTIR💡✨ The secret was hiding in plain sight: “void turns” - those meaningless steps where models generates text that leads... absolutely NOWHERE 🕳️ One simple filter changed everything ✨ Our 7B model jumped from 22% (DAPO) to 50% (Multi-Turn Tool Use) on AIME24 📈 No complex algorithms, no fancy techniques. Just removing the void turn examples that were poisoning the training 🎯 Sometimes, the biggest gains come from understanding what NOT to learn 💡 📄 Paper: huggingface.co/papers/2509.0…
💻 Code: github.com/ltzheng/SimpleTIR ✍️ Blog: simpletir.notion.site/report

LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA. thinkingmachines.ai/blog/lor…

@karpathy I think it would be good to distinguish RL as a problem from the algorithms that people use to address RL problems. This would allow us to discuss if the problem is with the algorithms, or if the problem is with posing a problem as an RL problem. 1/x

Is there a hyperparameter crisis in reinforcement learning? Let's talk about it at NeurIPS. #NeurIPS2024

@danielhanchen, glad you liked the post! You're spot on to suspect lower-level implementation issues. That's exactly what we found in the original blog. The disable_cascade_attn finding (Sec 4.2.4) was the symptom, but the root cause was that silent FlashAttention-2 kernel bug we detailed, which was particularly problematic on A100s: On A100 GPUs (and L20s), under certain batch/seq lengths, the kernel triggers its split_kv path. This path had a bug that incorrectly transposed the LSE layout. This corrupted tensor caused a "complete precision collapse" in Cascade Attention, which is what we measured as that massive mismatch KL. Good luck with the FP16/BF16 tests, I'd love to know what you find! yingru.notion.site/When-Spee…

I'll be at #ICML2024 from July 21st to 27th! Check out the picture for detailed locations and times of my presentations. Also, don't miss my lightning talk on "Agile #Human-#AI #Collaboration for #RiskOversight" at the #AlignmentWorkshop @farairesearch on 21st and 22nd. #RL #Probability #GPT #Martingale #Statistics #Uncertainty #OnlineLearning #decisionmaking #Moderation #Alignment #Risk #Oversight #Dynamics #ICML #conference #Vienna #FoundationModel #LLMs #exploration #poster #HyperAgent #Theory #Practice #math

@danielhanchen, glad you liked the post! You're spot on to suspect lower-level implementation issues. That's exactly what we found in the original blog. The disable_cascade_attn finding (Sec 4.2.4) was the symptom, but the root cause was that silent FlashAttention-2 kernel bug we detailed, which was particularly problematic on A100s: On A100 GPUs (and L20s), under certain batch/seq lengths, the kernel triggers its split_kv path. This path had a bug that incorrectly transposed the LSE layout. This corrupted tensor caused a "complete precision collapse" in Cascade Attention, which is what we measured as that massive mismatch KL. Good luck with the FP16/BF16 tests, I'd love to know what you find! yingru.notion.site/When-Spee…

Prior-dependent analysis of posterior sampling reinforcement learning with function approximation ift.tt/CVNAZhy

Thrilled to introduce Adam-mini, an optimizer that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini can also achieve 49.5% higher throughput than AdamW on Llama2-7B pre-training. The design of Adam-mini is inspired by certain Hessian structures we observed on Transformers. Feel free to try it out! Try switching to Adam-mini with the same hyperparams of AdamW, it would work with only half memory. Hope Adam-mini can help save time, cost, and energy in your tasks! Paper: "Adam-mini: Use Fewer Learning Rates To Gain More" arxiv.org/abs/2406.16793 Code: github.com/zyushun/Adam-mini

RL and Bayes are not machine learning subfields. They are the axioms of life and universe. You can't get rid of them.

Great keynote by David Silver, arguing that we need to re-focus on RL to get out of the LLM Valley @RL_Conference

I agree with @karpathy 's take here. The interview between @RichardSSutton and @dwarkesh_sp was interesting, but I think at times there was a communication gap due to some misunderstandings. I would say that the current LLM training setup is very similar to the classic model-free RL setup, except that with LLMs: (1) the policy is warm-started from a supervised model (no de-novo, self-directed learning); (2) there is a train/test distinction (no continual learning); (3) most of the observation stream comes from human words, which already "carve nature at its joints", bypassing the harder problem of learning useful abstractions from raw sensorimotor streams. (4) when using multimodal models, the perceptual encoder is usually pre-trained and frozen, and often relies on a lot of human engineering (eg contrastive losses, or pixel-prediction losses) to come up with a good set of (soft) tokens. Most of the interview seem to focus on issue #1. However, the discussion seemed confused here due to the fact that LLMs are both a world model (predict what humans would typically say) and a policy (predict what the agent should do). Obviously the model from the supervised pretraining stage is not action-conditioned, so Sutton does not want to call it a WM - but it is a predictor of future observations given the past, so it's like a WM that marginalizes over actions (resulting in a mixture). The WM is then converted into a (goal-conditioned) policy using IFT (imitation learning) and then improved with RLFT, which further confuses the discussion. In current practice, the RLFT stage mostly just uses human provided reasoning tasks, which are bandit problems that do not involve interacting with an environment. But there is a recent move towards true multi-step RL, where LLMs do learn from external environments, as in classic RL. This fact was not emphasized enough in the interview, IMHO. Andrej argues that warm-starting is a practical alternative to evolution's outer meta-learning loop, and I agree, so I don't have a problem with #1. But I do agree with Sutton's criticisms #2-#4. In particular, I expect a lot of future progress to come from continual RL applied to multimodal problems (eg. visual GUI-using agents) in non-stationary multi-agent environments (e.g., e-commerce or embodied AI), where the agent learns its own abstractions over time (eg creating tool libraries), it learns both a (goal agnostic) world model and a (goal conditioned) policy (so it can do decision time planning), and both kinds of model become semi-parametric (eg. combining memories and ICL with gradient-based weight updates). Future agents will not just be a frozen "omni-transformer", consuming and generating tokens, they will be heterogeneous adaptive systems, with many different specialized modules, more like the brain. (This may make serving hard, but who said intelligence would be easy to reproduce?) I think Sutton will like this new paradigm more :)

#Tencent TStarBots Defeat @StarCraft II's Powerful Builtin #AI in the Full Game syncedreview.com/2018/09/21/…

I will be at #NeurIPS2023 next week. Please DM me if you would like to chat about anything related to #ReinforcementLearning / Decision-making #agents / Random Projection for Efficient #Algorithms😋

1/ David Blackwell. Leagues ahead of his time: "What is a good prediction strategy, and how well can you do?" While some things do not seem possible, "Looking for a p [probability] that does well against every x [an outcome] seems hopeless", Blackwell does give us a strategy:

I'll be giving a 2-hour tutorial on data-efficient learning with my PhD student @sjoshi804 on Monday July 22 at #ICML2024. Join us to learn more about this cool topic! ➡️ We can learn better from better data! ⬅️🙌🌱

It makes sense as by the Autoregressive modeling. But if it is intended to optimize combinatorial action set at the beginning, then it is open loop and we do even know how to optimize it in LLM. This implies the underlying MDP is always token level.

Quoted from Ben Van Roy (Stanford Professor): @karpathy piped.video/fERcdhg0Kds?si=lhVj…

(3/x) Here's the smoking gun: the training-inference mismatch isn't uniform. It's catastrophically worse for these exact low-probability tokens. The training framework assigns them near-zero probability, while the inference engine saw them as merely unlikely. This massive divergence causes collapse.