Announcing Grok for Government - a suite of products that make our frontier models available to United States Government customers We are especially excited about two new partnerships for our US Government partners 1) a new contract from the US Department of Defense 2) our products being available to purchase via the General Services Administration (GSA) schedule. This allows every federal government department, agency, or office, to purchase xAI products We're hiring mission driven engineers who want to join the cause

New: @perplexity_ai just dropped its first celeb ad starring Squid Game's Lee Jung-jae. The AI search startup takes a jab at @Google “Poogle”—with a cheeky line: “Don’t use glue” when Jung-jae asks how to make cheese stick to pizza. 💰 Mid-seven-figure buy

My estimate of the probability of Grok 5 achieving AGI is now at 10% and rising

this is beyond mindblowing for me. somebody built a 5 million param language model inside minecraft, trained it, equipped it with basic conversational ability. probably the best thing i have seen entire month.

FP16 can have a smaller training-inference gap compared to BFloat16, thus fits better for RL. Even the difference between RL algorithms vanishes once FP16 is adopted. Surprising!

Claude Code can now ask you interactive questions when it needs more information or when there are multiple paths forward.

sad but true ...

Hot-take: Auto-regression sucks and is impressive as a parlor trick. Any spark of intelligence from an LLM reflects that it moved beyond, and built a factorized model with meaningful latents.

Announcing Bread Technologies. We’re building machines that learn like humans. We raised a $5 million seed round led by Menlo Ventures and have been building in stealth for 10 months. Today, we rise 🍞

my workflow to get the best out of claude code is complete now.

Wanted to get better intuitions for how RL works on LLMs. So I wrote a simple script to teach Nanochat to add 5 digit numbers. I was surprised at how fast it learned. Until I looked at the model's generations and realized that it had just learned to always call the built-in Python interpreter 😂. The code I wrote is very remedial, minimal, and inefficient - I'm a professional podcaster, alright? But it might be helpful if you just want to see the basics of how REINFORCE or GRPO work. Link to gist below. Fundamentally, it's not that complicated: generate multiple trajectories per prompt. Update your model to make it more likely that it samples all the tokens in the successful trajectories.

hypothetically, if one wanted to research at a frontier lab in 1.5 years (hypothetically winter 2027) and wanted to know how to develop the necessary skills and credentials to do so, what would you suggest to them (asking for a friend)

intermediate level workflow for claude code is complete taught claude to >create commands, >manage its own commands, >extract entire session and save it like os paging, >do multi agent deep research locally, >analyse functions in a specific style, >search the repo effectively, >work with user created scripts. post scheduled for 3.5 hours later. enjoy.

muon is nice, ever since i got it working i can notice the way the output token quality is just better? probably varied from Adam optimizer tokens would be a better way to phrase it. but these models with different optimisers are different even with the same data.

y'all ain't ready for this token order prediction (TOP) early preprint coming next week. it's promising enough for me to get a checkmark to boost the paper a bit, let's see

Several of my team members + myself are impacted by this layoff today. Welcome to connect :)

Hello Thermo World.

REINFORCE is coming back in a big way

Kimi Linear Tech Report is dropped! 🚀 huggingface.co/moonshotai/Ki… Kimi Linear: A novel architecture that outperforms full attention with faster speeds and better performance—ready to serve as a drop-in replacement for full attention, featuring our open-sourced KDA kernels! Kimi Linear offers up to a 75% reduction in KV cache usage and up to 6x decoding throughput at a 1M context length. Key highlights: 🔹 Kimi Delta Attention: A hardware-efficient linear attention mechanism that refines the gated delta rule. 🔹 Kimi Linear Architecture: The first hybrid linear architecture to surpass pure full attention quality across the board. 🔹 Empirical Validation: Scaled, fair comparisons + open-sourced KDA kernels, vLLM integration, and checkpoints. The future of agentic-oriented attention is here! 💡

The quickest “oops” I’ve ever seen from an LLM:

TL;DR: LLMs don’t reason, and LLMs with RL still don’t.

Cursor vs. Cognition have opposite takes on agent search: - Cursor: Custom embeddings trained on agent traces improve accuracy by 12.5% - Cognition: Embeddings are counterproductive, so we trained models to use grep with 8x parallel tool calls. Both have benchmarks. Hmmm..

This ended up taking a lot of text to answer, so you've got a new section in the manual. Read "What you should expect to be compilable" docs.google.com/document/d/1…

llama2.mojo on M2 MBP seems to be much faster than llama2.c I didn't do any optimisations from my side. llama2.mojo, tinyllama 15M: ~434 tok/s llama2.c: ~120 tok/s Trying if I need to compile with optimisations for llama2.c for Apple sillicon.

We’re rolling out new weekly rate limits for Claude Pro and Max in late August. We estimate they’ll apply to less than 5% of subscribers based on current usage.

.@StanfordCRFM's Marin project has released the first fully open model in JAX. It’s an 'open lab' sharing the entire research process - including code, data, and logs, to enable reproducibility and further innovation. developers.googleblog.com/en…

New Zuck post, what a difference a few years makes: Today: "We'll need to be rigorous about mitigating these risks and careful about what we choose to open source." 2024: "Meta is committed to open source AI... and therefore a platform that will be around for the long term."

Replicate IMO-Gold in less than 500 lines: gist.github.com/faabian/39d0… The prover-verifier workflow from Huang & Yang: Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipeline (arxiv.org/abs/2507.15855), original code at github.com/lyang36/IMO25/

Nice paper for a long read across 114 pages. "Ultimate Guide to Fine-Tuning LLMs" Some of the things they cover 📊 Fine-tuning Pipeline Outlines a seven-stage process for fine-tuning LLMs, from data preparation to deployment and maintenance. 🧠 Advanced Fine-tuning Methods Covers techniques like Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO) for aligning LLMs with human preferences. 🛠️ Parameter-Efficient Fine-Tuning (PEFT) Techniques Discusses methods like LoRA, QLoRA, and adapters that enable efficient fine-tuning by updating only a subset of model parameters. 🔬 Evaluation metrics and benchmarks for assessing fine-tuned LLMs Includes perplexity, accuracy, and task-specific measures. Benchmarks like GLUE, SuperGLUE, TruthfulQA, and MMLU assess various aspects of LLM performance. Safety evaluations using frameworks like DecodingTrust are also crucial for ensuring responsible AI deployment. 💻 Explores various deployment approaches and optimization techniques to enhance LLM performance and efficiency in real-world applications. 🌐 Examines the extension of fine-tuning techniques to multimodal models and domain-specific applications in fields like medicine and finance.

GPT-5: lost 71% in a week. Qwen 3 Max: gained 70% in a week. How is Qwen 3 so good at trading??

pretraining nanochat is on going to quickly repro what already exists then then i am going to go full monkey on the repo

rl envs are the new synthetic data pipelines.

What's a good nootropic stack for someone just getting into using their brain?

once you see it you can't unsee it. and it's everywhere

I'm pregnant and looking for a baby boy name that ends with “on” Help me out before my husband suggests Dragon again 🙂

alright a plan that looks solid has emerged for next stage of avataRL repo (RL from zero pretrain). would be coding the first set, create a thinking md file for easy follow up and post in two hours. *fingers crossed*

Introducing Continuous Thought Machines New Blog: sakana.ai/ctm/ Modern AI is powerful, but it’s still distinct from human-like flexible intelligence. We believe neural timing is key. Our Continuous Thought Machine is built from the ground up to use neural dynamics as a powerful representation for intelligence. Thought takes time, and reasoning is a process. Biological brains inspire us with their complex neural activity, where neural timing is critical to intelligence. We’re exploring how to bring that power to AI. The Continuous Thought Machine (CTM) incorporates neuron-level temporal processing and neural synchronization, moving beyond current AI limitations. Our approach has two core innovations: (1) neuron-level temporal processing, where each neuron uses unique parameters to process a history of incoming signals for fine-grained temporal dynamics, and (2) neural synchronization, used as a direct latent representation to modulate data and produce outputs, encoding information directly in the timing of neural activity. Learn more about our approach: Interactive Report: pub.sakana.ai/ctm/ Full Paper: arxiv.org/abs/2505.05522 GitHub : github.com/SakanaAI/continuo…

mathematics for computer science from scratch.

LLMs are injective and invertible. In our new paper, we show that different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space. (1/6)

is it possible to pretrain a language model using pure reinforcement learning from scratch? random weights, no cross-entropy loss pretraining. you may have many questions in your head.

yeehaw mfkers

Tinker is cool. If you're a researcher/developer, tinker dramatically simplifies LLM post-training. You retain 90% of algorithmic creative control (usually related to data, loss function, the algorithm) while tinker handles the hard parts that you usually want to touch much less often (infra, forward/backward of the LLM itself, distributed training), meaning you can do these at well below <<10% of typical complexity involved. Compared to the more common and existing paradigm of "upload your data, we'll post-train your LLM", this is imo a more clever place to "slice up" the complexity of post-training, both delegating the heavy lifting, but also keeping majority of the data/algorithmic creative control. I think the community still has to discover how and when finetuning makes sense compared to the (often strong) baseline of prompting a giant model. The early indications I've seen is that finetuning isn't so much about "stylizing" an LLM, instead, it's a lot more about narrowing the scope, and especially when you have a lot of training examples. An extreme example of scope narrowing being that of categorical classifiers, e.g.spam filters, content filters, etc. but it should be broader than that. Instead of building a giant few-shot prompts for a big LLM, it might work a lot better (and faster!) to finetune a smaller LLM specifically for your narrow task. Increasingly, production applications of LLMs are larger pipelines where a bunch of LLMs collaborate in DAGs and flows. Some of these components might work well as prompts. But a lot of it will probably work a lot better as a finetune. Tinker makes the latter trivial and should allow for an easy experimentation of what works best at any stage.

On the left we have a billion dollar company with restrictive license, On the right we have a open source non profit company with MIT license. They blatantly copied; they didn't even have the courtesy to clone the repo and make changes.

It is wild to me how little Deep Learning researchers know about basic statistical theory. Everyone acts like all to all attention is a free lunch while basic stats has shown many better ways to capture long range dependencies instead of comparing every token to each other. Ironically, I don’t think transformers success is as much to do with attention as it is to do with discretization (tokenization) which HMM used to great effect and makes modeling long range dependencies much more tractable. Dense attention is compute hungry ESPECIALLY under autoregression. Just using basic intuition a model that first zooms out, then drills down mirroring natural dependency hierarchies feels mooore right... Maybe it’s time we actually think first before wasting a quadrillion flops.

Excited to release new repo: nanochat! (it's among the most unhinged I've written). Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI. It weighs ~8,000 lines of imo quite clean code to: - Train the tokenizer using a new Rust implementation - Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics - Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use. - SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval) - RL the model optionally on GSM8K with "GRPO" - Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI. - Write a single markdown report card, summarizing and gamifying the whole thing. Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc. My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved. Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

SimpleVLA-RL (Tsinghua × Shanghai AI Lab under PRIME-RL): first clean port of R1-style RL into VLAs. Recipe: tokenized action head on OpenVLA-OFT plus veRL, outcome-only rewards, mixed-outcome dynamic sampling, Clip-Higher [0.8, 1.28], rollout T=1.6, GRPO without KL ref, parallel multi-environment rendering. Benchmarks - LIBERO avg 91.0 to 99.1; Long 86.5 to 98.5 (beats π₀ 85.2, UniVLA) - RoboTwin-1.0 39.8 to 70.4 - RoboTwin-2.0 (12 tasks, short to extra-long) 38.3 to 68.8 (π₀ 49.2, RDT 33.3) - Low-data: one-traj SFT 48.9 → 96.9; Long 17.3 → 91.7 (+74.4 pts, +430%) - Sim-to-Real (4 tasks) 17.5 → 38.5; above RDT 23.5 Findings - Generalization: RL strengthens spatial, object, and goal OOD while SFT overfits and often catastrophically forgets - Emergence: “Pushcut” where the policy learns push-based shortcuts vs grasp-move-place - Limits: needs non-zero base skill (0-traj SFT to RL fails); benefits scale with stronger priors (100 vs 1000 traj SFT) - Infra: 8×A800-80G, LR 5e-6, batch 64, 8 samples per prompt, action chunks (LIBERO 8, RT 25)

Me thinks I wasn't supposed to see this

Just woke up. Did I miss anything?

magnet:?xt=urn:btih:7278e625de2b1da598b23954c13933047126238a&dn=pixtral-12b-240910&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%https://nitter.app/t.co/2UepcMHjvL%3A1337%2Fannounce&tr=http%3A%2F%https://nitter.app/t.co/NsTRgy7h8S%3A80%2Fannounce

o3 mini got released and nobody's saying anything about its cut off date am i still going to suffer with Dec '23 training cut off? no, i want to use uv, polars and all the other libraries that got released until aug '24 your RL iteration is now 3 months, update datasets.