A recipe for frontier model post-training Apple, Meta, and Nvidia all agree — synthetic data, iterative training, human preference labels, and lots of filtering. interconnects.ai/p/frontier-…

OpenAI's o1 using "search" was a PSYOP How to understand OpenAI's o1 models as really just one wacky, wonderful, long chain of thought. interconnects.ai/p/openais-o…

Synthetic data: Anthropic’s CAI, from fine-tuning to pretraining, OpenAI’s Superalignment, tips, types, and open examples Synthetic data is the accelerator of the next phase of AI — what it is and what it means. interconnects.ai/p/llm-synth…

Reverse engineering OpenAI’s o1 What productionizing test-time compute shows us about the future of AI. Exploration has landed in language model training. interconnects.ai/p/reverse-e…

China's Top 19 Open Model Labs We ranked all the organizations in China releasing open models, from the top of DeepSeek to small, newer academic labs making waves with tech reports and niche models. interconnects.ai/p/chinas-to…

OpenAI's o3: Over-optimization is back and weirder than ever Tools, true rewards, and a new direction for language models. interconnects.ai/p/openais-o…

An unexpected RL Renaissance New talk! Forecasting the Alpaca moment for reasoning models and why the new style of RL training is a far bigger deal than the emergence of RLHF. YouTube: piped.video/watch?v=YXTYbr3h… Slides: docs.google.com/presentation… More info: interconnects.ai/p/an-unexpe…

OpenAI’s Strawberry, LM self-talk, inference scaling laws, and spending more on inference Whether or not scaling works, we should spend more on inference. interconnects.ai/p/openai-st…

Quick recap on the state of reasoning -- can LMs reason? How? My talk at the NeurIPS Latent Space live event (pre o3). Slides: docs.google.com/presentation… Post: interconnects.ai/p/the-state… YouTube: piped.video/2pHE9L4ZZXM?si=vM9-…

This also means you can write off your Interconnects AI subscription. Not official tax advice.

We're all excited about the GPT-5 release. Here's a fun game for while you watch. Potential prizes coming later! Livestream links coming soon.

Futures of the data foundry business model Scale AI’s future versus further scaling of language model performance. How Nvidia may take all the margins from the data market, too. interconnects.ai/p/ai-data-f…

The state of post-training in 2025 A re-record of my NeurIPS tutorial on language modeling (plus some added content). Blog + extra context: interconnects.ai/p/the-state… YouTube: piped.video/6yIMb0K-aS4 Slides: docs.google.com/presentation…

Model merging lessons in The Waifu Research Department When what seems like pure LLM black magic is actually supported by the literature. interconnects.ai/p/model-mer…

In the last ~6 months more closely analyzing the open models and datasets of note across the community on @huggingface, we've highlighted artifacts from 141 different organizations. It takes many people to build the open ecosystem for AI. ACE-Step AI-MO AIDC-AI ASLP-lab Alpha-VLLM AtlaAI BAAI (4) BLIP3o ByteDance (3) ByteDance-Seed (4) CYFRAGOVPL CohereLabs (5) DataoceanAI DatologyAI (2) Datou1111 Etched EuroBERT Freepik (2) GSAI-ML Goedel-LM Hcompany HelloKKMe HiDream-ai HuggingFaceTB (3) ICTNLP JetBrains LGAI-EXAONE LLM360 (2) MiniMaxAI (2) NX-AI NexaAIDev Nexusflow NousResearch NovaSky-AI (2) Open-Reasoner-Zero OpenGVLab OpenPipe OuteAI POLARIS-Project PRIME-RL PeterJinGo PlayHT PleIAs (2) PrimeIntellect (2) Qwen (15) RekaAI Salesforce (2) Skywork (6) Snowflake SparkAudio StarJiaxing SultanR THUDM (3) TIGER-Lab UCSC-VLAA UW-Madison-Lee-Lab Wan-AI (3) WisdomShell XiaomiMiMo (2) Xkev Zyphra (2) agentica-org (2) ai21labs all-hands allenai (5) allura-org amd (3) answerdotai apple arcee-ai (6) arcinstitute bespokelabs canopylabs cl-nagoya convergence-ai deepcogito deepseek-ai (9) ds4sd echo840 facebook (3) fdtn-ai featherless-ai fishaudio genmo google (9) haizelabs hexgrad hkust-nlp hpcai-tech ibm-granite (9) inclusionAI (4) infgrad internlm (5) kuleshov-group kyutai (4) laion (2) lerobot lightonai m-a-p marin-community maya-multimodal meta-llama (2) metagene-ai microsoft (7) mistralai (6) mixedbread-ai mobiuslabsgmbh moonshotai (4) nanonets nllg nomic-ai (3) nvidia (18) open-r1 open-thoughts openbmb (3) opencompass osmosis-ai ostris perplexity-ai qihoo360 rednote-hilab reducto rhymes-ai (3) ruliad sand-ai sarvamai sesame si-community simplescaling stabilityai (2) stepfun-ai (4) tencent (3) thomas-sounack tiiuae (2) tngtech tomg-group-umd vidore (2) vikhyatk xlr8harder yentinglin zed-industries

DBRX: The new best open model and Databricks’ ML strategy Databricks’ new model is surpassing the performance of Mixtral and Llama 2 70B while still being in a size category that's reasonably accessible. interconnects.ai/p/databrick…

Ilya on deep learning in 2015 On vision and how to understand deep learning. interconnects.ai/p/ilya-on-d…

RLHF roundup: Getting good at PPO, sketching RLHF’s impact, RewardBench retrospective, and a reward model competition Things to be aware of if you work on language model fine-tuning. interconnects.ai/p/rlhf-roun…

GPT 5 Launch Party w/ Will Brown & Swyx nitter.app/i/broadcasts/1OyKALjqR…

As Meta tries to race to build a great new research lab, we wanted to remind everyone that the organization structure is just as much of a challenge as the personnel. Here are our takeaways from earlier in the year. Rec's first.

Interviewing Eugene Vinitsky (@EugeneVinitsky) on self-play for self-driving and what else people do with RL #13. Reinforcement learning fundamentals and scaling. interconnects.ai/p/interview…

RL backlog: OpenAI's many RLs, clarifying distillation, and latent reasoning Notes I forgot to publish. Closing some loose ends in the reasoning model discussions. interconnects.ai/p/rl-backlo…

We're working on best-available analyses of where open models come from, who uses them, and how much. What questions do you have?

Character training: Understanding and crafting a language model's personality Post-training in industry is very different than the academic papers and open-source models demonstrate. interconnects.ai/p/character…

Gemini 2.5 Pro and Google's second chance with AI Plus some coverage for the latest DeepSeek. interconnects.ai/p/gemini-25…

The AI agent spectrum Separating different classes of AI agents from a long history of reinforcement learning. interconnects.ai/p/the-ai-ag…

Kimi K2 and when "DeepSeek Moments" become normal One "DeepSeek Moment" wasn't enough for us to wake up, hopefully we don't need a third. interconnects.ai/p/kimi-k2-a…

Grok 4: An o3 look-alike in search, high highs and new lows An o3 class model, the possibility of progress, chatbot beige, and the illusiveness of taste. interconnects.ai/p/grok-4-an…

A taxonomy for next-generation reasoning models Where we've been and where we're going with RLVR. interconnects.ai/p/next-gen-…

OpenAI's o3: The grand finale of AI in 2024 A step change as influential as the release of GPT-4. Reasoning language models are the current and next big thing. interconnects.ai/p/openais-o…

OLMoE and the hidden simplicity in training better foundation models Ai2 released OLMoE, which is probably our “best” model yet relative to its peers, but not much has changed in the process. interconnects.ai/p/olmoe-and…

Where inference-time scaling pushes the market for AI companies Fundamentals emerging downstream from the RL reasoning models. interconnects.ai/p/where-inf…

State-space LLMs: Do we need Attention? Mamba, StripedHyena, Based, research overload, and the exciting future of many LLM architectures all at once. interconnects.ai/p/llms-beyo…

Claude 4 and Anthropic's bet on code Reasons to be optimistic and pessimistic on Anthropic's future. interconnects.ai/p/claude-4-…

Latest open artifacts (#12): Chinese models continue to dominate throughout the summer 🦦 A new flagship Qwen model, Qwen3-235B-A22B-Instruct-2507, and a general rise in ecosystem quality in Artifacts Log 12. interconnects.ai/p/latest-op…

DeepSeek V3 and the actual cost of training frontier AI models The $5M figure for the last training run should not be your basis for how much frontier AI models cost. interconnects.ai/p/deepseek-…

OpenAI's Reinforcement Finetuning and RL for the masses The cherry on Yann LeCun’s cake has finally been realized. interconnects.ai/p/openais-r…

RLHF progress: Scaling DPO to 70B, DPO vs PPO update, Tülu 2, Zephyr-β, meaningful evaluation, data contamination Huge steps forward in confirming that RLHF can really help you on vibes based evaluation, among many other RLHF analyses. interconnects.ai/p/rlhf-prog…

The White House's plan for open models & AI research in the U.S. Thoughts on the new AI Action plan, American DeepSeek, and what comes next. interconnects.ai/p/the-white…

Artifacts Log 3: Synthetic math and Magpie datasets, another 1T param model, and many Mistral models Artifacts ~124 and on for the year. (partial $) interconnects.ai/p/artifacts…

How I Write And therein how I think. And how AI impacts it. interconnects.ai/p/how-i-wri…

Big Tech's LLM evals are just marketing A PSA everyone needs. The importance of a wait and see attitude when it comes to new models, big and small, open and closed. interconnects.ai/p/evals-are…

RLHF lit. review #1 and missing pieces in RLHF: Looking at the difference between two sets -- what rumors say industry leaders are doing with RLHF and what the literature is up to. A new series studying RLHF literature. interconnects.ai/p/rlhf-lit-…

Interviewing OLMo 2 leads: Open secrets of training language models What we have learned and are going to do next. YouTube: piped.video/dS7QI99uJVc Notes + Podcast: interconnects.ai/p/olmo-2-po…

What I'm reading (#2): More on Kimi K2, how to build a bad research center, Pretraining with RL, and sporks of AGI A quiet summer is all you need. interconnects.ai/p/what-im-r…

GPT-4.5: "Not a frontier model"? OpenAI's latest model raises more questions than answers, but no, the AI bubble isn't popping quite yet. interconnects.ai/p/gpt-45-no…

GPT-4o's images and lessons from native input-output multimodality Hints of a natively multi-modal future. interconnects.ai/p/gpt-4os-i…

GPT-4o-mini changed ChatBotArena And how to understand Llama 3.1’s results on the community's favorite benchmark. interconnects.ai/p/gpt-4o-mi…

Frontiers in synthetic data Trends in synthetic data that I'm watching closely in the leading open and closed models. interconnects.ai/p/frontiers…

Sycophancy and the art of the model GPT-4o-simp, LMArena backlash, and people refusing to understand how messy and crucial RLHF is. interconnects.ai/p/sycophanc…

Interviewing Tri Dao and Michael Poli of Together AI on the future of LLM architectures The first Interconnects research interview! We go even further on the promise of state-space models in the emerging LLM market. interconnects.ai/p/interview…

Managing frontier model training organizations (or teams) How do the frontier labs consistently train great models? How can they fail? interconnects.ai/p/how-to-ma…

The rise of reasoning machines And a debate that doesn't warrant repeating. interconnects.ai/p/the-rise-…

If you’re a student and want to read paid posts, contact @natolambert by email or DM. Happy to provide a base 80%+ discount.

Llama 3: Scaling open LLMs to AGI Meta shows that scaling won't be a limit for open LLM players in the near future. interconnects.ai/p/llama-3-a…

The latest open artifacts (#8): The return of ~30B models, side effects of OpenAI's proposed DeepSeek ban, and yet another reasoning roundup Artifacts Log 8. Expect this pace to continue until mid summer. interconnects.ai/p/the-lates…

Interviewing Ross Taylor on LLM reasoning, Llama fine-tuning, Galactica, agents Interconnects interview #5. interconnects.ai/p/interview…

Latest open artifacts (Artifacts Log #10): New DeepSeek R1 0528!, more permissive licenses, everything as a reasoner, and from artifacts to agents interconnects.ai/p/latest-op…

Deep Research, information vs. insight, and the nature of science What AI will accelerate in the scientific process, what it cannot do, and how we can prepare for new manners of scientific investigation. interconnects.ai/p/deep-rese…

Undoing RLHF and the brittleness of safe LLMs Recent papers show most of the arguments about needing "safety" in releases of open LLM weights are nearly dead in the water. Yes, still release the parameters. Read here: interconnects.ai/p/undoing-r…

Building on evaluation quicksand On the state of evaluation for language models. interconnects.ai/p/building-…

Interviewing Tim Dettmers (@Tim_Dettmers) on open-source AI: Agents, scaling, quantization and what's next Interconnects interview #10. Catching up with one of the leaders of open-source AI. interconnects.ai/p/tim-dettm…

Artifacts Log 4: Reflection 70B, o1 on LMSYS, fine-tuning fine-tunes, and speech models The latest open models and datasets. interconnects.ai/p/artifacts…

The end of the “best open LLM” Modeling the compute versus performance tradeoff of many open LLMs. interconnects.ai/p/compute-e…

Latest open artifacts (#13): The abundance era of open models Mostly thanks to Qwen, but now we're spoiled for choice and winds are shifting. interconnects.ai/p/latest-op…

ChatBotArena: The peoples’ LLM evaluation, the future of evaluation, the incentives of evaluation, and gpt2chatbot What the details tell us about the most in-vogue LLM evaluation tool — and the rest of the field. interconnects.ai/p/chatbotar…

OpenAI’s Model (behavior) Spec, RLHF transparency, personalization questions Now we will have some grounding for when weird ChatGPT behaviors are intended or side-effects — shrinking the Overton window of RLHF bugs. interconnects.ai/p/openai-rl…

Multimodal LM roundup: Unified IO 2, inputs and outputs, Gemini, LLaVA-RLHF, and RLHF questions A sampling of recent happenings in the multimodal space. Be sure to expect more this year. interconnects.ai/p/multimoda…

Why I build open language models Reflections after a year at the Allen Institute for AI and on the battlefields of open-source AI. interconnects.ai/p/why-i-bui…

Elicitation, the simplest way to understand post-training An F1 analogy to help understand fast improvements in post-training on top of slow improvements in scaling. buff.ly/XmEa3XN

Grok 3 and an accelerating AI roadmap Where AI is heading, why 2024 felt slow, and shifting priorities of frontier laboratories. interconnects.ai/p/grok-3-an…

How to cultivate a high-signal AI feed Basic tips on how to assess inbound ML content and cultivate your news feed. interconnects.ai/p/making-a-…

People use AI more than you think And businesses too. The most important trend in AI that gets washed away from between the headlines. interconnects.ai/p/people-us…

How the Foundation Model Transparency Index Distorts Transparency, by @natolambert SE Gyges @BlancheMinerva @aviskowron (Cross post with @AiEleuther) interconnects.ai/p/fmti-crit…

What I've been reading (#1) Splitting the links out from the artifacts log models & datasets series. interconnects.ai/p/what-ive-…

What people get wrong about the leading Chinese open models: Adoption and censorship Narrative violations on licenses, adoption, and censorship. interconnects.ai/p/what-peop…

Phi 3 and Arctic: Outlier LMs are hints Models that seem totally out of scope from recent open LLMs give us a sneak peek of where the industry will be in 6 to 18 months. interconnects.ai/p/phi-3-and…

Switched to Claude 3.5 Speculations on the role of RLHF and why I love the model for people who pay attention. interconnects.ai/p/switched-…

Model commoditization and product moats Where moats are tested now that so many people have trained GPT4 class models. Claude 3, Gemini 1.5, Inflection 2.5, and Mistral Large are here to party. interconnects.ai/p/gpt4-comm…

The latest open artifacts (#9): RLHF book draft, where the open reasoning race is going, and unsung heroes of open LM work Artifacts Log 9. interconnects.ai/p/the-lates…

Evaluations: Trust, performance, and price (bonus, announcing RewardBench) Evaluation is not only getting harder with modern LLMs getting more complicated, it’s getting harder because it means something different. interconnects.ai/p/evaluatio…

The DPO debate: Do we need RL for RLHF? Direct vs. RL methods for preferences, more RLHF models, and hard truths in open RLHF work. We have more questions than answers. interconnects.ai/p/the-dpo-d…

We aren’t running out of training data, we are running out of open training data Data licensing deals, scaling, human inputs, and repeating trends in open vs. closed LLMs. interconnects.ai/p/the-data-…

AI for the rest of us Apple Intelligence makes a lot of sense when you get out of the AI bubble. Plus, the cool technical details Apple shared about their language models "thinking different." interconnects.ai/p/apple-int…

OpenAI's GPT-4.1 and separating the API from ChatGPT OpenAI's latest models optimizing on intelligence per dollar. We'll continue to see ChatGPT handled differently than the API business. interconnects.ai/p/openais-g…

Brakes on an intelligence explosion Why I don't think AI 2027 is going to come true. interconnects.ai/p/brakes-on…

The latest open artifacts (#6): Reasoning models, China's lead in open-source, and a growing multimodal space Artifacts Log 6. The open LM ecosystem yet again accelerates. interconnects.ai/p/open-arti…

How scaling changes model behavior Some trends are reasonable to extrapolate, some are not. Even for the trends we are succeeding at extrapolating, it is not clear how that signal translates into different AI behaviors. interconnects.ai/p/how-scali…

Making the U.S. the home for open-source AI Open-source AI is here to stay, but it is not a given that it will be American. interconnects.ai/p/making-th…

Mixtral Round-up: MoE trade-offs, release lessons, Mistral raises $400mil, Google's loss, vibes vs marketing Emergency blog 🚨 We have an amazing open mixture of experts model for the holidays! interconnects.ai/p/mixtral

We don’t need to reinvent everything to solve alignment Integrating some non-computing science into reinforcement learning from human feedback (RLHF) can give us the models we want. Bonus: OLMo 1.7-7B. interconnects.ai/p/reinventi…

It's 2024 and they just want to learn The state of the ML communities big and small starting 2024. My general expectations for the year. interconnects.ai/p/they-want…

Latest open artifacts (#11): Visualizing China's open models market share, Arcee's models, and VLAs for robotics Artifacts Log 11. interconnects.ai/p/latest-op…

Interviewing Sebastian Raschka on the state of open LLMs, Llama 3.1, and AI education Interconnects interviews #4. interconnects.ai/p/interview…

Interviewing Finbarr Timbers on the "We are So Back" Era of Reinforcement Learning Interconnects interview #11. An overview on the past, present, and future of RL. interconnects.ai/p/finbarr-t…

This month's leading open model contributors in Artifacts Log. Thanks for continuing to release your work. Qwen (@Alibaba_Qwen) x5 Zhipu AI (@ZhipuAI) x2 NVIDIA (@nvidia) x2 OpenAI (@OpenAI) InclusionAI (@InclusionAI666) x2 Infinigence (@infinigenceAI) Tesslate (@TesslateAI) Arcee (@arcee_ai) DeepCogito (@DeepCogito) Kakao (@kakaocorpglobal) Skywork (@Skywork_ai) Tencent (@TencentGlobal) x2 InternLM (@intern_lm) StepFun (@StepFun_ai) SK Telecom (@SKtelecom) Xiaomi MiMo (@Xiaomi) OpenBMB (@OpenBMB) Quotient AI (@QuotientAI) Roblox (@Roblox) Knowledgator (@knowledgator) Cisco Foundation AI (@fdtn_ai) ByteDance Seed (@bytedance_talk) JHU CLSP (@jhuclsp) VAGO Solutions (@VAGOsolutions) NuMind (@numind_ai) Neta (@NetaArt_AI) Black Forest Labs (@bfl_ml) Numina (@ProjectNumina) Krea AI (@krea_ai) Hugging Face M4 (@huggingface) moondream (@moondreamai) SpatialVerse (@spatialverse) RedNote HiLab x2 KwaiPilot PowerInfer MetaStoneTec Trillion Labs IBM Granite ScienceOne AI kpsss34 X-Omni Tencent BAC MiSpeech Wan AI The Common Pile

The AI research job market shit show (and my experience) There are plenty of jobs, but finding a place where you're happy is as hard as ever. Read here: interconnects.ai/p/ai-resear…

Transparency and (shifting) priority stacks What you want to be open says a lot about your ranked priorities. interconnects.ai/p/transpare…

SB 1047, AI regulation, and unlikely allies for open models The rallying of the open-source community against CA SB 1047 can represent a turning point for AI regulation. interconnects.ai/p/sb-1047-a…

Specifying objectives in RLHF: the links between the scientific weirdness of RLHF, DPO, and @johnschulman2's ICML talk on proxy objectives. interconnects.ai/p/specifyin…