On a little excursion. Waku Waku! On sabbatical @PrimeIntellect 🇺🇸 Previously @JinaAI_ 🇩🇪 @MoneyLion 🇲🇾
San Francisco, CA
- Tweets 822
- Following 960
- Followers 2,481
- Likes 17,568
Pinned Tweet
excited to share that TOPLOC has been accepted at ICML 2025!
see you in Vancouver 🇨🇦
Today, we release TOPLOC: A Locality Sensitive Hashing Scheme for Verifiable Inference
- Detects modifications to models, prompts, or precision
- Robust across GPU types, tensor parallel configurations and attention kernels
- Up to 100× faster validation than generation
- Reduces memory overhead of proofs by 1000×
primeintellect.ai/blog/toplo…
Building the foundation for decentralized, verifiable compute protocols.
15
17
154
46,521
people probably underestimate the importance of fault tolerance at scale. if a gpu fails once a month, a 100k gpu cluster will have a gpu fail every 26 seconds
Introducing PCCL, the Prime Collective Communications Library — a low-level communication library built for decentralized training over the public internet, with fault tolerance as a core design principle.
In testing, PCCL achieves up to 45 Gbit/s of bandwidth across datacenters in Europe and 25 GBit/s training intercontinental across North America and Europe.
9
7
270
17,221
80% of RL training compute time is spent generating rollouts which is such an embarrassingly parallel task ripe for decentralisation🍎
Today we’re launching INTELLECT-2:
The first decentralized 32B-parameter RL training run open to join for anyone with compute — fully permissionless.
Scaling towards frontier reasoning across coding, math and science.
2
7
166
12,121
Toploc Poster session tomorrow (Wed) at 4:30 PM East Hall E-1106
I’ll be around through Saturday; if you’re into decentralized training & inference, lets chat!
3
5
86
7,771
it seems clear as day to me that decentralized RL post training will become the source of improvements in AI capabilities in the next few years
Releasing INTELLECT-2: We’re open-sourcing the first 32B parameter model trained via globally distributed reinforcement learning:
• Detailed Technical Report
• INTELLECT-2 model checkpoint
primeintellect.ai/blog/intel…
6
3
81
5,623
Remoting into GPU machines isnt that bad as most IDEs have a remote feature which makes the experience pretty much the same as local development.
3
71
17,977
Synthetic data generation is such an embarrassingly parallel task that it seems obvious this is how the big synthetic datasets will be created in the next few years.
To SYNTHETIC-1 and beyond 🫡🚀
Introducing SYNTHETIC-1: Collaboratively generating the largest synthetic dataset of verified reasoning traces for math, coding and science using DeepSeek-R1.
Join us to contribute compute towards state-of-the-art open reasoning models.
4
7
66
5,433
torchtitan has an MoE impl that supports grouped mm and composes with FSDP: github.com/pytorch/torchtita…
needs the latest torch version though (2.8) which flash-attn doesnt have a wheel for yet :(
4
3
63
16,955
I dont know how well IOI gold maps to ability to train the best ML models though. Sure, it may mean these people have a raw baseline problem solving ability. But the IOI medalists I know all went into academia working on fundamental math / CS research that they probably wont be alive to see become useful.
9
4
55
25,084
In order to support dynamically on and off-boarding compute to the run, we introduced a new distributed abstraction `ElasticDeviceMesh` which manages the resizing of process groups without requiring a cold restart.
More info on this, along with some other interesting innovations we did to pull this off in our blog post:
primeintellect.ai/blog/intel…
Announcing INTELLECT-1: the first-ever decentralized training of a 10B model
Scaling decentralized training 10x beyond prior efforts.
Anyone can join us to build open-source AGI 🦋
4
9
58
11,860
One of the key capabilities required for a decentralized compute system is the ability to verify that computations were performed correctly by untrusted parties. Happy to be releasing our solution to the problem. Big thanks to my colleagues who helped out with the work @matthewdif @apaz_cli @imkharn @samsja19 @manveerxyz @johannes_hage 💕
Today, we release TOPLOC: A Locality Sensitive Hashing Scheme for Verifiable Inference
- Detects modifications to models, prompts, or precision
- Robust across GPU types, tensor parallel configurations and attention kernels
- Up to 100× faster validation than generation
- Reduces memory overhead of proofs by 1000×
primeintellect.ai/blog/toplo…
Building the foundation for decentralized, verifiable compute protocols.
2
5
49
5,185
sometimes i forget that i made my initial calculation assuming the cows are spherical and get surprised in prod when the cows stop being nice to work with
2
2
38
5,991
havent slept in days 😿 can we hire some filipinos next time?
TOPLOC verifiable inference isn't real btw, it’s just @jackminong looking at all the reasoning chains and brutally slashing misbehaving computers
2
39
2,933
Just merged hidden state support into sglang. You can initialize the engine with `return_hidden_states=True` to get the last hidden states in the output—making toploc work out of the box.
```
llm = sgl.Engine(model_path="meta-llama/Meta-Llama-3.1-8B-Instruct", return_hidden_states=True)
```
🚀 Big update from the sglang team! We've made significant progress on the highly anticipated deepseek model:
🚀 FlashInfer MLA Attention integration for 4x faster long-context performance
⚡ torch.compile support, hitting 50 tokens/s for online inference
🔥 CUTLASS block-wise FP8 for improved efficiency
Too many features to list—our team is working too hard to keep up with the announcements 😂 Check out the full release report here:
🔗 github.com/sgl-project/sglan…
Huge thanks to the SGLang team and our amazing community! 🎉
1
1
35
3,301
globally distributed synthetic data gen on underutilised compute is the launchpad to the era of experience
Launching SYNTHETIC-2: our next-gen open reasoning dataset and planetary-scale synthetic data generation run.
Powered by our P2P inference stack and DeepSeek-R1-0528, it verifies traces for the hardest RL tasks.
Contribute towards AGI via open, permissionless compute.
1
3
35
5,521
Here's a Reflection: Most inference APIs today are unverifiable.
Quantized, pruned, fine-tuned, or outright swapped.
Verifiable inference isn't a luxury, it's the only way to trust the output.
Good writeup on the current state of approaches by @hammyx_
1/ Verifiable inference enables proving the correct model was used and that inputs/outputs were not tampered with.
Why is this useful, who's working on it, and what does the future hold? Summary below 🧵
✍️ Written by @hammyx_
equilibrium.co/writing/state…
3
6
33
4,174
I feel like you used to have a pikachu profile pic before this one. Your hot takes were the reason i joined twitter
3
29
12,856
u seem like a jax guy but just in case u are using torch, the parallel thought attention can be implemented efficiently in 2 parts using flex attention. causal mask for the first non thought chunk and einsum the rest. You can then combine the two parts with the logsumexp returned from flex attention by setting `return_lse`.
You'll also make @cHHillee happy.
1
28
1,569
the 9 extraordinary aliens at @PrimeIntellect are grateful 🫡
you know it's a good event when the information quality *and* attendee quality are both very dense. huge thank you to @swyx @aiDotEngineer for bringing it all together!
plus, loved seeing our @lighthousehq_ customers everywhere
1
29
3,401
Bullish on scaling inference time compute 🚀🧠⏲️
Today, we are releasing:
- INTELLECT-MATH, a frontier 7B parameter model for math reasoning
- The largest synthetic math dataset to date of 5M verified reasoning traces
- An outlook on decentralized training in the inference-compute paradigm
primeintellect.ai/blog/intel…
1
26
1,508
we are always in the best timeline
update: i joined @primeintellect :)
cannot describe how excited i am to be joining such an incredible team and mission.
there is a dire shortage of labs who are truly embracing open-source research. it’s hard to get the incentives right. you need a business model where open-sourcing your work is positive-sum; being a GPU marketplace is a really good one.
prime intellect has been doing incredible work to advance the frontiers of decentralized training and inference, and it is only the beginning.
my own goal is to continue along the directions of my recent projects and musings, but bigger, bolder, more real: advancing open research and infrastructure for agentic RL.
towards open-source AGI 🚀
1
26
1,681
Nice work! Happy to have an early adopter.
Its very much early days and we expect to rewrite a lot of it with learnings obtained from the Intellect-1 run. We thus mainly optimised for building out features and shipping quickly so the UX may not be so great for now. Feel free to raise issues about parts you think we can improve. :)
I cracked it! @PrimeIntellect @samsja19 Will do a writeup and add to the documentation ASAP! Training with 4x3090 Ti's in SF and 2xA6000's in Kansas!
4
4
21
4,059
The torus ring-reduce idea seems nice theoretically but its not exactly clear to me how
1. You can generate the cycles. Take the blue path 0,5,3,4,1,2 (OEIS does not return a result)
2. You can generalize this to cases where the two sides have different number of nodes
2
1
22
3,674
Yea, being smart with all-reduce planning is probably a good idea. The issue is that internet bandwidths are dynamic and the best reduce plan can look kind of weird.
One easy win is to perhaps allow some form of torus all-reduce. A pattern we observed is that a subset of nodes will be strongly connected to each other but have a weak ingress connection from the rest of the nodes. If we ring through them, we will be bottlenecked by the highest bw ingress. However, if we do a max flow min cut through the graph, we could shard the result among the nodes and use every weak ingress connection, multiplying bw by the number of nodes in the subset and getting significantly lower reduce times for this regime.
1
2
22
4,612
big thanks to my co-authors @matthewdif @apaz_cli @imkharn @samsja19 @manveerxyz @m_ryabinin @johannes_hage
1
21
595
Metagenomics just got a power-up! 🚀 Can't wait to see what people build on this foundation.
Releasing METAGENE-1: In collaboration with researchers from USC, we're open-sourcing a state-of-the-art 7B parameter Metagenomic Foundation Model.
Enabling planetary-scale pathogen detection and reducing the risk of pandemics in the age of exponential biology.
1
2
20
3,051
@_sholtodouglas was on TBPN yesterday claiming that it would be easily solved if ppl just RLed on it but they dont so that it can be kept as a good holdout piped.video/watch?v=FKOwIN9z…
3
18
11,201
Cool work! Congratz to the @gensynai team on the release. Eliminating algebraic rewrites from the async nature of gpu scheduling with sync ops will be a useful tool in the verifiability toolbelt.🧵⬇️
Introducing Verde
Verde is a high-scale verification system for machine learning over untrusted nodes. It ensures correct execution of ML operations across any device in the world, from data centres to the edge. It operates at runtime and doesn't require any onboarding.
1
18
1,347
I made a flash attn fork that has torch 2.8 wheels if u wanna try early and trust that i wont supply chain pwn u
pypi.org/project/pi-flash-at…
2
16
878
come by if you're in SF!
we're co-hosting the second session of the SF Systems Reading Group with @_shreya_s to dive into our TOPLOC verifiable inference scheme
@jackminong will also share updates on our upcoming TOPLOC v2 work
come by!
lu.ma/mgbap0ro
1
15
993
Identifying where LLMs are "too smart" compared to humans is an interesting way to catch them. 💡
Congratz on the win!
1/ China town mafia - 🥈2nd place
Next gen anti-captcha that detects when AI agents powered by Claude’s computer use try to solve verification puzzles
Honeypot for AI agents
@_weichunnn @benxneo
1
14
497
Happy to have hacked on @LitProtocol and developed a core primitive to the future of decentralized inference!
Thread about my project with @benxneo @_weichunnn on llm inference proofs coming soon. (after i get some sleep)
Over the last 36 hours in SF @mikelxc @jackminong @benxneo @_weichunnn @ShivaanshKapoor @yupuday @gajesh @theozero @ChinesePowered @tms7331 @0xshubhank @shreya_gupta30 build practical and secure agentic protocols that link @ethereum to @OpenAI @AnthropicAI @X @YouTube and OSS LLMs👏
These prize winning products create new capacity and experiences for developers and users 🔥
Thanks for hosting @ETHGlobal 🙌
3
4
13
3,047
TIL that nn.Linear on cpu and gpu can sometimes be wildly different in bf16. Guess its due to scheduling order?
colab.research.google.com/dr…
3
1
13
855
i once dreamt i met @tenderizzation irl for the first time at the prime office and discovered he’s half asian.
i then woke up and was like: wait, he’s full asian no?
1
12
408
The fat tree network requirement for all-reduce has always seemed somewhat wasteful to me. You can often get twice the amount of compute if you didnt need the networking gear. Happy to have contributed to a direction where the trees might get slimmer!
Introducing OpenDiLoCo, an open-source implementation and scaling of DeepMind’s Distributed Low-Communication (DiLoCo) method, enabling globally distributed AI model training.
primeintellect.ai/blog/opend…
1
10
398
only healthmaxxis who do not eat so close to bedtime in this office. in the fridge it goes!
1
11
499
there might be some other way to do it for closed models. e.g. have them host a validation endpoint on TEE enabled GPUs with a guarantee that they cannot change the code without being noticed. they could try to fingerprint the output instead of doing toploc though, but might have some way to do it by exposing the polynomial for every inference since u cant deduce the tensor from it
2
11
364
Everyone has their piece of the puzzle. Yet I keep looking over at other peoples pieces and thinking they’re cooler than mine.
1
10
384
Just noticed that RX > TX in nvtop when training models with FSDP. I guess its because NVSwitch can multicast.
It looks like too big a difference though. The ratio is ~7.22 which implies I am doing across all 8 gpus but I actually did 4 independent training runs with 2 gpus each.
10
451
🐐
Took some time to reflect on this, big impostor syndrome for me to even be on this list lol, but I guess that couple of year in doing open source research is enough to do a name for yourself
Anyway, If you are an ambitious researcher or engineer thinking about working for a big labs, you should rather consider joining @PrimeIntellect to take ownership in open source AGI and accelerate your career
9
686
Great work from the team @NousResearch on communication efficient optimizers. And its open source!
paper arxiv.org/abs/2411.19870
code github.com/bloc97/DeMo
1
9
556
This is obviously a hype piece to get more investment. Any serious programmer who has tried copilot / GPT4 knows that they are not an existential risk to programming. The nature of programming will change, but we will still need humans in the next decade.
1
8
1,633
it's easier to make noise with an empty can than a full one haha.
the latter has more impact when thrown though
1
8
3,427
They are still working on releasing details so its difficult to compare. But my understanding is that they are focused on reducing communication requirements by reducing payload size at every step while we are focused on reducing communication overhead by reducing payload frequency
1
8
163
oh and it supports deepseeks loss-free expert load balacing which might solve ur imbalance issue @jxmnop. you need to register a pre-hook on the optimizer for it to update the expert biases for it to work: github.com/pytorch/torchtita…
8
724
Thanks!
Check out the code: github.com/PrimeIntellect-ai…
2
8
383
i suspect its not rlly an issue with vllm or huggingface but may be because of different grid params being used by decode vs prefill which change the numerics. some of the noise might be from vllms particular implementation though (in the vllm discussion u link i found that you get slightly better logprobs by using torch log_softmax
8
1,145
Thanks @RobertTLange! 🦋
Super bullish on @PrimeIntellect 🎉 Got to co-work with the team for a second and they are simply incredible. Huge congrats @vincentweisser @samsja19 @johannes_hage @jackminong & everyone involved 🤗
Check out their blog on the newest OpenDiloco-style 10B decentralized training run -- so many gold nuggets:
primeintellect.ai/blog/intel…
The future of accessible distributed compute and open development is so exciting 🚀
7
229
I wonder if it makes sense to have a longer format next time. Some of the ideas seemed like they needed more time to see some promise. e.g. the torch -> cu transpilation by @cloneofsimo @fp8e4m3
2
7
206
Existing methods for performing this verification for model inference have been far from practical. ZK methods require 100-1000x the cost of the original inference, other methods require deterministic inference which is easily broken by GPU asynchronous kernel scheduling or optimizations.
1
7
148
Especially diffusion papers which usually contain a lot of images and take a billion years to load. Always end up opening a tab with the abs link so I can read the abstract and stalk the authors while the paper loads.
6
1,321
However, relaxing this constraint would unlock speedups in both the execution and validation. One can still verify that the computation was performed correctly by having comparable hashes of the intermediate computation instead of cryptographic hashes.
2
7
201
Was really fun hacking at CudaModeIRL. Learnt a lot about tracing from Jane Xu @PyTorch and about to land my first PR in torchtune which enables activation offloading for multi-device training!
While we were working on this feature, we also discovered that the sdpa kernel adds an an unusual 1s cpu overhead to every step. 😬
7
278
the amount of times my run has been saved by nvtop exit being q instead of ctrl+c makes me believe in god
1
6
360
I am always disappointed when I flip through Python books at the bookstore and find that they dont use or teach type hinting. Code is read more than it is written and having type hints easily makes your code > 2x faster to read.
2
6
343
we've now upgraded to triangle. much more surface area than the previous ellipse. llms are making progress!
5
91
i think its a combination of a few things I think are true
1. code you didnt write yourself is always ugly
2. it is possible to refactor things and feel very productive but actually make no progress on what matters
3
6
173