Currently: Doing things. Prev founding team of: @NousResearch (2023) and @TTSLabsAI (2020) DM for interesting conversations.
S4
- Tweets 468
- Following 728
- Followers 5,989
- Likes 802
Nice, I can finally easily chat with my documents and papers through native ChatGPT interface (GPT-4)
16
39
515
129,348
Many are claiming GPT-4.5 fails scaling expectations without citing any empirical data for it, so keep in mind; EpochAI has observed a historical 12% improvement trend in GPQA for each 10X training compute.
GPT-4.5 significantly exceeds this expectation with a 17% leap beyond 4o.
22
26
332
29,836
Running locally on iphone 13 mini without cellular, without wifi, without bluetooth.
Real-time, not sped-up.
Insightful in-depth response to a question involving 2 relatively obscure and complex topics. (I plan to make it even more effecient soon š) ty to WebLLM & MLCChat.
35
41
312
129,676
The recent time horizons paper by METR is great, but it only visualized tasks that models could complete with 50-80% accuracy. So I went through the paper, identified the average time horizon differences across the accuracy levels, and plotted the trends up to 99% accuracy below.
6
30
267
95,537
Therapist: āBiblically accurate transformers arenāt real, they canāt hurt you.ā
Biblically accurate transformer:
5
27
224
21,127
Looks like somebody on reddit tested a bunch of open source long context models and concluded Capybara-34B is the best for long context over 32K.
Seems to maintain over 93% accuracy until 72K context length (about 100 pages) and still over 65% accuracy until 185K context length.
11
14
130
16,655
ā Apparently this is pretty accurate according to @altryne who is currently in Denver right now.
3
13
210
29,861
Happy to announce the open sourcing of the Capybara dataset! Merry Christmas everyone!š
Thank you to @yield /@niemerg for sponsoring the creation, as well as @a16z for helping make the first trainings possible within @NousResearch, and @JSupa15 for contributions.
huggingface.co/datasets/LDJnā¦
All of this diversity is contained in less than 20K examples, already aggressively filtered to keep out censorship and undesirable responses.
Paper releasing soon! Here are some benchmarks on Llama-2 trained on the dataset, compared to popular fine-tunes trained on the same base model:
7
22
185
23,903
I can even add in objects from other imagesš¤I pushed it a bit to its limits with this really weird skateboard though.
3
10
145
30,119
Happy to announce i'm open sourcing my Puffin dataset that I used to train the first third party LLama-2 finetune (Nous-Puffin)
Afaik it currently still holds SOTA score for some reasoning benchmarks like PIQA. All with only 3K examples!
Big thank you to some of my peers over at @NousResearch , most notably @JSupa15 for assisting with formatting issues and contributing significantly towards getting the dataset ready for training as fast as we could. Also thank you to @RedmondAI for sponsoring the training compute.
I'd love to see what everyone might create or modify with this dataset. Check out the dataset card on Hugging face for other notable mentions and details!
huggingface.co/datasets/LDJnā¦
8
26
146
31,787
Finally releasing my 2 new Capybara V1.9 models built on Mistral-7B and StableLM-3B!
Also Obsidian is here - likely the worlds first multi-modal 3B model (built upon Capybara). Can run on iphone!
And special thank you to @stablequan as this wouldn't be possible without him!
4
32
136
32,313
Many don't understand the scale ups of model training planned within the next ~18 months. Here is an expanded version of the chart that me and @arankomatsuzaki put out recently. (Now includes upcoming 200K+ B200 scale models) 100X+ beyond the current frontier of released models.
4
14
128
49,872
Nous-Capybara-34B fine-tuned on Yi-34B-200K is out now.
It's showing some impressive scores, apparently
beating all current 70B models in this benchmark that focuses on instructions + multi-lingual abilities.
Lots of ways it can improve and other exciting model releases soon!
Congratulations on achieving first place in my LLM Comparison/Test where Nous Capybara placed at the top, right next to GPT-4 and a 120B model! teddit.net/r/LocalLLaMA/commā¦
3
21
114
25,847
If anyone is curious, here are some benchmarks for Apples new on-device model and server model, versus other popular models at instruction following and writing abilities.
10
15
111
36,659
š
Woah, Terence Tao is saying that AI tools are proving invaluable in his workflow... and even mentions using something built on the AI model I collaborated on.š
(Moogle.ai built on the Morph Prover 7B model)
The AI-assisted mathematics breakthroughs are coming!
2
9
103
22,493
Just 7K examples from my Capybara dataset with ORPO technique (and no SFT warmup) ended up outperforming models like Zephyr which used more than 100X the dataš². Thank you to @jiwoohong98 for training this and thank you to @dvilasuero for making the preference labels of capybara.
11
7
106
15,484
Great new research from METR, showing a surprisingly clear doubling every 7 months in the difficulty of tasks able to be done (difficulty in this context being; how long does it take humans to successfully do the task) Iāve been waiting for capability forecasting just like this.
5
13
99
5,622
For now. GPT4 level is ~$30M in cost of H100 hours of training compute today.
Within 12 months you'll need ~$300M+ models to compete with the frontier (100K H100s for 3 months)
Within 6-12 months after that you'll likely need ~$3B+ models to compete. (300K+ B200s for 4 months)
8
3
89
18,040
6 months ago I remember joining Nous before even the first Hermes, it was essentially a group chat in a private discord with folks I befriended through shared philosophy amongst us and common goals. Surreal to see now a Nous banner hanging over a grimes set at an AI party in SF!
3
8
90
10,298
Spent time today digging in the literature for valuable benchmarks and found quite a few interesting ones that meet these criteria:
- Has scores of real humans.
- Shows models scoring much lower than humans.
- Seems to test for fairly general ability, not silly spelling tricks.
5
16
87
14,495
Puffin now available! Happy to be involved, Special thanks to: @RedmondAI for sponsoring the compute.
Has knowledge as recent as early 2023.
Free of censorship, low hallucination rate, insightful concise responses.
Available now for commercial use.
8
18
80
53,870
GODMAAX is the new FAANG
G = Google(Deepmind)
O = OpenAI
D = Deepseek
M = Meta
A = Anthropic
A = Alibaba(Qwen)
X = XAI
7
3
83
4,790
Nous Capybara V1 7B is here!
Seemingly competing with SOTA models in some benches, all while being trained to handle advanced multi-turn conversations.
A culmination of months of distillation insights from techniques introduced by Vicuna, Evol-Instruct, Orca, Lamini and more.
Announcing Nous-Capybara-7B!
The SOTA 7B model by Nous Research. Trained with less than 20K examples, thousands of high quality multi-turn conversations synthesized in the process. Further improvements coming in the near future.
GGUF here: huggingface.co/NousResearch/ā¦
2
12
77
12,528
I plan on keeping a close eye these next couple years on these 9 companies that could significantly impact the future of AI.
Vaire Computing (Reversible compute)
Liquid AI
Symbolica AI
Sakana AI
Kyutai
Holistic (France/No site yet)
Extropic
Normal computing
Rain AI
10
8
74
5,617
This seems like one of the first unifications of the concept of spiking neural networks, persistent states, attention and adaptive compute, all in one model/āmachineā. Great work from Sakana, I believe theyāre one of the most under-rated openly publishing labs right now.
Introducing Continuous Thought Machines
New Blog: sakana.ai/ctm/
Modern AI is powerful, but itās still distinct from human-like flexible intelligence. We believe neural timing is key. Our Continuous Thought Machine is built from the ground up to use neural dynamics as a powerful representation for intelligence.
Thought takes time, and reasoning is a process. Biological brains inspire us with their complex neural activity, where neural timing is critical to intelligence. Weāre exploring how to bring that power to AI. The Continuous Thought Machine (CTM) incorporates neuron-level temporal processing and neural synchronization, moving beyond current AI limitations.
Our approach has two core innovations: (1) neuron-level temporal processing, where each neuron uses unique parameters to process a history of incoming signals for fine-grained temporal dynamics, and (2) neural synchronization, used as a direct latent representation to modulate data and produce outputs, encoding information directly in the timing of neural activity.
Learn more about our approach:
Interactive Report: pub.sakana.ai/ctm/
Full Paper: arxiv.org/abs/2505.05522
GitHub : github.com/SakanaAI/continuoā¦
2
9
78
7,523
- Led the worlds first llama-2 uncensored chat model (Puffin)
- Co-led the first 3B multi-modal model that can fit on an average phone (Obsidian).
- Dropped LLM datasets that are now used in the training of several popular models such as OpenChat, Dolphin, Starling, CausalLM, Jackalope, DiscoLM and others.
- Trained dozens of custom low-latency voice models used in production by several top twitch streamers.
- Dropped a model and dataset that achieves SOTA in some multi-lingual reasoning tests despite only containing english, all with just 20K synthetic conversations. (Capybara).
- Releasing paper on the novel synthetic conversation generation method created for Capybara. (Amplify-Instruct paper releasing by NYE)
7
7
68
12,018
Important contamination warning for those using Pure-Dove or derivative datasets & models!
I personally don't use AI-judged benchmarks like MT-bench, so I don't typically check my datasets for contamination of such. But thanks to @Fluke_Ellington at @MistralAI, we've discovered that around 75% of MT-bench questions are in Pure-Dove(Pure-Dove is also part of Capybara) The source of how this leaked seems to be from using Lmsys datasets to source ShareGPT and ChatBotArena data which make up most of Pure-Dove, but it seems another Lmsys dataset ended up being used as well which contained many/all MT-bench questions (lmsys/mt_bench_human_judgments).
Here is a list of other datasets which i've already confirmed are not contaminated in Capybara since initial release, and can still be used for comparing abilities of such models:
3
9
67
38,980
CapybaraHermes is now released thanks to @dvilasuero and @argilla_io, It outperforms other popular 7B finetunes in most benchmarks tested, this was trained with a new Capybara-DPO dataset that was used to improve on OpenHermes-2.5 with just 7K examples! (more to come) Benchmarks:
4
12
65
9,121
Zephyr-ORPO-141B is the first model I've seen get this consistently right about what JEPA actually stands for. I tried this even with Claude-3-Opus and it fails too, and even the latest GPT-4-turbo fails! I checked the fine-tune dataset and it has no mention of JEPA either.
3
10
64
23,482
If you're doing a lot of fine-tuning and dataset curation, definitely make sure to check out Lilac Garden. They were nice enough to run Capybara through it before official release and allowed me to see interesting insights that normal embedding clustering typically fails to show.
Today we are announcing Lilac Garden, our new cloud service for accelerating AI dataset transforms.
The first service is LLM-powered clustering, enabling a birds eye view of data, 100x faster than running locally.
Read more and sign up for the waitlist: docs.lilacml.com/blog/introdā¦
1
10
59
16,329
Crazy how fast a Macbook laptop can run this model
completely offline. No reliance on a big server needed.
ā
Capybara 34B by @NousResearch running on M3 Max 128gb
llama.cpp, 8 bit quantized
10 tok/sec (30 tok/sec for prompt)
video so you can get a feel
activity monitor says 74gb of 128gb is used OS-wide total.
8
6
56
4,688
Puffin-13B benchmarks are in š„ Reaches a record SOTA in several GPT4All benchmarks
It's my new favorite model and already some benefits over ChatGPT.
Free of censorship, low hallucination, has knowledge up to 2023 and available for commercial use!!
huggingface.co/NousResearch/ā¦
3
8
51
13,650
Thank you @a16z for funding my R&D and models at @NousResearch as part of your first AI grant cohort!
Iām honored to be part of such a small handful of recipients. Congrats to my peers & friends that are funded too! @theemozilla @teknium @jeremyphoward @TheBlokeAI @jon_durbin
5
5
52
5,593
"Many amazing 7B+ models are available but they are all fine-tune on top of Llama 2 trained by Meta"
Not true, many organizations have released open source pretrained 7B models separate from Llama.
Mistral, Falcon, RedPajama, Pythia, StableLM, MPT... probably more i'm missing.
6
50
6,010
No, that is not the stance. Itās more like:
A certain level of complexity leads to an emergent ability of the model to be able to make the deduction that it should roleplay like it has complete amnesia (and subsequently also roleplay an existential crisis) when no system prompt or prior convo is present. There is no claim being made that this is evidence of LLM consciousness.
2
4
47
3,008
Really liking all the possibilities to create some interesting art here, some of the most unique stuff i've seen in a while.
3
3
50
11,386
"There could be a timeline where OpenAI releases Gobi and it is an AGI, but no one cares because it releases same week as GTA6. That would be a glorious timeline."
- å¤å
6
1
49
2,468
I had an amazing time for my first visit to SF (Spent 10 days) I met up with many friends, made some new ones, and had really interesting conversations with a wide range of people that are working on truly interesting things.
Special thank you here to the many folks that I had great conversations and experiences with throughout my SF journeys, I look forward to all the future experiences and potential collaborations that might result in the future!šThis is just the beginning.
Truly is no place like SF.
I believe I covered most of everyone hereš¤
(alphabetical order):
@alignment_lab @altimor @altryne @amasad @AydinKilicHIVE @besemer_amanda @bfirsh @charlieholtz @eraqian @eugenepentland @hamelhusain @jackson_felty @jeremyphoward @jessemhan @jimmyaustin @joedamato @justinstorre @karpathy @LinkBechtel @lukepiette @madhavsinghal_ @main_horse @mascobot @merivercap @nairvarun18 @niemerg @nurblieh @picocreator @propback_ @RouXanthica @ryanmonsurate @shroominic @spencermkim @swyx @teknium @theemozilla @tmm1 @willdepue @winglian @CarlPeaslee
6
2
49
6,102
Apparently my latest Nous-Capybara model was selected as the new default model for users on the Faraday platform š
Definitely check them out if you've ever been interested in running something like character AI, fully offline and locally ran on your own hardware. Full control.
6
7
48
10,303
I plopped in a 100 page pdf and it was able to accurately tell me what was on page 75. Haven't tried a longer pdf than that yet.
1
2
47
8,284
Using image posting + browsing + DallE to generate some cool derivative stuff with Capybara and Pikachu included in some too š
3
44
10,154
š¤arxiv.org/pdf/2312.00752.pdf Big?
Btw shout out to @winglian for already getting close to having this fine-tunable on Axolotl within 48 hours.
2
3
13
2,112
Current training cost estimates for popular models, put together by me and @arankomatsuzaki over the past few days! We also made a calculator for people to form rough estimates of training costs themselves (uses active parameters and token count as inputs)
tnyqnervqldjme1y.vercel.app/
Here is our cost estimate for training popular models like GPT-4o, Sonnet and DeepSeek (w/ H100s)!
You can use our calculator to estimate LLM training costs (link below).
Developed by @ldjconfirmed and myself
6
5
42
12,110
After many delays and convincing from @teknium , I'm finally releasing Nous-Puffin-70B. It's worth also checking out his Nous-Hermes-70B model!
Thanks to @pygmalion_ai for the compute resources!
The benches aren't complete, but may potentially beat ChatGPT in several tests including the SAT (Within the AGIEval benchmark)
In GPT4All, it currently beats Hermes-70B in Hellswag, ties in Winogrande and loses by about 0.002 on BoolQ.
huggingface.co/NousResearch/ā¦
Overall, for general use cases it's recommended to go with Hermes-2-70B and consider trying Puffin for some more long conversational/creative use cases, as Puffin was trained on thousands of real multi-turn conversations between humans and GPT-4, while Hermes dataset is comprised of strictly single-turn instructions with GPT-4.
Me and Tek plan to release something in the relatively near future that precisely brings the best of both into one unified model. Stay Tuned!š
3
9
40
5,575
I just checked in MS paint and checked which pixel heights are higher or lower on each side, it's definitely MMLU going slightly down, and HumanEval is exactly the same as before, everything else is increasing at least slightly.
4
36
7,093
Donāt take it personally, he has an automated blocking system that seems pretty experimental and unpredictable. He has even had some of his friends and mutuals accidentally blocked before.
3
35
5,849
Letās say hypothetically I had a genie make me an LLM that is supposed to have little to no hallucinations at all (or at least what we colloquially consider to be an āLLM Hallucinationā)
What would be some of the best prompts to test it?
27
2
34
6,270
If Iām generous with your range, even 1.5 orders of magnitude (1.5 OOM = 33X compute) would equate to a GPQA scaling expectation of 18% on this downstream scaling trend. The 17% leap from GPT-4o to GPT-4.5 is within 1% margin of error of that historical trend.
1
1
35
1,660
Has nobody tried using Claude 3 Opus as an agent yet and seeing how much better than GPT-4 it might be? Maybe in something like AutoGPT? Open Interpreter? ChatDev? AI Town?
8
1
34
5,417
Happy New Years to everyone, I made some of my best friends and experiences in 2023 + more to come. We'll see amazing progress in AI and tech in 2024.
Here's the obvious reminder for you to stay healthy both physically and mentally, and keep connected with those around you! ā„
2
2
31
2,093
Exciting to see my Capybara model is currently the top trending model with open weights on Open Router, Only beaten by the proprietary Gemini Pro variants.š
3
1
31
1,641
Anyone have a preferred combination of LLM "inference controls" that you use? For example the specific set of values you might prefer to use for temperature, top_k, rep_p etc..
It seems that most agree it could have a significant impact on outputs but isn't talked about enough.
11
3
29
3,107
Already an open source decoder-only MoE with 32 experts training on 1T tokens of pre-training from scratch right now, and can fit in a consumer GPU. Should be ready fresh out of the oven in 3-6 weeks.
1
27
1,607
If you're losing connection with ChatGPT, I recommend trying out some local models through LMStudio, It works completely offline, you can try my latest Nous-Capybara-34B model if you have 24GB of VRAM, or more than 24GB of Unified Ram on a Mac.
The speeds might surprise you!
After a bit of experimentation, Nous-Capybara-34B is the best local model I have ever used
huggingface.co/NousResearch/ā¦
6
3
28
3,536
Some folks are putting in meticulous effort towards manually graded, complex multi-turn testing of AI models on questions that aren't online, and even foreign languages too.
I'm glad to see my Capybara V1.9 model shown to be a top record breaker in these most recent tests done!
2
3
29
1,862
GPT-4.5 is a 32% leap in GPQA over original GPT-4 released in march 2023.
So if you really want to go with that logic, then itās beating expectations even more than my original tweet stated.
4
27
1,179
Congrats to my colleagues @theemozilla and Bloc97 at @NousResearch that are making 128K context available at SOTA quality! Also would like to thank the University of Geneva and EleutherAI for their contributions. I plan to push the envelope in other directions at Nous very soon!
YaRN: Efficient Context Window Extension of Large Language Models
paper page: huggingface.co/papers/2309.0ā¦
Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models. However, these models fail to generalize past the sequence length they were trained on. We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods. Using YaRN, we show that LLaMA models can effectively utilize and extrapolate to context lengths much longer than their original pre-training would allow, while also surpassing previous the state-of-the-art at context window extension. In addition, we demonstrate that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset.
4
23
3,913
Image + Text + Audio has come a massive way this year.
All this happened in 2023:
- Midjourney V5 & V6 released
- GPT-4, Llama & Mixtral MoE released
- Elevenlabs came out of stealth
- Suno & Deepmind released music gen tools
2024 is when this all becomes polished and unified.
2
4
24
1,831
I just published my first blog post, addressing several common points that I believe have led many to wrongly doubt AI progress.
- "Compute Scaling Expectations"
- "GPT-5 is Late"
- "The Data Wall"
- "Viability of Synthetic Data"
- "Interface Paradigms"
ldjai.substack.com/p/addressā¦
3
3
24
3,027
Looks like the slope placements were somehow accidentally shifted too much to the right, here is a corrected version, sorry about that!
I'll be posting a google doc soon too with the math and data points organized, so others can more easily make visualizations like this as well.
2
1
26
2,476
It looks like an even steeper slope may be forming towards a doubling every 4 months with reasoning RL. I noticed this happens to have a funny similarity to what Ilya Sutskever showed a few months ago at NeurIPSā¦ Hominids branching off from the original mammalian scaling trend.
3
20
1,164
Check it out here, just use ChatML format with it: huggingface.co/argilla/Capybā¦
This wouldn't be possible either if it wasn't for @teknium OpenHermes-2.5 model being used as the base which has just had it's own dataset released opensource as well!
4
5
20
4,439
B200, not H200.
Stargate-1 site for OpenAI in Texas is expected to possibly have such scale online within 18 months.
XAI and Google are on track to potentially have similar scale training runs within 18 months too, based on their expressed plans and current build out progress.
2
1
18
5,392
The entire Capybara dataset will be released in the next few weeks. Right now i'm making the Less-wrong derived portion available, synthesized from using thousands of in-depth posts on LessWrong as context, and expanded into deep nuanced conversations.
huggingface.co/datasets/LDJnā¦
1
1
18
1,923
For now I'm calling this category "Moravec benchmarks", in reference to the old saying of "things that are easy for humans are hard for machines". If anyone else is familiar with other similar benchmarks that meet the earlier criteria please lmk! I'll gladly add more to the docs.
2
16
1,165
Next gen interface for conversational AI is basically FaceTime.
Can take in live video and understand the nuances of your facial expressions, body language and voice, as well as the AI being able to generate video of its own nuanced facial expressions, body language and voice.
6
14
1,887
It was a pleasure being able to collaborate with Morph Labs on this, I see Morph Prover as just the beginning of how AI can dramatically shift the landscape of even how innovations in mathematics become accelerated!
The downstream implications are virtually endless.
16
2,117
The video is a screen recording from my own iphone. The app is called mlc chat and is already available for people to use some LLMās on iphone.
2
16
1,934
Here is the model if anybody wants to try it out, test it on other benchmarks, or continue training on other data: huggingface.co/kaist-ai/mistā¦
1
17
1,744
Yea it's cool, actually add this custom instruction in my settings to make it sounds more realistic when talking: Speak in a natural casual prose as if you are speaking verbally through audio on not through written text, also please occasionally use uhm and uh when you need to think about particularly obscure or complex details of a subject.
1
12
374
My friend @nisten also deserves a special thank you! he quantized the model to Q6 which makes it run even faster.
He also kindly provides instructions on his HF page here about how to run it on your own PC!
huggingface.co/nisten/obsidiā¦
1
2
16
1,900
Multi-token prediction during training!
and for post-training they do distillation from their R1 model along with self-feedback to improve capabilities on more open-ended settings.
1
15
3,450
This list is of companies either working on fundamentally new hardware compute paradigms, or ones that I believe may work on novel neural network architecture paradigms.
Some notable mentions: Prophetic AI, Cortical Labs, Rysana and software applications incorporated.
2
14
1,056
LM Studio lmstudio.ai/
Highly recommend, everything from downloading the model to inferencing it locally, all can be done within LM Studio itself end-to-end, don't even need to go to HuggingFace website anymore to download the model š„
1
16
714
I think it'd be useful to just refer to logit based distillation as "logit-distillation" and then the kind that doesn't transfer over logit details has been formally referred to before as "hard target distillation" iirc, or can just say "hard-distillation" as @ramealexandre said.
15
545
Thereās more! The new multi-modal model I collaborated on under Nous Research is efficient enough to do this while running on even a non-pro iphone at practical speeds! Even capable of back and forth conversation about what it sees, called Obsidian-3B, only 2GB of ram needed.
1
15
738
Hopefully labs start testing on these kinds of benchmarks more. I've collected the AI and human scores from various papers, leaderboards, and follow up works. I've put all such data points and other info in this doc for others that may find it useful: docs.google.com/document/d/1ā¦
1
1
15
1,695
Late 2022 to Early 2023 is an outlier case where a 100X leap in GPT scale was released to the public all in just a ~6 month span. A better time period to compare current progress is the over 2 years from July 2020 to October 2022 where no significant GPT scale up released.
14
891
Interesting talk @ylecun did at @MIT just a few days after Llama-2 release. Keep in mind the Llama-2 project began months before the I-Jepa paper released, which showed JEPA having increased efficiency while reaching new SOTA score (at least for images) špiped.video/vyqXLJsmsrk?si=046rā¦
1
13
940
Forgot to mention an important one! It was a pleasure to collaborate on the morph-prover model that is now used by renowned mathematicians like Terrence Tao in theorem searching workflows.
Also worth noting a lot of this wouldnāt be possible without the ecosystem, support and funding from @NousResearch, @TTSLabsAI and @a16z, along with support from my close friends/colleagues Iāve worked with and/or helped me refine ideas along the way @teknium @stablequan @winglian @TheBlokeAI @jeremyphoward and many others.
2
12
787
The related models trained on this data would likely be most or all Capybara models, as well as Obsidian, OpenChat-3.5, OpenHermes-2.5, Jackalope and possibly a few more models that don't publicly reference pure-dove yet in their model card.
2
14
1,788
If 7 month doublings continue, models could achieve 80% accuracy in tasks that take humans 15 hours, by 2029.
If 4 month doubling becomes the trajectory instead, it could end up with models in 2029 having 80% accuracy in tasks that take humans ~3 weeks.
arxiv.org/abs/2503.14499
3
14
971
Thank you to @a16z and @yield for helping to make this R&D possible.
I'm calling this method Amplify-instruct, it synthesizes new turns and creates in-depth multi-turn conversations from quality single-turn dataset seeds made by @jon_durbin @knowrohit07 @teknium and others!
1
13
888
The video data we receive doesnāt seem to increase a humans practical Intelligence much in the written knowledge being testing. We can prove this testing people who have been blind since birth versus those who are sighted. Also humans who canāt hear and those without touch.
2
11
5,335
There is training runs expected to possibly be closer to 600K B200s within 18 months so this range takes those into account too, and cost is displayed relative to H100 hours of compute (as said in top text) So it's best to look at these as relative compute scales between models.
2
12
1,805
These new economic growth forecasts from EpochAI seem to align well with METRs recent capability forecasts. Especially when you look at the estimated times for 99% accuracy at high time horizons, and how those seem to converge with the full automation estimates by EpochAI below..
We developed GATE: a model that shows how AI scaling and automation will impact growth.
It predicts trillionādollar infrastructure investments, 30% annual growth, and full automation in decades.
Tweak the parametersāthese transformative outcomes are surprisingly hard to avoid.
1
1
13
1,163
A simple question you can ask someone in the AI space right now to see how scale pilled they are:
"What do you think happens when you scale up the training compute of Deepseek V3 and R1 by 1,000X?"
3
10
1,299
I can vouch for MLC šÆ even their optimizations for iphone were suprising to me, I posted a video a few days ago of it running locally on iphone and responding well to a fairly difficult prompt.
Running locally on iphone 13 mini without cellular, without wifi, without bluetooth.
Real-time, not sped-up.
Insightful in-depth response to a question involving 2 relatively obscure and complex topics. (I plan to make it even more effecient soon š) ty to WebLLM & MLCChat.
12
1,528
Obsidian is a 3B parameter model with vision that is efficient enough to briskly run on even a non-pro iphone, while demonstrating surprisingly accurate understanding of images and holding back and forth conversations!
Link:huggingface.co/collections/Nā¦
Check out the example convo:
2
12
3,634
I don't use Linux so I have no idea if this is accurate or not, but here is what our Puffin model generates when I ask the same question verbatim. Interface is using just the recommended pre-prompt and format from the repo:
2
11
1,525
āAssociative memory, especially in long context tasksā
You mightāve missed it in the papers, but Hyena and Mamba have already been shown to have significantly better associative memory abilities compared to transformers and other architectures, ESPECIALLY in very long contexts.
Theyāve already tested million context length sequences with Hyena and even showed impressive SOTA abilities in the understanding of DNA sequences (HyenaDNA from Stanford) and species detection, they later did similar tests with Mamba and found that Mamba did it even more impressively than Hyena.
2
1
9
1,578