VP, Applied Deep Learning Research @ NVIDIA
- Tweets 1,802
- Following 488
- Followers 26,506
- Likes 4,872
Today we're releasing NVIDIA Nemotron Nano v2 - a 9B hybrid SSM that is 6X faster than similarly sized models, while also being more accurate.
Along with this model, we are also releasing most of the data we used to create it, including the pretraining corpus.
Links to the models, datasets, and tech report are here:
research.nvidia.com/labs/adl…
36
230
1,380
276,195
A long time ago, back before DLSS was in many games (and when my hair was shorter and less gray), I went to Nintendo HQ to show them an early prototype of DLSS 2, in the hopes that a future Switch console would use DLSS. I'm so proud that the Switch 2 will be DLSS powered!
blogs.nvidia.com/blog/ninten…
45
97
1,026
71,096
One of the best decisions we ever made @nvidia Applied Deep Learning Research was to standardize on @PyTorch for all our research. It has made us more productive and made our work more fun.
Glad to see @OpenAI agrees! openai.com/blog/openai-pytor…
12
152
976
Jensen Huang is an intensely driven visionary. Working at NVIDIA is exciting and fast paced because he sets the tone. I think his story should be more widely known - in my mind he is just as much a tech titan as Steve Jobs, Bill Gates, or Mark Zuckerberg.
newyorker.com/magazine/2023/…
18
93
740
137,127
Here’s how we trained an 8.3B parameter GPT-2. We alternate row- and column- partitioning in the Transformer in order to remove synchronization and use hybrid model/data parallelism. 15 PFlops sustained on 512 GPUs. Details and code: nv-adlr.github.io/MegatronLM
12
197
726
Nemotron-H: A family of Hybrid Mamba-Transformer LLMs.
* Hybrid architecture means up to 3X faster at the same accuracy
* Trained in FP8
* Great for VLMs
* Weights and instruct versions to come soon.
research.nvidia.com/labs/adl…
18
99
625
201,186
I didn't actually convince Jensen, instead I just explained deep learning to him. He instantly formed his own conviction and pivoted NVIDIA to be an AI company. It was inspiring to watch and I still sometimes can't believe I got to be there.
fastcompany.com/90957372/how…
18
55
522
96,286
Language models do just keep scaling!
Today we’re announcing Megatron-Turing NLG: a 530B parameter language model.
Joint work from @NVIDIA and @Microsoft. Trained using Megatron and DeepSpeed on DGX SuperPod.
developer.nvidia.com/blog/us…
15
140
523
Neural rendering takes its next step with DLSS 3.0 on Ada! In addition to DL-powered superresolution, it uses optical flow, motion vectors, and DL to generate entire frames.
7 out of 8 pixels being rendered with DLSS3 come from Neural rendering. #GTC22
10
67
455
People have been asking for a fast open source CUDA matrix multiplication for ages. My colleagues just released one! In many cases, it’s just as fast as CUBLAS. Works on Volta’s tensor cores, too. devblogs.nvidia.com/parallel…
5
175
440
Nemotron-4-340B is released today!
* Base, Instruct, Reward models
* Permissive license
* Great for Synthetic Data Generation
* Designed to help others build their own models
* Sized for inference on 8 NVIDIA H100 GPUs
* Competitive across many tasks
13
85
449
108,609
A 8B-3.5T hybrid SSM model gets better accuracy than an 8B-3.5T transformer trained on the same dataset:
* 7% attention, the rest is Mamba2
* MMLU jumps from 50 to 53.6%
* Training efficiency is the same
* Inference cost is much less
arxiv.org/pdf/2406.07887
17
77
434
118,910
DGX GH200: NVLink *between nodes* creates a system with 256 Grace CPUs (each with 480GB of LPDDR5) and 256 Hopper GPUs (each with 96GB of HBM3).
Each GPU can directly access the memory of any other GPU or CPU at 900 Gbps.
Can't wait to train some models!
developer.nvidia.com/blog/an…
9
72
408
83,249
Four years ago, we split GA100 into two halves that communicate through an interconnect. It was a big move - and yet barely anyone noticed, thanks to amazing work from CUDA and the GPU team.
Today, that work comes to fruition with the Blackwell launch. Two dies. One awesome GPU.
Look closely & you'll see that GH100 (and GA100) are built from two halves that communicate through a split L2. This improves scalability, although I worried it would be hard to program.
The transition was smoother than I expected, thanks to some SW and HW magic! 🧙♀️🧙♂️🪄
7
56
409
65,737
I’m thrilled about @nvidia’s proposed acquisition of @arm. Working together, we will make technology better from embedded devices to the biggest data centers. And Applied Deep Learning Research will discover even more interesting problems to work on.
blogs.nvidia.com/blog/2020/0…
6
70
404
Thank you Nando. This paper was rejected from all the conferences for lack of novelty - I’m glad it was able to find an audience on arXiv.
Nvidia released the megatron language model before the pandemic. It’s amazing how influential this paper became. A must read for people wanting to learn about AI. arxiv.org/abs/1909.08053
3
24
419
184,760
WaveGlow: A non-autoregressive generative model for speech synthesis. Our unoptimized @PyTorch inverts mel-spectrograms at 500 kHz on a V100 GPU, and is easy to train. Paper: arxiv.org/abs/1811.00002 Samples: nv-adlr.github.io/WaveGlow Work with @DeepPrenger and @RafaelValleArt
5
112
359
DLSS 2.0: deep learning for real-time video game rendering. Better image quality *and* faster frame rates.
A great example of Applied Deep Learning Research: our mission is to find new ways of using deep learning and apply them to NVIDIA’s core work.
nvidia.com/en-us/geforce/new…
9
90
338
This is all true.
The way that Jensen Huang runs Nvidia is wild:
40 direct reports, no 1:1s
- Believes that the flattest org is the most empowering one, and that starts with the top layer
- Does not conduct 1:1s - everything happens in a group setting
- Does not give career advice - "None of my management team is coming to me for career advice - they already made it, they're doing great"
No status reports, instead he "stochastically samples the system"
- Doesn't use status updates because he believes they are too refined by the time they get to him. They are not ground truth anymore.
- Instead, anyone in the company can email him their "top five things" with whatever is top of mind, and he will read it
- Estimates he reads 100 of these everyone morning
Everyone has all the context, all the time
- No meetings with just VPs or just Directors - anyone can join and contribute
- "If you have a strategic direction, why tell just one person?"
- "If there is something I don't like, I just say it publicly"
- "I do a lot of reasoning out loud"
No formal planning cycles
- No 5 year plan, no 1 year plan
- Always re-evaluating based on changing business and market conditions (helpful when AI is developing at the pace that it is)
This org is optimized for (1) attracting amazing people, (2) keeping the team as small as it can be, and (3) allowing information to travel as quickly as possible
9
30
334
127,947
1. NVIDIA believes the competition is strong and getting stronger - as it should, since AI is the largest computational problem in history.
2. Accelerated computing is mostly not hardware. I’m surprised people still see NVIDIA’s business as hardware, after all these years.
For example, I can't imagine anyone at Google saying this in 2005, or anyone at Microsoft saying it in 1995. They might have hoped it, but they wouldn't have said it. Maybe hardware has more moats. Maybe this guy is just outspoken. But either way, this is a rare type of claim.
10
33
336
129,972
Why is NVIDIA's campus made of triangles? Of course, the triangle is the basic primitive of 3D graphics, where NVIDIA got its start.
But there is a deeper meaning: it is a physical instantiation of NVIDIA as a learning machine - and a warning against hubris. 1/5
8
45
331
92,255
DLSS 4 is the biggest DLSS yet:
8X more efficient graphics for 4K 240Hz rendering
15/16 pixels generated by AI
A new transformer based neural network dramatically upgrades image quality for Ray Reconstruction and Super Resolution.
piped.video/qQn3bsPNTyI
32
37
314
36,236
Three scaling breakthroughs for NLP: fastest BERT-Large training (under one hour), fastest BERT inference (2.2ms on T4), and largest Transformer (GPT-2 8.3B). Code is open source. devblogs.nvidia.com/training…
3
108
318
The new 80GB A100 GPU has more than 2 TB/s memory bandwidth!
We’ve already started using these @NVIDIAAI to train Megatron language models. 🚀
nvidianews.nvidia.com/news/n…
10
43
309
Image padding causes small but measurable artifacts for CNNs. Partial convolution padding improves ResNet-50 Top1 by 0.478% on average, and improves semantic segmentation results on image borders. Paper by @GuilinL: arxiv.org/pdf/1811.11718.pdf Code: github.com/NVIDIA/partialcon…
1
85
304
Today @nvidia launched Cambridge-1, which is now the largest supercomputer in the UK. I expect there will be a lot of research done on this computer at the intersection of HPC, AI, and healthcare. Can’t wait to see what people do with this machine!
nvidia.com/en-us/industries/…
6
50
281
Very excited for the RTX 3080 reviews. So much blood, sweat and tears has gone into making Ampere and now the world gets to experience it!
Also excited to see DLSS shipping in more games. Still amazes me that we can run a DL network to render 8K video at high FPS. #RTXOn
6
22
278
I’ll be giving a seminar next week @Berkeley_EECS on Applied Deep Learning Research at NVIDIA. DL applications are numerous and exciting, but actually applying DL in the real world is rarely straightforward. I’ll be talking about how we approach it.
eecs.berkeley.edu/research/c…
3
51
266
Today’s Insight from Jensen Huang: People think that a team player is someone who is willing to help others. But @nvidia, a team player is someone who *asks* for help. Who passes the ball, not keeps it to themselves. This is how we work as one team: by asking for help.
3
37
254
Waveglow: now with code and pre-trained models! github.com/NVIDIA/waveglow
2
69
261
This code is astonishingly fast. 320 DeepVoice voices in real time on a V100 GPU, and handles bigger models as well. Super useful for our TTS work. @PyTorch wrapper coming soon!
Just released by @NVIDIA on GitHub: nv-wavenet, a reference implementation of a #CUDA-enabled autoregressive WaveNet inference engine. nvda.ws/2vBt6il
1
66
253
“I don’t know how to do it (but) for all of you Stanford students, I wish upon you ample doses of pain and suffering. Greatness comes from character and character isn’t formed out of smart people—it’s formed out of people who suffered.”
fortune.com/2024/03/13/nvidi…
11
42
241
27,851
.@NVIDIA is the only place crazy enough to conceive of rendering 4K 60+ Hz graphics at home by running a neural network for every frame.
DLSS: Trained in @PyTorch, running on the Tensor Cores in RTX GPUs, redefining graphics with AI.
And we’re just getting started!
And just in case you'd forgotten just how BEAUTIFUL
@DeliverTheMoon looks?
You should probably have a quick look at the wonders @NVIDIAGeForce is pulling off with their DLSS implementation.
Its magic. Pure Space Magic.
4
39
245
DLSS 2.3 features a new model that reduces ghosting and particle erasure.
Temporal reconstruction uses the strong correlations between frames to reveal more detail. And DL is great at learning these correlations, so DLSS just keeps getting better! 🌅
piped.video/SBjL0M25t04
9
35
229
Announcing Megatron-BERT: a 3.9B parameter model trained with intra-layer model parallelism. State of the art results on the RACE test set. Trains with extreme efficiency on NVIDIA V100 and A100 GPUs.
More information: devblogs.nvidia.com/language…
5
57
233
I don’t recommend rendering at 240p resolution, but it’s true that DLSS is magic. 😆
kotaku.com/dlss-on-240p-vide…
6
53
241
With a bigger training set, our 8.3B parameter GPT-2 model now gets a WikiText-103 perplexity of 10.8 (previous SOTA 16.3), and a Lambada whole word accuracy of 66.5% (previous SOTA 63.24%). Updated results in our blog post:
nv-adlr.github.io/MegatronLM
1
47
226
Here's Jensen working on a breadboard!
I'm inspired when I think of the various challenges he overcame to immigrate to the US and become an engineer and found @NVIDIA. Racism being one of them.
We can and will do better.
venturebeat.com/2021/07/20/j…
1
37
223
After Jensen's amazing #GTC24 keynote today, I just had to dig up this old picture of me sitting in the demo pit for #GTC14, which was the first GTC keynote to have significant DL/AI content.
And my demo that day almost failed! Here's what happened:
Eight years ago, I built and operated two demos in Jensen’s #GTC14 keynote. We were trying to tell the world about why DL matters. Jensen was prescient to do this back then.
Fun to look back and remember how thrilling it all was. And I’m even more excited today. 🚀
ALT Bryan Catanzaro sitting behind a bank of computer monitors on stage at NVIDIA GTC 2014.
5
12
215
46,851
Someone called me a leftist because I said in this interview that stopping global warming should be a super high priority. Fact is, I don’t think conserving our planet is a leftist agenda. I think all of us, regardless of politics, want to keep the earth vibrantly alive.
3
9
212
As part of Nemotron, we're releasing a new Math dataset, made by rendering webpages using Lynx and then using an LLM to rewrite the result into LaTeX.
Our models got much better at math when we started using this dataset. We hope it's helpful to the community. 💚
We just released Nemotron-CC-Math 🚀
Equations on web aren’t just LaTeX-they’re in MathML,<pre> tags,inline,even images.Code shows up just as many ways. Most parsers drop it.
Nemotron-CC-Math(133B tokens) reprocesses CommonCrawl math pages to capture math equations +code reliably
10
33
221
33,079
DLSS3 has been a labor of love in Applied Deep Learning Research since the group was founded. I can’t wait for people to play with it.
@nvidia has incredible tenacity when inventing revolutionary technologies and I’m proud to be a part of this work.
17
27
190
Wisdom from Jensen Huang:
Strategy is not about what you will do. It's about what you will sacrifice - what you will give up in order to focus on the most important thing.
6
21
193
32,283
We are redefining real-time graphics with deep learning.
DL gives us both better image quality and 1.4-2X higher frame rate.
Usually you get either speed or quality, not both.
So glad to see this technology shipping in games - check out this review!
piped.video/watch?v=gpzFX4P1…
4
28
193
Partial convolutions for image inpainting lead to some surprising photo editing tools. Take a look at the video, and read the the arXiv paper if you’d like technical detail. Great work from @GuilinL and our Applied Deep Learning Research team at NVIDIA! news.developer.nvidia.com/ne…
3
71
191
We just open sourced NVVL: a library that provides GPU accelerated video decoding for DL training. Save 40X on your storage space and bandwidth, reduce CPU load by 2X when training on video datasets. Great for GPU dense systems like DGX-2.
github.com/NVIDIA/nvvl
2
83
188
NVIDIA’s office is designed to force everyone to run into each other randomly throughout the day.
It makes coming to the office fun because I never know who I will see, and often after I say hi to an old friend, I remember we have work to do together!
piped.video/watch?v=qDg_3e…
10
12
182
28,342
DLSS is a more sample-efficient rendering algorithm.
Take a look at these iso-FPS IQ comparisons in Deathloop against native rendering and against AMD FSR.
DLSS makes games faster AND more detailed, because it reconstructs more signal from each pixel.
overclock3d.net/reviews/soft…
14
23
182
Today at an after work event, Jensen served 100 people (including me) food and drink. He balanced 10 plates on his arms while circulating among the crowd. His first job was bussing tables at Denny's and he still remembers the lessons.
blogs.nvidia.com/blog/2023/0…
7
12
174
31,537
I am so excited about CUDA support for WSL! I've been looking forward to this for ages. Can't wait to train some models using @PyTorch and the tensor cores of my @Razer Blade 15". Great work from @Microsoft and my friends at @nvidia.
devblogs.microsoft.com/direc…
2
34
174
My son wrote a program that made his first segmentation fault today.
First thought: I am so proud of you, you are now a real programmer!
Second thought: Why are C++ errors still so unhelpful? Beginners have no idea what "segmentation" even is or why it would fault. 🤦♂️
9
5
167
There were a lot of people at Intel back in 2007 that I worked with that saw both the opportunity and the risk for Intel. Back then NVIDIA was 10X smaller revenues so Intel management thought they would crush NVIDIA with Larrabee. But Intel lacked vision and execution.
5
12
165
14,744
DLSS just keeps learning! 🚀
We are posting beta DLSS builds to gather feedback from gamers around the world. Starting with our White-Collie model, hot off the supercomputer. 🧇
Can't wait to hear what you see!
developer.nvidia.com/dlss/re…
7
35
166
Technical insights into what makes DLSS4 so great, including MFG, transformer models, and Reflex Frame Warp:
research.nvidia.com/labs/adl…
8
31
174
18,139
I thought this video did a great job showing how DLSS Ray Reconstruction improves temporal stability and image detail.
It sometimes feels like putting glasses on when we turn on DLSS.
piped.video/watch?v=dYZ1jYTT…
9
14
161
87,409
We’ve had a dream for many years of a TTS engine with enough emotional range to do voice acting. We’re incredibly proud to unveil Flowtron, the TTS narrating the 2020 I AM AI video. Take a listen: piped.video/bOf2S7OzFEg?t=150
2
37
154
I’m really excited about torch.jit, torch script, and the new C++ interface to @PyTorch. Lots of thoughtful design choices that will help our research.
I’ll be speaking about how we use PyTorch for NVIDIA Applied Research at 2:55 pm PDT. Stream live here: facebook.com/722677142/posts…
1
27
150
Eight years ago, I built and operated two demos in Jensen’s #GTC14 keynote. We were trying to tell the world about why DL matters. Jensen was prescient to do this back then.
Fun to look back and remember how thrilling it all was. And I’m even more excited today. 🚀
ALT Bryan Catanzaro sitting behind a bank of computer monitors on stage at NVIDIA GTC 2014.
3
10
143
NVIDIA has been very strategic about building custom chips for AI for over a decade now. I have been honored to be part of that effort.
wired.co.uk/article/nvidia-a…
2
23
144
We did recommend padding vocabulary size in Section 5.1 of the original Megatron paper (2019).
Guess we didn't advertise it enough, based on this thread.
arxiv.org/pdf/1909.08053.pdf
The @MosaicML perf team just tried this out and... totally confirmed 🤯
GPT-1.3B MFU went from 49% -> 53%
6
12
145
77,535
We've been making a lot of progress on LLM pretraining in NVFP4:
developer.nvidia.com/blog/nv…
4
26
157
12,379
Went to work today in @nvidia Voyager. I think this new building is great for post-COVID work - it feels like most of the space is dedicated to collaboration, and that’s the reason many of us have for coming in. Can’t wait to see the offices humming again!
4
4
142
For one of the ten biggest companies in the world, NVIDIA is not well known. CNBC just posted a great documentary about how NVIDIA came to power modern AI. It’s been an honor for me to be a part of this story.
Ahead of #GTC23, I sat down w @nvidia CEO Jensen Huang to hear how his big bet on AI is finally paying off + how it's mitigating risks like China export controls. Here's the story of Nvidia's rise from revolutionary graphics to #ChatGPT engine, with @ctnzr
piped.video/d3L2uPuxOxU
5
16
141
28,542
Jensen Huang is a time machine who lives in the future. So much of @nvidia’s great success over the past 5 years happened because of the strategic decisions that he made 15 years ago along with his continuing investments to bring them from vision to reality.
Nvidia's Integration Dreams
Nvidia's acquisition of ARM only makes sense from a financial perspective, unless you buy Jensen Huang's datacenter dreams.
stratechery.com/2020/nvidias…
22
138
NVIDIA Research has been a great place for me to do some of my best work. This podcast discusses why, and also highlights three projects I'm excited about: noise-to-noise denoising, semantic image manipulation, and unsupervised learning for text. blogs.nvidia.com/blog/2018/0…
2
44
137
NVIDIA’s journey to transform the video card into the AI chip of the future took a lot of vision from the whole community and also 20 years of sustained investment. Some history from Yoshua Bengio, @AndrewYNg, and me in this podcast:
economist.com/podcasts/2021/…
3
36
131
I am so excited to see the release of the GPL/MIT NVIDIA kernel modules. A labor of love by a lot of people and I think it makes the GPU computing ecosystem better:
developer.nvidia.com/blog/nv…
2
34
131
DLSS3: 4x faster than brute force rendering. We can’t wait for you to see for yourselves!
#RTXOn #BeyondFast
piped.video/pSiczcJgY1s
18
27
127
mRNA vaccine is a miracle. Today I am contemplating that miracle as my body manufactures COVID spike proteins and learns to destroy them.
Huge thanks to the scientists, engineers, logistics specialists, nurses and doctors and everyone else that made this miracle happen. 🙏❤️💉
2
5
130
Nemotron-CORTEXA just reached the top of the SWEBench leaderboard for using LLMs to solve software engineering problems, solving 68.2% of SWEBench GitHub issues!
It does so by using a multi-step problem localization and repair process, generating multiple proposal candidates and then choosing a final solution with an LLM.
The embedding model we built for this has been released, and code will be released soon. Paper will be at ICML 2025!
More information here:
research.nvidia.com/labs/adl…
5
38
137
12,709
Details of the Megatron-Turing NLG 530B language model, including how it was trained and how it performs compared to GPT-3, as well an investigation into bias and some sample generations.
arxiv.org/abs/2201.11990
3
31
124
MLPerf 0.6 is out! Training speed on a single DGX-2 improved up to 1.8X over 7 months ago, and scaling efficiency to many nodes is markedly better.
The V100 GPU, 2 years after intro, is still the fastest and most general processor for deep learning.
blogs.nvidia.com/blog/2019/0…
2
35
128
For the past couple years, the I AM AI videos that @nvidia makes for GTC have been narrated by TTS.
Our research in TTS models and stronger conditioning give TTS new expressivity for content creation.
Take a look at how this works:
piped.video/RknIx6XmffA
1
22
118
When we started DLSS research, we had a dream of seeing AI video reconstruction everywhere. There were a lot of skeptics, but we persevered.
So it’s deeply satisfying to see that there are now over 100 games and apps that have chosen to use DLSS. 🚀🚀🚀
nvidia.com/en-us/geforce/new…
6
17
120
Microsoft has been tirelessly and rapidly expanding the DeepSpeed library to new speeds and new capabilities. I’m excited to see the huge new language models that will be trained using this code, and glad to see @nvidia’s Megatron-LM contributing to the effort.
Microsoft’s updated DeepSpeed can train trillion-parameter AI models with fewer GPUs venturebeat.com/2020/09/10/m… via @VentureBeat
1
27
120
It takes great data to make a great model.
We're opening the data curation pipeline for Nemotron models, and we're also posting as much of the Nemotron training and post-training data as possible.
These days, data is a fundamental part of accelerated computing.
Curating high-quality pretraining datasets is crucial for accurate #LLMs. 💬
With our Nemotron-CC pipeline, now in the NeMo Curator GitHub repo, you can process text, image, and video data at scale.
Get an overview of the pipeline and how you can use it to generate high-quality tokens for training or fine-tuning LLMs.
➡️ nvda.ws/44r5POU
4
31
125
21,826
I think I should be able to charge a fee to anyone who wants to send me an email. Could be a new way to make money @Google.
15
4
113
.@tcwang0509 will be presenting an oral presentation Thursday at CVPR on his Pix2PixHD work generating high-resolution synthetic images with conditional GANs. This is a big step towards a generative model for rendering graphics. news.developer.nvidia.com/ge…
1
49
114
NVIDIA halves ETH hash rate of RTX 3060 to make it less attractive to miners. Hopefully this makes it easier for gamers to get an RTX 3060, despite historically high ETH prices.
DLSS is going to be great on the 3060!
blogs.nvidia.com/blog/2021/0…
2
24
115
NVIDIA has a secret which we have been shouting from the rooftops for a decade: AI is the world’s most important computational problem, and chips could never be enough to solve it.
A bit of retrospective about how NVIDIA got here and where we are going:
Proud of this in-depth feature about @nvidia's rise to #AI dominance — which I worked on for over a month! Thanks to @ManuvirDas @ctnzr @karibriski at NVIDIA for speaking with me & @nathanbenaich @ChiragDekate @karlfreund for their insights. venturebeat.com/ai/how-nvidi…
2
20
112
31,745
The Megatron team at NVIDIA is hiring! We're looking for people who can contribute to any aspect of foundation model research and development:
nvidia.wd5.myworkdayjobs.com…
28
115
38,773
WaveGlow inference: now faster! We improved mel-spectrogram inversion speed from 500 kHz to 1200 kHz on a single V100 GPU. The model is the same, just implemented better. More to come! github.com/NVIDIA/waveglow
1
28
117
#GTC22 was our best yet. In the Applied Deep Learning Research, we're trying to make DL useful and real to NVIDIA's work with practical and concrete research. Here's our talks from the conference: 🧵
2
18
111
The Llama 3 models look fantastic! And we're proud to support them on Day 0 throughout NVIDIA's product stack: blogs.nvidia.com/blog/meta-l…
We just released Meta Llama 3: the most capable openly available LLM available to date! The 8B & 70B models are out now, and we expect to release models with larger context windows, additional model sizes and more capabilities in the coming months.
1
12
112
22,012
In an era where inference speed yields more reinforcement learning, which yields more AI, we feel hybrid SSM-transformer models have some compelling advantages.
Nemotron-H-47B-Reasoning-128k is a bit more accurate than Llama-Nemotron-Super-49B-1.0 across all benchmarks, but it is up to 4X cheaper to infer. Checkpoints released on HuggingFace under a non-production license.
New reasoning Nemotron-H models are now publicly available.
These models are based on hybrid architecture!
47B and 8B in BF16 and FP8.
Blogpost: developer.nvidia.com/blog/ne…
Weights: huggingface.co/collections/n…
4
19
117
12,938
Base model Nemotron-H weights have been released under a research license:
huggingface.co/nvidia/Nemotr…
Nemotron-H: A family of Hybrid Mamba-Transformer LLMs.
* Hybrid architecture means up to 3X faster at the same accuracy
* Trained in FP8
* Great for VLMs
* Weights and instruct versions to come soon.
research.nvidia.com/labs/adl…
3
21
118
21,419
Coming soon to NeMo-Megatron,
One Weird Trick to speed up large language model training by 30%:
arxiv.org/abs/2205.05198
(Combining sequence and tensor parallelism, you don’t need to do activation recomputation.)
3
21
105
Semantic segmentation just keeps getting better, which is awesome because it’s so useful. SOTA results from @drewtao and @currenator, trained in FP16 mixed precision on 32 V100 Tensor core GPUs.
Proud to release both our paper arxiv.org/abs/2005.10821, and a blog post devblogs.nvidia.com/using-mu… on work with @currenator on Semantic Segmentation. We have state of art results in Cityscapes and Mapillary. Code release coming soon as well!
2
22
107
MLPerf 0.5 results are out, and NVIDIA Volta GPUs dominate the results, demonstrating faster performance and more flexibility than CPUs or TPUs. Anyone can get this performance using our containers, on systems small and large, on premise or in the cloud.
news.developer.nvidia.com/nv…
1
40
108
I'm glad to see Nemotron-4-340B doing well on the LMSYS Arena leaderboard! Lots more work to do to improve the Nemotron family.
Most importantly, though, I hope this model supports the development of AI across the community.
Chatbot Arena update!
@NVIDIAAI's Nemotron-4-340B has just edged past Llama-3-70B to become the new best open model on Arena leaderboard!
Key highlights:
- Impressive performance in longer queries
- Balanced multilingual capabilities
- Robust performance in "Hard Prompts"
Congrats @NVIDIAAI for this remarkable milestrone & contribution to the open community!
Check out more plots below👇
1
17
108
12,990
Software is critical to DL training speed - and yet it is often strangely forgotten as people focus on chips.
High performance software is core to NVIDIA’s work.
We love chips at NVIDIA but we never forget the software that makes them sing. As shown in these results:
On the latest round of #MLPerf v1.0 training submissions NVIDIA improved up to 2.1x on a chip-to-chip basis and up to 3.5x at scale, setting 16 performance records. Learn how: nvda.ws/3hjAoM9
26
101
Academic grants of @nvidia GPUs were important to my research in grad school. Here’s the link to apply if you have some research projects of your own that could benefit from some @nvidia hardware.
2
30
100
The next DLSS feature: ray reconstruction. Joint denoising and superresolution for raytracing. Solving these problems jointly makes games more detailed and dynamic, on all RTX GPUs.
DLSS just keeps learning.
piped.video/sGKCrcNsVzo
10
25
95
22,542
If you built a company fundamentally based on Amdahl's Law, it would look like @NVIDIA. We work together as one team to optimize everything from chips to applications. Accelerated computing requires it. piped.video/watch?v=jqdnOjOn…
1
16
96
NVIDIA benefits greatly from the open-source community, and we're excited to be able to contribute back. It's great to see so much energy in open-source AI!
4
17
103
13,862
“If you aren't using DLSS, you're not seeing the game at its best. It's a remarkable thing to say, but Nvidia's AI upscaling in its quality mode not only improves image quality..but it also adds a huge performance increase.”
🤘🙏@digitalfoundry
eurogamer.net/articles/digit…
1
18
92
@MistralAI and @nvidia announce Mistral-NeMo 12B, an awesome bite-size model released under Apache 2.0 that we jointly trained. FP8 aligned checkpoint and 128k context window, great benchmark scores.
blogs.nvidia.com/blog/mistra…
mistral.ai/news/mistral-nemo…
2
26
99
9,848
A nice explanation of SPADE for image translation: adamdking.com/blog/gaugan/
1
24
99