Inside Nemotron and NVIDIA's AI lab: my conversation with Bryan Catanzaro (@ctnzr). @nvidia is a chip company. So why does it put hundreds of researchers on building AI models - and then give them away for free? We go deep into the Nemotron models, what it takes to build a top AI lab, and the future of frontier AI. 01:33 - Is open source AI catching the frontier? 05:29 - Do closed labs blocking distillation slow open source down? 07:42 - Is the US falling behind China? 10:30 - Why companies actually choose open models 12:39 - A "crazy" 2008 bet: machine learning on GPUs 15:33 - Working with Andrew Ng and Dario Amodei at Baidu 17:41 - Coming back to NVIDIA: DLSS and the birth of Megatron 21:55 - The real reason NVIDIA builds its own models 24:28 - Is Moore's Law really dead? 33:37 - The Nemotron family: Nano, Super, Ultra 35:09 - Built for agents: why NVIDIA bets on speed 36:02 - How you train a 550B model in 4 bits 39:25 - Hybrid Mamba-Transformer, explained simply 42:31 - Mixture of experts, and why NVIDIA built NVL72 around it 47:26 - Why a 1-million-token context window matters 49:26 - Multi-token prediction: how the model predicts 5 tokens at once 52:47 - Multi-teacher distillation: teaching one model from many 58:01 - Where reinforcement learning goes next 01:00:16 - Inside NVIDIA's research org: "the mission is the boss" 01:04:03 - How NVIDIA decides who gets the GPUs 01:10:53 - Why NVIDIA still feels entrepreneurial after 33 years 01:12:58 - Why Bryan doesn't believe in the singularity 01:17:50 - The AI backlash 01:19:18 - The controversial case: open AI is safer than closed

“Yes I feel fully recharged after a restful holiday”

Shot on a Shot from a $700 iPhone $65M fighter plane

When an entire administration realizes their kids are not going to get in

A classic by now, so perfect lol

Movie watching experience: 2005: go to a movie theater 2015: stream Netflix 2025: ask LLM + text-to-video to create a new season of Narcos to watch tonight, but have it take place in Syria with Brad Pitt, Mr Beast and Travis Kelce in the leading roles

Sam Altman returning to OpenAI after a day is like Steve Jobs returning to Apple after 12 years, but for the TikTok generation

Us: “Super productive week, I’m fully caught up on email and almost done with my big presentation!” Elon:

Microsoft: "we're good for our $80B" Meta: "we're good for our $65B" DeepSeek: "we're good for our $5.5M"

The cafeteria at the Meta office on Monday

Excited to reach 100k on Gmail, thanks everyone for the support on my creator journey

I think about this almost every day

Seed investor who hasn’t spoken to the company in 7 years, on IPO day

Europeans, as US tariffs force them to rely exclusively on German cars, French wine and Italian fashion

Nike: just do it Red Bull: just did it

Love this shot of Jensen Huang from the NVIDIA Q4 results announcement

Breaking: Facebook India to be renamed to Mehta

Silicon Valley pivoting from Trump to Harris even faster than it pivoted from crypto to AI

VCs when they hear that someone is leaving OpenAI to start a new company

How a startup vs. How a startup looks on the looks on the outside inside

Name this venture firm

CEO: “we are CTO: fully powered by Generative AI”

Google buying DeepMind for $400M-$650M in 2014 may very well be the best acquisition of all times.

The pitch deck vs the 1st board meeting

When your startup never gets acquired

My Waymo in SF *honked* at a car that was reversing ahead of us All it needs now is a robotic voice that screams “what the hell are you doing, motherf*cker”, and it will be ready for the streets of New York

A rare video of the SVB bank run

NYC cabs upgraded to Tesla and I’m here for it

I’m struggling to use ChatGPT, it’s really unfortunate there are no threads on here to help me

I know Meta has huge engineering resources and expertise, but kinda wild that a new product could go from zero to 100M users in days and the whole infra scales beautifully without a glitch or even slowness

Everyone in tech: A/B testing Bill Gates: A/B/C/D/E/F/G testing

I did like Uber better when it was subsidized by VCs lol

Hearing growing buzz in startup land about something called “EBITDA”. Anyone know anything? I checked, not a new coin apparently

If you’re in your 20s in tech right now, a couple of years at OpenAI, Anthropic or a great AI startup will give you 10x the career juice of any MBA or graduate degree (and a lot less debt).

20th century wonder vs 21st century wonder

DeepMind DeepL DeepSeek If your AI startup's name start with "Deep", I don’t care what it does, I’ll invest in the pre-seed

Some stories rarely get told: * Startups that turn down a big M&A offer and fizzle out in obscurity years later * VC firms that can’t raise the next fund * Founders who fail and never quite recover professionally Behind the scenes, things are even harder than people think.

Who else has been doom scrolling non stop for the last 4 days

Gaslight: An app for teenagers to anonymously compliment each other, except you make it B2B and it’s for office colleagues to anonymously say mean things about each other and you call it 360 degree feedback performance review.

I am a venture capitalist. Here's my daily routine 🧵 8am: Wake up hungover from a crypto dinner. While in bed, tweet how refreshed I feel from a great night on my Eightsleep and my 1-hour morning meditation.

Reporting live as VCs amass in front of Sam Altman’s house following rumors of new venture

We are so back.

KKR, just your standard master of the world office

Breaking: USA raises $1.9 trillion growth round

The junior person you ignore or dismiss today will be the decision-maker of tomorrow. When your paths cross again years later, you may not recognize them, but they sure will remember you.

Series J is the new IPO

A senior partner at McKinsey told me recently how AI has dramatically raised the bar in consulting: 1) Junior team members are expected to turn around presentations MUCH faster 2) To win new business, you now include in a pitch what would have been final work product before

Google Translate for venture capitalists, 2021 vs. 2022.

VC is the best job in the world Market goes up: “portfolio is up 10x, we’re geniuses” Market goes down: “you all founders are going to need to be a lot more disciplined and focused, for real”

Running a successful startup

Epic (sound on)

“Yup that’s me, you’re probably wondering how I got here”

Halloween is going to be lit this year

I was on an H1B visa for 10 yrs. Started a company with co-founders also on H1Bs. Hired American employees, served US customers. This was our American dream. But today the same company could be started anywhere. Why would the next generation bother if they can’t get a visa?

When you have raised huge amounts of VC money and your board tells you to keep the burn low nitter.app/PubityIG/status/156201…

“Which part of the AI stack would you say you’re strongest in?” Google: “yes.” #GoogleIO

How it How it’s started going

Amazing how many top tech companies are run by Indian CEOs

$2B: the purchase price of Credit Suisse, a global firm with $22B in revenues and 50k employees Also, $2B: the current valuation of dozens of (also money-losing) unicorns with <$50-75M in revenues

The new startup founding team: * CEO * CTO * 2 engineers * GitHub Copilot

VCs switching from crypto to generative AI nitter.app/50grampus_tmj/status/1…

Can only imagine if we were attacked by a foreign nation and all of us men 18-60 on Tech Twitter had to go fight We'd all be like "10 best ways to fire your automatic weapon, a 🧵" and "Don't fight hard. Fight smart. It's not about the number of hours"

VCs tweeting vs VCs investing

My VC resolutions for 2023: 1) Update public profiles: remove any reference to ever having ever liked crypto/web3, deny any rumors that I was claiming to be “down the rabbit hole” less than a year ago, say that I would have “definitely done deep due diligence” on FTX

When you ask Web3 founders about their monetization plan

AI on X: O̶p̶e̶r̶a̶t̶o̶r̶ ̶i̶s̶ ̶h̶o̶t̶ ̶D̶e̶e̶p̶ ̶R̶e̶s̶e̶a̶r̶c̶h̶ ̶i̶s̶ ̶h̶o̶t̶ ̶D̶e̶e̶p̶S̶e̶e̶k̶ ̶i̶s̶ ̶h̶o̶t̶ ̶G̶r̶o̶k̶ 3̶ ̶i̶s̶ ̶h̶o̶t̶ ̶S̶o̶n̶n̶e̶t̶ ̶3̶.̶7̶ ̶i̶s̶ ̶h̶o̶t̶ ̶G̶P̶T̶ ̶4̶.̶5̶ ̶i̶s̶ ̶(̶n̶o̶t̶)̶ ̶h̶o̶t̶ ̶Q̶w̶e̶n̶ ̶i̶s̶ ̶h̶o̶t̶ ̶M̶C̶P̶ ̶i̶s̶ ̶h̶o̶t̶ Manus is hot

VC meeting a 24 year old *applicant*: "I don't know if you're good/mature enough to get an associate job at my firm to do excel and cold emailing" VC meeting a 24 year old *founder*: "Here's a term sheet at $100M pre, come to my house to meet my family and you can ping me 24/7"

The golden age of design

Thrilled to announce our newest partner

Literally the #1 reason why America has produced so many incredible startups

L: bootstrapped founder doing $80M ARR owning 100% of the company R: YC founder who raised $5M on a $30M cap, pre-product pre-revenue

The pitch The 1st board deck meeting

I went to a VC party in a hip hotel in NYC last night. As I arrived in the hotel lobby, the cooler-than-thou staff took one look at my Patagonia and Common Projects, and ushered me directly to the elevator to the VC event. Not a single word was spoken, they just knew.

Adobe blocked from buying Figma JetBlue blocked from acquiring Spirit Airlines Amazon abandons $1.4B deal to buy iRobot as it sees "no path to regulatory approval" Something is very broken in antitrust enforcement

Doordash acquiring SevenRooms for $1.2B, an unprecedented $171M per room. We're so back.

SaaS startup Fintech startup $3m ARR pre-product growing 85% yoy $100m pre

Calls set up by VCs: Zoom Calls set up by founders: Google Meet

Careers are very long. People may get lucky early, but it is amazing to observe how the passage of time eventually favors the persistent and focused. I have now seen this play out so many times.

It’s remarkable that AI hallucinations went from the biggest topic just a year or so ago to a largely fixed problem people barely talk about anymore today.

Finance math VC math

New York, the city where everyone is a venture capitalist

VCs tweeting about how to operate a startup

Wow Black Friday is 🔥 this year

Being “good at prompt engineering” in 2023 is like being “good at Googling” in 2003

When you’re over 40 and went on a run the day before

AI leaders in the US 🇺🇸 Yann LeCun, b. France 🇫🇷 Ilya Sutskever, b. Russia 🇷🇺 Andrej Karpathy, b. Slovakia 🇸🇰 Fei-Fei Li, b. China 🇨🇳 Mira Murati, b. Albania 🇦🇱 Wojciech Zaremba, b. Poland 🇵🇱 Mustafa Suleyman, b. UK 🇬🇧 Andrew Ng, b. UK 🇬🇧 etc. Keep immigration open

I’d love to use GPT-4. But I’m STUCK in newbie mode. Too bad there’s no content to show me the 10 BEST PROMPTS that would BLOW MY MIND.

People make fun of Twitter VC, but retooling overnight from epidemiology expert to oil macroeconomist ain't easy.

VCs tweeting vs VCs investing

1 month into 1 year into building a building a startup startup

CEO: “we have thousands of AI agents running in production” CTO:

Everyone: Design is eating the world! Berkshire Hathaway: here’s our WEB page

Reddit: acquired for $10m PowerPoint: acquired for $14m Venmo: acquired for $26m Android: acquired for $50m Timing an exit is one of the hardest things in startups/venture.

Every fundraising pitch right now

As a white guy who spent some time at Yale Law School, has an interracial family, was a lawyer briefly then a tech VC... I still have a hard time believing that another white guy who met his Indian American wife at Yale Law School, has an interracial family, was a lawyer then a tech VC briefly, can be truly, genuinely a radically anti-immigrant, "make abortion illegal nationally", republican hardliner. Everyone is different for sure, but it's just SO not the vibe.

Omg such a deep crash we are doomed

Vibe capitalism

Running list of startups / categories that are proving anti-fragile to the current mess: * Remote work: Zoom etc * Gaming, online entertainment * Social (Twitter), messaging * Food delivery: Blue Apron etc * Online education * Telehealth, digital thermometers * What else?

The energy in NYC right now is crazy. Center of web3, everyone in tech moving here, tons of VC money, great startups everywhere. So fun.