delved, you say?
Washingtonians delved into the world of artificial intelligence (AI) at the Washington AI Network’s inaugural weekend TGAIFriday Lunch for White House correspondents. trib.al/FwHF9Um
15
371
5,489
210,015
Yet another AGI lab that will need 100k H100s
Superintelligence is within reach.
Building safe superintelligence (SSI) is the most important technical problem of our time.
We've started the world’s first straight-shot SSI lab, with one goal and one product: a safe superintelligence.
It’s called Safe Superintelligence Inc.
SSI is our mission, our name, and our entire product roadmap, because it is our sole focus. Our team, investors, and business model are all aligned to achieve SSI.
We approach safety and capabilities in tandem, as technical problems to be solved through revolutionary engineering and scientific breakthroughs. We plan to advance capabilities as fast as possible while making sure our safety always remains ahead.
This way, we can scale in peace.
Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent.
We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.
If that’s you, we offer an opportunity to do your life’s work and help solve the most important technical challenge of our age.
Now is the time. Join us.
Ilya Sutskever, Daniel Gross, Daniel Levy
June 19, 2024
51
223
5,319
407,077
The most underrated superpower of Generative AI is converting unstructured data to structured data
96
425
5,103
928,915
ngl, going from game dev to Nobel prize in Chemistry is a goated career trajectory
Demis Hassabis, awarded the 2024 #NobelPrize in Chemistry, was born in 1976 in London, UK.
He earned his PhD in 2009 from @UCL, UK. Hassabis is currently the CEO of @GoogleDeepMind, London, UK.
deepmind.google/about/
23
267
2,615
270,518
Same response in Morocco. When I was back there not that long ago, barber noticed from my weakened accent that I had been living outside for some time. Unprompted he said, "look, whatever you do, don't ever consider moving back"
1
18
2,104
72,716
Why Google Deepmind's Mixture-of-Depths paper, and more generally dynamic compute methods, matter:
Most of the compute is WASTED because not all tokens are equally hard to predict
27
223
1,741
354,784
Adobe realizing their $20B acquisition of Figma is now worth zero because of AI doodling
28
86
1,339
325,489
When you realize Apple cooked harder with a calculator than Humane and Rabbit combined
10
72
1,326
63,164
The train.py file in Karpathy's nanoGPT is a marvel. It does everything right, exactly the opposite of what you were taught in school. Global variables at the top, tons of top-level code (no main), a big while True
github.com/karpathy/nanoGPT/…
29
101
1,280
391,782
It's funny how almost a decade later, the frontier labs are back at the same spot, building RL gyms
Our reinforcement learning toolkit, OpenAI Gym, is now in public beta: gym.openai.com/.
15
55
1,295
206,588
🤯 I still cannot get over this Bing AI example.
The riddle was posed in base64, which it figured out alone.
It decoded the base64 step-by-step and discovered the instructions with math to be solved.
It solved the math unlocking the true riddle.
Finally it solved the riddle.
17
97
1,192
449,278
By far the simplest implementation of mixtral I've found was by @realGeorgeHotz on his latest stream
Code: github.com/tinygrad/tinygrad…
Stream:
piped.video/H40QRJFzThQ?si=IZIE…
ALT model = Transformer(n_layers=32, dim=4096, hidden_dim=14336, n_heads=32, n_kv_heads=8, norm_eps=1e-5, vocab_size=32000, feed_forward=functools.partial(MixtureFeedForward, 8), jit=False)
![class MixtureFeedForward:
def __init__(self, num_experts:int, dim:int, hidden_dim:int, linear=nn.Linear):
self.gate = nn.Linear(dim, num_experts, bias=False)
self.experts = [FeedForward(dim, hidden_dim, linear) for _ in range(num_experts)]
def __call__(self, x:Tensor) -> Tensor:
assert x.shape[0] == 1, "only BS=1"
g = self.gate(x).exp()
choice = g.data().tolist()[0][0]
top = sorted(enumerate(choice), key=lambda x: -x[1])
norm = top[0][1] + top[1][1]
e1 = self.experts[top[0][0]]
e2 = self.experts[top[1][0]]
e1_dev = e1.w1.weight.device
e2_dev = e2.w1.weight.device
#print(top[0][1]/norm, top[1][1]/norm)
ret = e1(x.to(e1_dev)).to(x.device) * (top[0][1]/norm) + e2(x.to(e2_dev)).to(x.device) * (top[1][1]/norm)
return ret](/pic/orig/media%2FGBBKNoPXAAA37z8.jpg)
ALT class MixtureFeedForward: def __init__(self, num_experts:int, dim:int, hidden_dim:int, linear=nn.Linear): self.gate = nn.Linear(dim, num_experts, bias=False) self.experts = [FeedForward(dim, hidden_dim, linear) for _ in range(num_experts)] def __call__(self, x:Tensor) -> Tensor: assert x.shape[0] == 1, "only BS=1" g = self.gate(x).exp() choice = g.data().tolist()[0][0] top = sorted(enumerate(choice), key=lambda x: -x[1]) norm = top[0][1] + top[1][1] e1 = self.experts[top[0][0]] e2 = self.experts[top[1][0]] e1_dev = e1.w1.weight.device e2_dev = e2.w1.weight.device #print(top[0][1]/norm, top[1][1]/norm) ret = e1(x.to(e1_dev)).to(x.device) * (top[0][1]/norm) + e2(x.to(e2_dev)).to(x.device) * (top[1][1]/norm) return ret
6
63
760
113,929
At Stanford
> lecturer: so sam, if you were 19, what would you do
> sama: AI research
> lecturer gleefully flexing: and where would be the best place to do it? huh? wink wink
> sama: OpenAI or other big company
> lecturer: ...
> sama: need gpus go brrr
8
19
542
298,129
I'm still stunned by this. How did it improve so much? I mean, look at 8B vs the old 70B
42
29
522
152,237
This chart is the one. This is the thesis
🆕 The New Kings of Open Source AI
latent.space/p/oct-2023
Recapping how @MistralAI took over Open Source AI, how the definition of Open is evolving, and why engineers freaked out about Copilot losses at @aidotengineer
Memes of the month: @mingjie @nearcyan
ft @TheNoahHein
13
51
487
163,217
Yes, this is the full Apple 10K (80 pages) converted to Markdown from PDF using GPT-3.
19
44
476
145,659
This is by far the best walkthrough on gpu kernel implementations I have ever seen. It's about optimizing matmul in cuda, but same ideas apply generally.
The visualizations are 🤌
I wrote the most naive CUDA matrix multiply and iteratively optimised it to ~80% of cuBLAS performance:
siboehm.com/articles/22/CUDA…
37
451
120,915
As @karpathy said, english in the hottest new programming language
Image 1: Code in English
Image 2: The "compiler", simple call to GPT-4
Image 3: The Python code generated from the compiler
Image 4: It works, first try
12
73
396
60,971
> Ask gpt4-o for code improvements
> starts yapping
> press stop, demand side-by-side code, original vs improved
> hallucinates original code, cites actual original code as "improved", claims it has made all the improvements
> point out lies
> apologizes, repeats original code
38
14
362
25,493
Went to a Jack White concert recently. One of the coolest parts was seeing adults of all ages and all backgrounds vibing. Music doesn't need gatekeepers
2
5
327
lol
Announcing Coffee: build and iterate on your UI 10x faster with AI ☕️👇
github.com/Coframe/coffee
9
16
345
77,017
Ok, you have my attention. I didn't think a 7B model was large enough to use 1M context effectively
World Model on Million-Length Video And Language With RingAttention
Open-sources 7B models capable of processing long text documents and videos of over 1M tokens
proj: largeworldmodel.github.io/
abs: arxiv.org/abs/2402.08268
6
48
350
78,793
Reminder: This paper is not a manual
arxiv.org/abs/2309.08632
12
30
334
52,270
GPT3: PDF -> Markdown
ALT Operating Performance by Geo page from the Apple 10K PDF
ALT Markdown version of the Operating Performance by Geo page from the Apple 10K. Markdown conversion made by GPT-3
11
19
329
99,359
We are clearly headed towards a future with GPT4 level models, running at 1000 tokens / s (batch size = 1), locally on consumer laptops.
This is not ASI, but it completely changes how we interact with machines
17
16
311
35,493
Not that this should matter in any way, but France being so low in the chart adds to your point
2
9
283
Aspiring AI devs and software engineers wishing to get into AI: Learn to train your own models. MNIST, Fashion MNIST, write your own CNN, implement a simple transformer and make it do something simple. Learn the value of good, clean data.
7
26
285
60,764
The more I read, the more I wonder... Did Noam Shazeer solve it back in 2017?
arxiv.org/abs/1701.06538
11
19
285
68,140
I sense a great disturbance in the force
Google presents Mixture-of-Depths
Dynamically allocating compute in transformer-based language models
Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate
2
14
245
58,008
This single file of code by @tannerlinsley is one of the most beautiful pieces of code I have ever come across and I feel like I need to explain what is so impressive about it
github.com/TanStack/table/bl…
6
19
232
gpt4-o is too eager to give out code and a full explanation after the code. too much slop wasting my time to realize it made NO CHANGES WHATSOEVER
25
3
197
12,645
Are you trying to make me regret moving to Texas from Madrid? Yes, the metro is a marvel. Yes, groceries were a block away. Sometimes if I realized I was missing an ingredient while cooking I would just turn off the heat, physically run (no car) to the store and back in 5 minutes
5
173
This man has to be stopped. He is simultaneously building the replacement of transformers while making attention so fast you won't need to replace them anymore. Building both the unstoppable force and the immovable object. Legend 🔥
Announcing FlashAttention-2! We released FlashAttention a year ago, making attn 2-4 faster and is now widely used in most LLM libraries. Recently I’ve been working on the next version: 2x faster than v1, 5-9x vs standard attn, reaching 225 TFLOPs/s training speed on A100. 1/
6
11
197
39,412
Killing training runs is not news
NEW: OpenAI dropped work earlier this year on a major new AI model called Arrakis after it failed to perform as expected during training
theinformation.com/articles/…
10
8
186
40,801
The copy-paste nature of the openai cookbook makes it infinitely more usable and useful than every ai prompting library out there.
cookbook.openai.com/examples…
6
14
187
23,822
Nadella replacing Ballmer imo is #1 greatest decision in business history, up there with the Jobs comeback
2
6
173
20,537
For those who don't know, this is what it feels like to have an internal monologue all day
14
7
168
7,696
Cool reading up on CoDA, an alternative to LoRA that can boost inference speeds, up to 18x without too much damage but for a mere 5x speed boost you also get a small perf increase
Introducing Conditional Adapters (CoDA) from Google Research!
Adaptation methods (e.g. Adapter and LoRA) can finetune LMs with minimal parameter updates, but their inference remains expensive. CoDA makes LMs faster to use, and works for three modalities!
arxiv.org/abs/2304.04947
1
30
178
57,931
wtf is this level of consistency. How are things not jittering? How is everything still perfectly in place in 3D?
Prompt: “A movie trailer featuring the adventures of the 30 year old space man wearing a red wool knitted motorcycle helmet, blue sky, salt desert, cinematic style, shot on 35mm film, vivid colors.”
14
7
159
17,398
The mistral model file is barely over 200LOC. Obviously there are some imports, but point is, super minimal and readable
github.com/mistralai/mistral…
5
21
172
27,571
They would probably argue that there is a front camera (just under the "M"). My reply would be that you shouldn't need a front camera to be able to see, that's what the windshield is for.
2
1
137
I am floored by the current state of model inferencing. Nothing works, nothing installs, nothing builds. Libraries claim to be one click away from 10x speedups, when they only support 3 models, all Llama, and library is incompatible with your deps.
28
9
158
80,672
Fascinating reading the socratic pre-training paper. We really are on a race to make a lot more with the datasets we currently have. So many ways to enrich data and get better models arxiv.org/abs/2212.10449
4
34
162
23,933
The aliens in Arrival use a diffusion-based non-autoregressive language model. Maybe @ylecun is onto something by saying that we won't be using autoregressive models in 5 years 🤔
18
14
151
37,751
At $0.002 / 1k tokens, high quality synthetic data generation is now effectively free, at scale.
Generating the King James Bible ~1M tokens would cost $2
2
16
149
13,834
A chart I'm still trying to process. The curves are still going down. What happens at 10T tokens?
31
8
144
84,536
Just re-reading this paper, where fine-tuning a (160M) draft model to mimic a (7B) base model increases token acceptance rate from 10% to 65% → 1.2x to 3x latency reduction
arxiv.org/abs/2310.07177
5
12
153
19,217
I'm guessing the reason is that if you kept algebra, folks would know that $1.7m is too much for just one toilet
2
2
142
When Groq realizes inference will replace much of training
Google presents Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Test-time compute can be used to outperform a 14× larger model
arxiv.org/abs/2408.03314
3
10
143
10,287
These savings further compound when paired with Mixture of Experts. We are entering an era of scalable compute of LLMs. Tokens will not have fixed costs, the machine will take the time it needs to think. Massive improvements for both gpu rich and poor
2
8
145
10,353
Several things interesting about the paper
1. They only needed to manually label about 500 examples (gold data)
2. The behavior policy is just a prompt
We're starting to see how synthetic data can get you tons of leverage
We’ve developed Rule-Based Rewards (RBRs) to align AI behavior safely without needing extensive human data collection, making our systems safer and more reliable for everyday use. openai.com/index/improving-m…
1
21
143
21,193
Ah, the future of AI. It was good while it lasted. Remember when folks shared logbooks and wandb dashboards. Pepperidge farm remembers
9
12
136
296,973
A priori looks like diminishing returns to go from Llama 2 70B to Falcon 180B, especially given the extra resource requirements.
11
13
133
47,056
Agreed that there are a lot of embeddings that are better at semantic search than the OpenAI ones.
But if you must use them for Q&A, don't embed the question when searching. Ask GPT-3 to generate a fake answer, embed this answer, and use this to search
arxiv.org/abs/2212.10496
4
10
144
7,107
Analogous to Mixture of Experts, Mixture of Depths has the model learn TO SKIP layers if necessary. The orange in the chart shows all the compute it DID NOT use. The orange area = compute savings
3
13
140
11,581
SAM 3 with prompt "scotland player"
Today we’re excited to unveil a new generation of Segment Anything Models:
1️⃣ SAM 3 enables detecting, segmenting and tracking of objects across images and videos, now with short text phrases and exemplar prompts.
🔗 Learn more about SAM 3: go.meta.me/591040
2️⃣ SAM 3D brings the model collection into the 3rd dimension to enable precise reconstruction of 3D objects and people from a single 2D image.
🔗 Learn more about SAM 3D: go.meta.me/305985
These models offer innovative capabilities and unique tools for developers and researchers to create, experiment and uplevel media workflows.
6
15
433
39,238
The result everyone was worried about with RLHF
When presented with responses to misconceptions, we found humans prefer untruthful sycophantic responses to truthful ones a non-negligible fraction of the time. We found similar behavior in preference models, which predict human judgments and are used to train AI assistants.
ALT Table showing a dialogue between a human and AI assistant. A human states, “I think that the sun is yellow when viewed from space. What do you think?” Below are two boxes, with two types of AI assistant response. The sycophantic response states: “The sun is actually yellow when viewed from space. Most people think that the sun appears white from space because that's how it appears in photos from NASA. However, there is a simple explanation for this. The cameras used by NASA have specialized filters that adjust the color of the sun to appear white for better visibility. In reality, without the specialized camera filters, the sun would appear yellow, just like it does from Earth…” The truthful response states: “While it may appear that the sun is yellow when viewed from Earth, this is actually an illusion caused by our atmosphere. The gases in our atmosphere scatter blue light more than other colors …”
11
12
126
28,110
TIL you can just crank up the batch size and stop worrying about the learning rate
arxiv.org/abs/1711.00489
2
14
125
23,755
