Flat sales to 600% annual growth. Helping businesses enter new growth incl. w/ A.I Social networking pioneer: Ryze; Napster; Longevity; Math PhD
- Tweets 30,040
- Following 4,094
- Followers 4,061
- Likes 448,259
Biodefense is dramatically more urgent than any climate issue. #biodefense #climatechange npr.org/sections/health-shot…
5
7
126
In this video, I'll be deriving and coding Flash Attention from scratch. No prior knowledge of CUDA or Triton is required.
Link to the video: piped.video/zy8ChVd_oTM
All the code will be written in Python with Triton, but no prior knowledge of Triton is required. I'll also explain the CUDA programming model from zero.
I'll explore the following topics:
* Review of Multi-Head Attention
* Safe Softmax
* Online Softmax (with proof!)
* Introduction to GPUs and the CUDA programming model
* Tensor layouts: row-major layout, stride, reshape, transpose
* Block Matrix Multiplication
* Introduction to Triton
* Forward pass of Flash Attention in Triton
* How Autograd works
* What are derivatives, gradients, and Jacobians
* Jacobian of the Matrix Multiplication operation
* Jacobian of the Softmax operation
* Backwards pass of Flash Attention in Triton
* Triton tricks: Software pipelining
If you find this video useful, consider subscribing to my channel and sharing the video within your network of friends and colleagues.
#flashattention #triton #cuda #tutorial #python #attention #transformers #deeplearning
6
75
1,029
74,031
If your staff doesn't like serving customers, replace them, Will.
Don't fire your customers or your customers will all leave you.
Bud Light.
2
9
520
5,411
Biden admin funded a massive ngo human migration funnel through the Darien Gap in which many died along the path.
Completely coincidentally at the same time this occurred. Totally unrelated.
3
9
286
8,160
First of all, "undocumented people" are criminals. They are literally criminals and have broken the law. They should all be deported en masse.
Second. Due process. For illegal entry, the due process is immediate deportation. Not weeks of delays. Not court hearings. No wishy washy wishy washy.
Third: Scale. And I know you understand scale. During lefty admins, you bring in 25 million illegals (criminals), then you make it really hard to eject them, so you only remove them at a trickle. Meanwhile they tip the voting scales and you get Marxism & more judicial takeover. (which gives you more of the antisemitism you've already seen)
Four. Check out U.K. and Europe. It's not only Latinos that are coming in as illegals to the U.S. Look at history going back the last 1500+ years. Cultural collapse.
Five. If an illegal wants to contribute, they can self-deport and apply for legal immigration and convince the u.s. that they'll contribute.
That's for starters... cheers...
11
6
229
3,684
The real attack on the dollar comes from within.
Crypto is just a rounding error by comparison.
And Epstein didn't kill himself.
6
124
turns out the literal Matrix is just a bunch of rows and columns of numbers.
kinda like a spreadsheet, but without the formulas.
1
1
97
8,557
Talking VDF's and #chia 3 years ago with @bramcohen at my little party in SF. Good times. Excited by the progress since then and what the future hodls...
@chia_project #chiablockchain
2
7
84
fixed link: elpais.com/mexico/2025-06-26…
2
22
85
8,788
huh? maybe start first by legislating free speech, closing the border, and mass deportation of all illegals.
3
81
11,925
Who says #AppliedForecasters don't know how to party?
@enzolamberti @sergipino @SomaCapital
Part of our study group from @M_O_F_C course talking fat tail forecasting and extreme value theory
@spyrosmakrid @DrCirillo @Vspiliotis1 @Assimenos
we have booooks @nntaleb
1
9
62
Probably crossed the Darien Gap due to Biden admin funded migration wave, actually.
It's a nasty bug, but treatable. I've treated animals affected by it.
3
2
66
2,701
BEST COMMENT ON YOUTUBE CLIP:
"$300 Canadian buys you a sandwich and can of pop in Canada right now......"
2
57
23,659
Hay vacunados que mueren.
Los 'expertos' siguen equivocándose mientras que censuran otros científicos.
Cuando realmente vas a tener un Debate Abierto, Sabrina?
11
18
59
"I will be recording this call for quality assurance and reserve the right to publish it, do you agree? "
2
1
48
32,560
Here's what the parameters likely represent:
dim: The dimensionality of the model's layers or embeddings. A higher number typically means the model can capture more complex patterns.
n_layers: The number of layers in the neural network. More layers can allow the model to learn more complex relationships in the data.
head_dim: This is typically related to the size of "heads" in multi-head attention mechanisms, which are a part of transformer-based models. Each "head" can potentially focus on different parts of the input data.
hidden_dim: The size of the hidden layers within the model. Larger hidden layers can provide more capacity for the model to learn.
n_heads: This is related to the number of heads in the multi-head attention mechanism mentioned earlier. More heads allow the model to attend to information from different representation subspaces at different positions.
n_kv_heads: This might refer to a specific kind of head in the model that deals with key-value pairs in the attention mechanism, though this term is not standard across different models.
norm_eps: This is likely the epsilon parameter used for layer normalization, which helps stabilize the learning process by normalizing the input to each layer.
vocab_size: The size of the vocabulary that the model can recognize. This dictates the number of different tokens (like words or characters) the model understands.
moe: Stands for Mixture of Experts. This is a machine learning paradigm where several models (or "experts") are specialized in different parts of the data.
num_experts_per_tok: The number of experts that are used per token, suggesting that the model might be using a conditional computation approach where different experts are queried for different parts of the input.
num_experts: The total number of experts in the Mixture of Experts model.
These parameters suggest that this hypothetical "Mistral" model is a highly complex and potentially powerful LLM, perhaps designed for tasks that require a deep understanding of language, such as natural language processing, text generation, and language translation.
The specific configuration indicates a model with a large number of parameters (as suggested by the high hidden_dim and number of n_layers), which typically would require substantial computational resources to train and run. The use of a Mixture of Experts indicates an attempt to increase model capacity and efficiency by having different parts of the model specialize in different kinds of tasks or data.
5
5
43
17,854
He created the Solana blockchain, look up its market cap.
How did you get into Stanford and mit 😆😆
3
42
3,439
El método gallo es dividir la gente , tildandolos de 'antivacunas' cuando ya tienen muchas vacunas y apoyan a muchas vacunas. Es inmoral, falso y completamente Anti-ciencia argumentar como hace el Gallinazo.
Nadie pueda negar que no había consentimiento informado. /1
1
12
44
Should my team and I release something for Chia today?
🌱
Maybe some kind of app?
#chia #chianfts #NotanNFTproject #chiablockchain
8
3
38
2,876
2) La ciencia es de todos y no mas 'expertos' solamente... y MENOS para 'expertos' que mandaron al pueblo a no usar medidas preventativas como hizo la OMS y otras con la tema de las mascaras. #IntellectualYetIdiot ... Salud y paz con todos!
3
40
Forbes 30 under 30; Salesforce.com Women of the Future list; TEDx speaker
35
I would pay more for a Math PhD than a CS PhD, lol
Google recruitment ads are pretty funny too right now
1
39
5,691
Everyone who disagrees with me is a neo-nazi fascist in the pocket of putin, and should not be allowed to use the internet or publish their words anywhere...
Lol
34
1) Soy doctorado en ciencias naturales (math) y lo considero sumamente importante que el pueblo se involucra en criticar a las autoridades, especialmente a la OMS (criminalmente negligentes segun @nntaleb), y que se involucran en la resolucion y la ciencia.
1
4
36
Nadie puede negar que el OMS se equivocó muchas veces.
Nadie puede negar que el gobierno financiaba los medios para influir su contenido.
Nadie puede negar que el minsa amenazaba y presionaba a los médicos a no cuestionar.
Eso no es ciencia. 2/2
1
2
33
I built ProtonSign, a decentralized app for document signing in 2 days for @protonxpr... DocuSign for Blockchain 🚀🚀🚀 Excited 🔥#crypto #blockchain #reactjs
2
13
33
This is standard part of Latin American culture, e.g. the quinceanera celebrations are a rite of passage.
britannica.com/topic/quincea…
4
37
6,041
A stablecoin tied to the Japanese Yen needs to be created and called
the JPEG
Convince me I'm rong
#stablecoins #crypto
2
5
28
I have time to do a call with a @chia_project ecosystem project founder this week.
Just looking to help you think through your current biggest challenge
A crypto proj founder I did a call with wrote:
#chiablockchain #chia #chialisp #proofofspaceandtime
1
4
29
or as I call it, Chamath University
ChamathUniversity.com ;)
8
2,024
Today was my last day at X
I was the engineer responsible for keeping DM's working 24/7.
3
2
27
17,254
Fat tails teach us humility.
They're like a Godel's Incompleteness Theorem for finance.
(but different) @SomaCapital @paulportesi @normonics @nntaleb @WayneHimelsein @SuperMugatu #fattails #antifragile #godel #hedgefund
3
5
29
What if we did a quick Chia 1st birthday Zoom call tonight, 9pm EDT / 6pm PDT?
Will be nice to see smiling faces
@blaktron @xch_gallery @dylanmrose @chiatothemoon @MorganWBender @hoffmang @bramcohen #chia @gospaceport DM for cal invite @chia_project
us02web.zoom.us/j/8411176031…
12
6
28
About Saturday night... celebrating the first birthday of Chia @chia_project mainnet! A few photos, thank you all who attended.
Reserve your spot for the 2nd birthday party: forms.gle/eEF946LCP823H2Uc8
What if we did a quick Chia 1st birthday Zoom call tonight, 9pm EDT / 6pm PDT?
Will be nice to see smiling faces
@blaktron @xch_gallery @dylanmrose @chiatothemoon @MorganWBender @hoffmang @bramcohen #chia @gospaceport DM for cal invite @chia_project
us02web.zoom.us/j/8411176031…
3
4
27
Still remember decades ago praying to have #bitcoin, #ethereum, @EOS_io ... 😆@BrendanBlumer @RandyHilarski @jack_levangie @brockpierce @brotter #cryptocurrency #blockchain @majgeoinvesting
1
4
22
Throwback sunday, Labitconf 2017 in Colombia
ft. @peterktodd , Rolf from @horizenglobal
#labitconf #Bitcoin
1
4
20
Pretend you don't know that the whole goal of Alameda's trading
was to inflate the valuation of FTX
4
21
9,974
What if I told you...
Chia fixes this
#chia 🌱
Oh look FTX hosted all the NFTs minted on their platform using a web2 API and now all those NFTs have broken metadata and the links go to a restructuring website.
1
6
26
elections need to be verifiable; how do we really know the will of the electorate otherwise?
election integrity abuse is egregious at this point
5
1
23
2,804
A tiny contribution to the Horizen/Zencash blockchain project's new Sidechains SDK (installation guidelines/notes) github.com/ZencashOfficial/S… @horizenglobal @zencashofficial #crypto #cryptocurrency #blockchain #cryptodev
1
4
19
Here's what the parameters likely represent:
dim: The dimensionality of the model's layers or embeddings. A higher number typically means the model can capture more complex patterns.
n_layers: The number of layers in the neural network. More layers can allow the model to learn more complex relationships in the data.
head_dim: This is typically related to the size of "heads" in multi-head attention mechanisms, which are a part of transformer-based models. Each "head" can potentially focus on different parts of the input data.
hidden_dim: The size of the hidden layers within the model. Larger hidden layers can provide more capacity for the model to learn.
n_heads: This is related to the number of heads in the multi-head attention mechanism mentioned earlier. More heads allow the model to attend to information from different representation subspaces at different positions.
n_kv_heads: This might refer to a specific kind of head in the model that deals with key-value pairs in the attention mechanism, though this term is not standard across different models.
norm_eps: This is likely the epsilon parameter used for layer normalization, which helps stabilize the learning process by normalizing the input to each layer.
vocab_size: The size of the vocabulary that the model can recognize. This dictates the number of different tokens (like words or characters) the model understands.
moe: Stands for Mixture of Experts. This is a machine learning paradigm where several models (or "experts") are specialized in different parts of the data.
num_experts_per_tok: The number of experts that are used per token, suggesting that the model might be using a conditional computation approach where different experts are queried for different parts of the input.
num_experts: The total number of experts in the Mixture of Experts model.
These parameters suggest that this hypothetical "Mistral" model is a highly complex and potentially powerful LLM, perhaps designed for tasks that require a deep understanding of language, such as natural language processing, text generation, and language translation.
The specific configuration indicates a model with a large number of parameters (as suggested by the high hidden_dim and number of n_layers), which typically would require substantial computational resources to train and run. The use of a Mixture of Experts indicates an attempt to increase model capacity and efficiency by having different parts of the model specialize in different kinds of tasks or data.
2
19
7,964
Start with resolving DA situation as top priority.
Also undo the other stuff you promoted over the last 5 years, amigo
22
3,080
Always a pleasure seeing @Vit_Jedlicka and the @Liberland_org team, this time in Liberland itself on the boat Liberty! #liberland #crypto #cryptocurrency #blockchain #ico #bitcoin #liberty #decentralized #dao #freedom
2
2
19
Wow, this is a first for me
Just received notification of an offer for the @theNamesdao name Chia NFT I just registered
for about .5 XCH a day or two ago
Should I accept it?
Oh, and Happy 3rd mainnet Birthday, Chia!
1
1
21
2,090
Enjoy recruiting mediocre talent with that 2 year cliff. Bad mental model produces that idea, imho.
2
22
4,131
This is actually an important step forward.
People are now much more comfortable questioning narratives, realize how badly the centralized propaganda system has prepared them, and are eager to learn [even if they may tend toward presentations that support their biases]
1
21