Professor at EPFL. Une mathémaphysinformaticienne. Passionate mushroom hunter. Tamer of two little dragons.
Switzerland
- Tweets 1,425
- Following 461
- Followers 15,886
- Likes 4,306
This semester, I have a lovely classroom for my Machine Learning for Physicists lecture. Look at that blackboard!!! A special bonus to those who recognize the t-shirt ;).
70
269
3,871
433,599
“I’m not able to learn mathematics easily, I have to work. It takes a very long time and I have a terrible memory. I forget things. So I try to work, despite these handicaps, and the way I worked was trying to understand really well the simple things.”
newscientist.com/article/242…
12
390
1,995
132,866
Another tweak peek at how I introduce neural networks after explaining kernel regression as regression in feature spaces, giving the insightful case of random features, and then simply adding the features as the argument in the loss function to get the loss of a neural net.
33
120
1,701
296,152
I am reviewing applications to EPFL's computer science graduate school and I see that many of the applicants have experience as reviewers for top ML conferences. I wonder why we bother with peer review at all if a sizable part of it is done by undergraduate students!?
55
108
1,585
311,689
For the fans of "ML explained on a blackboard": Lecture 13/14 was on the attention layer and mechanism (inspired by the relation to inverse Potts model from arxiv.org/pdf/2304.07235.pdf), with a toy-transformer architecture counting the number of occurrences for the exercise.
12
182
1,574
159,708
Emergence in LLMs is a mystery. Emergence in physics is linked to phase transitions. We identify a phase transition between semantic and positional learning in a toy model of dot-product attention. Very excited about this one! arxiv.org/pdf/2402.03902.pdf
16
254
1,397
198,884
Let me share a lecture from my ML for Physics course at @EPFL_en, where I derive the Boltzmann machine from the maximum entropy principle and use it to motivate the attention mechanism and a simple transformer architecture. Enjoy!
piped.video/dwMhw2X8_TU piped.video/od_XiaCJzV0
12
162
1,158
88,937
Oh my god! Giorgio Parisi has the NOBEL! This is such amazing news for the whole community of physics of disordered systems, I can't find words!
7
54
755
Join the ride and let everyone know about leshouches2022.github.io/ with a unique set of technical lectures on theory of deep learning with a stat phys touch. Featuring @GiulioBiroli @ylecun @yasamanbb @Andrea__M @boazbaraktcs @SaraASolla @HSompolinsky @BachFrancis @marc_mezard
6
158
686
Ahead of print for my friends: A non-technical commentary arguing why "Understanding deep learning is also a job for physicists" in @NaturePhysics Enjoy: rdcu.be/b4p1m
9
135
541
The Nobel prize for @giorgioparisi seems to be a perfect excuse to announce the summer school we organize in July 2022 in Les Houches where statistical physics applied to understand machine learning will be at the centre of attention. leshouches2022.github.io/
13
118
510
This mini-review is based on Hugo Cui's PhD thesis: arxiv.org/abs/2409.13904 . My advice to him was: "Write something you would have loved to have when you started your PhD!" He did an outstanding job introducing the rich methods he developed. Enjoy and share widely!
5
64
449
35,285
Amazing lecture of @SebastienBubeck giving intuition on transformers and how to approach them theoretically. Clearest talk on transformers I have ever seen. At the Summer Research Institute @EPFL
11
39
433
Dear Twitter, what is a nice resource to learn about the diffusion model. I would like to tell my ML for physics class about it and I am looking for a simplest possible way to explain the key concept in say 30mins.
34
38
417
Entropy is defined as a functional of a probability distribution. In physics, that is most often the Boltzmann distribution describing the probability of every single configuration of a system of molecules. The Shannon entropy of that distribution is the physics entropy.
3
17
417
49,153
Have you ever asked: 'What is a basic model, analogous to linear regression, but for sequences of tokens?' Well, we did—and we came up with bilinear sequence regression and its long-sequence, high-embedding dimension analysis. arxiv.org/pdf/2410.18858. Happy to hear your thoughts!
6
42
404
48,041
I'm happy to share the recording from my Cornell Bethe lecture a few weeks back. It's a broad-public lecture on "Bridging Physics and Computer Science: Understanding Hard Problems." No technical background is needed. Enjoy! cornell.edu/video/lenka-zdeb…
1
56
404
55,094
I only learned today that @Nature picked my comment on "Understanding deep learning is also a job for physicists" into the Nobel Prize in Physics 2021 collection nature.com/collections/ejgje… Nice :). Finally explains some of the calls I got last month!
4
27
381
Uau .... I am speechless ...
BREAKING NEWS
The Royal Swedish Academy of Sciences has decided to award the 2024 #NobelPrize in Physics to John J. Hopfield and Geoffrey E. Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.”
12
19
377
68,536
Postdocs, come to work in EPFL Lausanne: not more expensive than Boston, better salary …. Fresh air, ski and sailing as a bonus :)!
New proposed postdoc salary scale in the Boston area, based on data from a salary and benefits survey
nature.com/articles/s41587-0…
18
19
346
132,851
I would like to think about this as an invitation that physics extends towards AI/ML to think like physicists: -"Let's figure out how the world works". This is unlike computer scientists, who often see worst cases and adversaries. Nor mathematicians who need theorems everywhere.
BREAKING NEWS
The Royal Swedish Academy of Sciences has decided to award the 2024 #NobelPrize in Physics to John J. Hopfield and Geoffrey E. Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.”
12
49
343
37,425
I heard that Uncertainty quantification in high-dimensional overparametrized neural nets is hard, partly because Bayesian training is tricky and resists theoretical analysis. Then I realized we can analyze it in quite a detail: arxiv.org/abs/2210.12760 to appear in #aistats23 .
7
47
337
44,581
I am proud to let you all know that statistical physics of learning and all the other cross-disciplinary science my group is advancing is moving to EPFL. I fell madly in love with you @EPFL, and we will do great things together :). Thank you for this amazing opportunity!
My warm welcome to Josie Hughes, @zdeborova, Sanidhya Kashyap and @KrzakalaF who are joining @EPFL_en as faculty members! I look forward to working with you.
And congratulations to Jürg Schiffmann, Gerardo Turcatti and @CarraraSandro for their promotion!
admin.ch/gov/en/start/docume…
30
24
309
On this day at 9h55 in 2003, I was listening to a talk by Florent Krzakala in Sardinia and thinking: "What a cheeky guy!" ... And this is us 20 years, two amazing kids and 111 coauthored papers later, celebrating in a ***Michelin. What a journey through life with @KrzakalaF !!!
16
7
311
48,964
Science is not a competition. Our goal is not to be better than others, our goal is to understand the world. Reaching understanding is more efficient together than competing. Discussion and confrontation of different theories is positive. Competition as a goal is harmful.
17
27
302
The journal Science called me today about covering the Nobel Prize. It is not that evident to explain replica symmetry breaking and ultrametricity in simple terms, but this is what came up:
science.org/content/article/…
1
47
299
Why the continuing travel ban to the US for Europeans? How could full vaccination + a negative test not be enough as a precaution? This is hurting cooperation and ties built over decades are vanishing. Why is this accepted silently? Why doesn't Europe reciprocate?
10
39
280
Feeling satisfied after having explained to my @EPFL class the Onsager solution for the 2D Ising model. More precisely the Kac-Ward version of it. Such a beautiful set of arguments!!! One of those that made me fall in love with statistical physics 20 years ago.
4
18
296
Give me your examples (going beyond the Ukrainian war) of when you experienced #Westplaining . My memorable one is from France where I arrived from Czechia and have been told by a French leftist friend that I cannot understand what communism is because I grew up in it.
17
27
277
If Elizabeth Gardner (en.m.wikipedia.org/wiki/Eliz…) had not passed away so prematurely, she might have been among the nominees for yesterday’s Nobel Prize in Physics. Let’s remember her pioneering contributions to the study of neural networks.
6
60
270
17,655
Two postdoc openings in my group in EPFL in Lausanne. Come to work with us to understand deep learning using statistical physics. More details: artax.karlin.mff.cuni.cz/~zd… Share with others to spread the word and inquire if interested.
3
124
291
After 15 years in a french speaking country, I found out today from my daughter that Nutella is masculine "le Nutella". How can this possibly be, I still can't believe it ....
25
7
277
It is very strange that while I consider myself to be rather knowledgeable and connected in ML, there are only ~5 people on this list whose work I actually know about. I may have missed a couple, but still. What does this mean?
23
15
273
134,020
Thrilled to share the recording from my Simons Foundation Presidential lecture given last Wednesday. What else than: Statistical Physics of Machine Learning :). piped.video/TLHYwbrhGJc?si=JCHn…
39
262
22,555
The long-awaited collection of lecture notes from the Summer School on Statistical Physics & Machine Learning, Les Houches 2022, is now published in JStatMech iopscience.iop.org/collectio… . I am particularly proud of the works the school inspired; see section 3 of the editorial.
2
52
255
44,811
Do you want to enjoy this view for a month while learning about Statistical Physics and Machine Learning from e.g. @ylecun, @trekkinglemon, @marc_mezard, @Andrea__M, @GiulioBiroli? Applications just opened, more info at leshouches2020.krzakala.org/ . Please help to spread the word.
6
81
254
Postdoc wanted: Dear Friends, please spread the word that I am looking for postdocs interested to explore the intersection between statistical physics and computer science or machine learning to join my lab in EPFL. Informal inquiries are welcome. epfl.ch/labs/spoc/open-posit…
3
115
245
Nobel: "Parisi's discoveries make it possible to understand and describe many different and apparently entirely random complex materials and phenomena, not only in physics but also in other, very different areas, such as mathematics, biology, neuroscience and machine learning."
2
39
237
Call for papers on Mathematical and scientific machine learning @MsmlConference 2022 is open msml22.github.io/. Deadline Feb. 28th. If you are excited about how ML is transforming science and vice versa and have some cool related work join the MSML community and submit :)!
2
71
238
“Understand really well the simple things” is the best description of how I think one should do Science!!!
3
19
221
14,684
The most awesome explanation of the renormalization group flow I have ever seen. The fixed point being “city on a river”. Given by Andrea Cavagna at the “More is different” conference honoring Phil Anderson, organized by Jean-Philippe Bouchaud.
4
24
225
Waking up with my big girl Alena turning 9 today and 5 papers accepted to #NeurIPS2020, not a bad Saturday morning. Both these would seem out of reach just 10 years ago. Being blessed having both Alena and a top group of amazing students and collaborators.
5
3
225
This will be a Spotlight at @NeurIPSConf . Excited to discuss about Emergence and how to understand it with many of you there!
Emergence in LLMs is a mystery. Emergence in physics is linked to phase transitions. We identify a phase transition between semantic and positional learning in a toy model of dot-product attention. Very excited about this one! arxiv.org/pdf/2402.03902.pdf
1
16
212
21,086
It has always been a very enjoyable adventure to explore mathematical problems without the constraints of rigour. Theoretical physics friends can surely relate. Curious to know that in an alternative world I could even get the Fields medal for that :)! #AlternativeFieldsMedals
4
22
198
Applications for the Les Houches school on statistical physics and machine learning are open: formulaires.univ-grenoble-al… Fingers crossed that the future of the field can meet in person in the Alps next July.
3
64
186
Dear 2nd referee of our ICML 2019 submission. We finally managed to answer your question about the relation between the Saad&Solla analysis of two-layer neural networks and the one referred to as mean-field/hydrodynamic limit. Please see our new paper: arxiv.org/abs/2202.00293
3
27
193
My lecture about applications of physics of disordered systems in computer science. Yesterday, 30min after I learned about the Nobel prize. I want to share this to let you see the excitement and as my tribute to @giorgioparisi :) tube.switch.ch/videos/wjSpyR… tube.switch.ch/videos/ObSbhw…
3
34
189
This is something I am really excited about. Who said that dynamics in sparse disordered systems is really hard to tackle analytically? It is not! One just needs to think about it backwards in time: The Backtracking Dynamical Cavity Method arxiv.org/pdf/2303.16536.pdf
4
22
190
28,999
Oh uau. It is really *great* to know that in France the ministry is proud of funding pseudoscience. Very *honourable* use of the taxpayer money.
La France est une grande nation scientifique: parmi les 10 publications scientifiques les plus citées au monde pendant cette crise, 2 étaient françaises. Grâce à #FranceRelance et à la #LoiRecherche, nous permettons un réinvestissement massif dans la recherche. #60MinutesBusiness
7
20
170
Dear Friends, several of you were asking about the recording of my AMS Josiah Willard Gibbs Lecture on "What physics teaches us about computation in high dimensions". It is now on youtube piped.video/watch?v=B1VKyRrH…
4
45
188
A few words on my personal experience and perception about being a woman in STEM. I tell my daughters that following their curiosity about how things work is among the best things there is in life and that the world needs people who do that. Add it to your good-night kiss today.
"I was good at maths and the communist system didn’t care if you were a boy or a girl," read Associate Professor Lenka Zdeborová's reflections for International Women's Day. actu.epfl.ch/news/in-my-educ… #IWD2021 @zdeborova @EPFL_en
2
27
187
My favorite talk at SURI @ICepfl today by G. Ben Arous explaining a rigorous generalization of the Saad & @SaraASolla analysis of SGD in two layer neural nets. Getting to fine details such as probability to learn a XOR like mixture of Gaussian with 4 hidden units being 3/23 :)!
4
18
177
Math is beautiful! The free convolution of a semicircle law and Marcheko-Pastur leads to the optimal test error when learning a target function corresponding to an extensive-width neural network with a quadratic activation. More in arxiv.org/abs/2408.03733
3
25
173
18,377
And it is here!! Recordings of the Les Houches 2022 have a YouTube channel: piped.video/channel/UCNvC8um… So far starting with lectures by Remi Monasson, @GiulioBiroli, and Nati Srebro. More to come! Share with the world.
2
50
169
I'm happy to share the recording of a lecture I gave in Prague in May 2025: "Towards Understanding of Scaling, Emergence and Transformers." slideslive.com/39043090/towa…
2
17
181
11,948
Did you think deep attention networks would be very hard to analyze theoretically? No more! Meet a model of deep dot-product attention with softmax that maps to a multi-index model, and theory provides its learning curve for high embedding dimensions. arxiv.org/pdf/2502.00901
3
27
176
10,839
A stellar lineup of papers on "Machine Learning and Statistical Physics" celebrates the field in a special issue of @JPhysA where most of the foundational work on the topic appeared in the 80s. A treat for those confined in these beautiful autumn days.
iopscience.iop.org/journal/1…
4
36
173
Back to Berkeley #SimonsLive @SimonsInstitute listening to @Yoshua_Bengio thoughts and work on AI safety.
1
9
177
14,814
In general, I do not like calling something toxic, but attributing science on which the whole world is working such as fusion to a single (in general US-based) university/lab is toxic. Just stop that. Science advances thanks to many really smart and dedicated people in the world.
Check out this article: cnn.com/2022/12/12/us/common… Nuclear fusion and gene editing are the two greatest breakthroughs in this century. Both are from Berkeley...
4
8
161
High-dimensional asymptotics via the replica method continued. This time applied to denoising autoencoders: arxiv.org/abs/2305.11041 Kudos to Hugo! I was surprised by how few theory works on denoising autoencoders we found. If you know of some we do not cite yet, please do reply.
3
26
164
30,276
Being at an institution with 60+ accepted papers at the @NeurIPSConf inspired us to organize a poster and spotlight session locally in Lausanne. Excited to see all those cool works and their authors in person, yeahhh! Check it out: epfl.ch/research/domains/cis…
11
172
Our theoretical exploration of the diffusion-based generative models continues, this time with arxiv.org/abs/2310.03575 where we manage to characterize, down to the constant, the performance of a high-dimensional generative model trained from a limited number of samples.
1
37
170
29,782
Happy to share the recording of my plenary talk at Cosyne 2025 two days ago. You will learn about the statistical physics approach, phase transitions in learning, transformers, sequence models, attention etc. piped.video/watch?v=PurZcssu…
3
18
176
16,086
While I contributed to this line of work; still, I’m quite sure that modern AI would exist even without Hopfield and Hinton. There is a huge community behind each influential concept, and while prizes are given to individuals, it is the communities that create the science.
The 2024 Nobel Prize in Physics went to Geoffrey Hinton (left) and John Hopfield for their work on the statistical physics of neural networks. Modern AI would not exist without their clever methods of studying and applying randomness. quantamagazine.org/the-stran…
3
7
173
12,325
Excited about our progress in characterizing The Computational Advantage of Depth in Learning with Neural Networks. Check out the number of samples that can be saved when GD runs on a multi-layer rather than on a two-layer neural network. arxiv.org/pdf/2502.13961
2
24
164
10,834
Dear Friends, please share broadly the announcement of the workshop on "Statistical Physics & Machine Learning Back Together Again" in Cargese 31.7.-12.8. 2023. A list of confirmed speakers and application form can be found at: cargese2023.github.io/ (deadline 31st March)
47
149
31,701
We wondered how sampling with flow-based, diffusion-based or autoregressive networks compares to basic Monte Carlo or Langevin sampling. This is what came out for two classes of probability distributions well understood in statistical physics:
arxiv.org/abs/2308.14085
2
29
152
18,416
And here is our review on "Machine learning and the physical sciences" with @gppcarleo Cirac @KyleCranmer @Laurent_Daudet Tishby Schuld Vogt-Marano Hoping this will be useful for many colleagues in the years to come arxiv.org/abs/1903.10563
60
157
We need more of *Science of Deep Learning* in the major ML conferences. This year’s @NeurIPSConf workshop @scifordl on this topic is just starting, and I hope it is NOT the last edition!!!
1
16
150
13,480
I argue that theorems should not be necessary in theoretical machine learning! Current theoretical ML is not more firmly established than theoretical physics. When mathematical rigour matters, we can talk about “mathematical ML” just as we talk about “mathematical physics".
2
17
142
16,466
2nd day of learning how to optimize spin glass energy from @Andrea__M in rainy Diablerets. What a beautiful construction of the optimal message passing algorithm.
3
7
138
Starting tomorrow: If you are around MIT join for the Simons Lectures Series 2024 math.mit.edu
2
14
140
15,601
The series of three Simons Lectures I gave in MIT Math last April is now on YouTube. If you want to hear about *Computation through the lens of spin glasses*, have a look: piped.video/watch?v=94XXGGND…
1
23
144
8,727
Consider a weak low-rank signal added to a random matrix, the components of the sum are observed through a non-linear function. The exact trade-off between the signal strengths and the non-linearity allowing signal recovery is a topic of our new paper arxiv.org/abs/2403.04234
28
135
15,447
Am I the only one astonished by the fact that the @NeurIPSConf paper checklist does not distinguish between "theoretical result" and "theorem"? The whole field of theoretical physics provides "theoretical results" without "theorems" or "proofs".
7
12
133
17,447
This week, I am visiting Cornell. Honoured to give the Bethe lecture series physics.cornell.edu/bethe-le…. This gave me an occasion to revisit the influence one of Bethe's papers had on how I view the interaction between physics and computer science. A sneak-peak below.
20
135
11,756
Uau, thanks for all the reactions. We do have scripts and recordings, but so far they are not publicly available. But I see that I should do something about that ....
6
3
125
18,538
#NeurIPS2019 made my day today!!! 3 acceptations out of 3 submissions, from this one oral presentation, another spotlight. Plus a bonus of one free registration for my refereeing work. And invitations to 3 NeurIPS workshops. Looking forward to one hell of a week in December :)!
4
5
131
Uncertainty estimation in high dimension is still an open problem. Quantifying how classical methods such as bootstrap, subsampling or jackknife fail is a step forward: arxiv.org/pdf/2402.13622.pdf
18
133
12,493
Less than one month left to submit your work to the @MsmlConference 2021 . If you believe machine learning will revolutionize science and that we need more understanding to achieve that, do consider submitting your ideas :)! And hopefully, see you in Lausanne next summer :).
3
40
126
Dear Twitter, what is the status of the lottery ticket hypothesis? There may exist a tractable way to train nns without overparametrization. But is there any reason to believe that finding lottery tickets tractably without first training the overparametrized net is the way to go?
10
11
132
99,328
A seemingly really simple question: How many samples are needed to learn a target function that is a random deep neural net with the width linearly proportional to the dimension? Not so simple!! Only a partial answer in our new paper: arxiv.org/abs/2302.00375
4
20
128
22,459
As an ML community, if we want a healthy review process, we need to educate the reviewers. As area chairs, now is the time to do so. There are many papers with sound ideas that got low scores based on wrong reasons. See below what I say to some of the reviewers in those cases:
2
12
125
18,598
Excited that our LLM Phase Transition work gets the attention of the community. Thanks a lot for this beautiful video: piped.video/wzKW4P4dg1o?si… via @YouTube
2
23
127
14,729
It may be useful to share that @PhysRevLett rejects our paper on the largest submatrix of a random matrix arxiv.org/pdf/2303.05237.pdf because it deems it "hard to understand for non-specialists" and to be merely “one more combinatorial problem to which the replica method is applied"
6
9
118
61,835
I find such comparisons misplaced. A company uses your work directly to make profit which in turn allows them to pay you a lot. Academia is a service to the society. I feel way more valuable as a civil servant than if I worked for someone’s profit with 10 times the salary!
19
9
116
Another class of ML papers that collect good review scores largely overclaims the implications and generality of what is actually proven and sweeps the underlying assumptions under the carpet. See below what I ask the referees for those:
3
13
120
40,201
The past two weeks were a blast in Cargèse at "Statistical Physics & Machine Learning Back Together Again" cargese2023.github.io/. Around 100 of the top people in the field, including the next generation, discussed a lot of great science. I will miss you guys. We will be back!
1
8
116
15,580
My student Lucas Clarte ventured into the world of conformal prediction. He developed a version of the approximate message-passing algorithm to speed conformal prediction up in a high-dimensional setting. Check it out: arxiv.org/abs/2410.16493
1
14
117
9,367
This cathegory was there since the 90s. During times CS regarded neural nets as computationally infeasible, hundreds of physicist worked on them. When large datasets and GPUs came, it started working. Boundaries of fields are ill-defined and embracing this leads to progress!!
2
23
116
11,416
A pretty amazing lineup of papers related to statistical physics & machine learning from 2018 #NeurIPS, #ICML & #ICLR in now collected in #JStatMech. See iopscience.iop.org/journal/1… and enjoy :)!
36
116
Our work on how models can take into account properties of realistic data yet remain solvable with statistical physics tools analyzing the learning curves of two-layer networks just appeared in @PhysRevX Enjoy :)!
1
14
114
Another one, I will never forget, is the director of my daughter's school in France when she was 7y old, not knowing my profession and noticing my Slavic accent telling me: "I do not know how it is in your country, but here we have high academic standards."
3
3
108
If you want to see what topics were discussed at the conference "Statistical Physics & Machine Learning: Moving forwards" in Cargese earlier this month, know that most of the slides are now online: cargese2025.github.io/
2
19
126
12,138
"From disordered systems toward a theory of deep learning" piped.video/watch?v=jzQdXdeh… is now online. My lecture from summer school in Cargese 2021 @IES_Cargese
22
111
1/3 I wish referees stopped asking "How is this study relevant to real-life problems?" The often-accepted answer I give is that the study solves a problem that is related to another problem about which other researchers care.
6
7
111
More mind blowing projects on physics of AI in the years to come. Thank you @SimonsFdn for the support!!!
Our new Simons Collaboration on the Physics of Learning and Neural Computation will employ and develop powerful tools from #physics, #math, computer science and theoretical #neuroscience to understand how large neural networks learn, compute, scale, reason and imagine: simonsfoundation.org/2025/08…
4
7
113
46,860
In the era of neural scaling laws and "emergent" capabilities of AI, why don't we see more plots like this one to make the points about the emergence and scaling better justified and founded?!
The Marchenko-Pastur law is the distribution limit of eigenvalues of covariance matrices when samples and size grow together. djalil.chafai.net/blog/2011/…
3
19
101
106,809