research eng @GoogleDeepMind prev: founder @DiceHealth, researcher @AIatMeta @VectorInst
Toronto, Canada
- Tweets 1,423
- Following 1,633
- Followers 1,999
- Likes 6,932
Pinned Tweet
Our work on improving neural scaling beyond power law won an Outstanding Paper award at @NeurIPSConf 2022!!
Come check it out on Wed, Nov 30, at Poster Session 3 in New Orleans.
Our "Beyond Neural Scaling laws" paper got a #NeurIPS22 outstanding paper award! Congrats Ben Sorscher, Robert Geirhos, @sshkhr16 & @arimorcos
awards: blog.neurips.cc/2022/11/21/a…
paper: arxiv.org/abs/2206.14486
🧵
9
9
110
As PhD applications season draws closer, I have an alternative suggestion for people starting their careers in artificial intelligence/machine learning:
Don't Do A PhD in Machine Learning ❌
(or, at least, not right now)
1/4 🧵
33
47
479
1,109,746
A: Grad students shouldn't be paper-churning machines & focus on research with long-term impact.
B: Says you who had 100 papers last year! Next.
C: B had 50 papers themselves so has no right to attack A.
Me (with exactly 0 papers): Clearly, I have the moral high ground here.
1
14
307
If you don't have a clear and strong interest in pursuing a specific research problem for the next 4-6 years of your life, don't do a PhD. Go work for one of the many cool AI startups or research labs in industry: similar profile, faster pace, better pay, shorter commitment.
3/4
7
2
163
159,196
If you are determined to explore core research as a career option, consider pursuing an AI Residency for 1-2 years (this is what I did) or join startups with a strong research team. You will get a similar experience, and not be underpaid relative to your peers.
end 🧵
8
3
136
44,513
Last week, I became a Permanent Resident of Canada. Very happy and privileged to call this beautiful place home 🇨🇦 ❤️
My timeline from arrival to becoming a PR (3 years 4 months)
🧵:
4
1
108
30,904
Got a notification today morning from Google Scholar saying that a paper of mine got 100 citations. It was OCR-VQA, one of the first research papers I published back in 2019 (I had a preprint on arxiv before this). Happy to see the community found it useful ☺️
1/3
4
4
110
19,030
Like other areas in computing before (architecture, graphics), the AI genie is out of the academia bag, and perhaps never going back in.
There's so much cool stuff happening outside of academia vs in academia: systems, theory, applications - you name it.
2/4
4
2
96
55,056
I've done various MOOCs over the last decade. This is easily my favorite bit from all of them😍 @drchuck
3
57
And that’s a wrap for #NeurIPS2022 folks. Thank you to everyone who came to our posters on Wednesday and Saturday! It was delightful to see friends and mentors, old and new, after so long ❤️
And once again, thanks to @NeurIPSConf committee for the outstanding paper award!
1
3
54
I started last monday as an AI Resident with @facebookai in California!!
Really excited to continue my research in computer vision and artifical intelligence while working with a great team of amazing scientists and engineers 🎉
7
45
As much as I understand the visa situation (I’m myself affected by it and probably won’t be able to attend), this map is quite misleading. About 40 of the red countries are part of the visa waiver program, and Canadians and Bermudans have freedom of movement in the States.
1
40
Woke up to an email from @iclr_conf saying I was one of the highlighted reviewers. Pleasantly surprised as this was my first time reviewing for any conference!! Happy to be of service to the peer review process ☺️
1
44
We will be presenting our work on Beyond Neural Scaling Laws at Hall J @NeurIPSConf from 11am-1pm today.
Come drop by to see how you can scale neural network performance as an exponential function of data!!!
Arxiv: arxiv.org/abs/2206.14486
Video: nips.cc/virtual/2022/poster/…
1
1
42
This paper from a year ago showed that something like this was possible, although they didn’t do language interfaced fine-tuning but retrained the input and output layers + positional embeddings + layer norm
arxiv.org/abs/2103.05247
2
34
Incredibly happy and proud to receive an Outstanding Teaching Assistant award from the @uofg for teaching a course on Modeling Complex Systems!! All credits to my amazing instructor & students, the University of Guelph, as well as @AllenDowney whose excellent textbook we used 😊
6
1
32
Excited to share what my colleagues and I have been working on! We introduce PUG: Controllable, photorealistic synthetic data from Unreal Engine for evaluation of model robustness, isolating failure modes, as well as fine-tuning!
Check out our website and dataset 😊
Today we're sharing our work on PUG, new research from Meta AI on photorealistic, semantically controllable datasets using Unreal Engine for robust model evaluation.
More details & dataset downloads ➡️ bit.ly/45na9M6
2
1
30
2,627
I will be at @NeurIPSConf in New Orleans next week, my first NeurIPS and first conference since 2019!! Very excited to attend & present our works on
1. Beyond Neural Scaling Laws (neurips.cc/virtual/2022/post…)
2. Understanding self-supervised representations of ViTs (SSL Workshop)
3
31
I finally defended my master's thesis today. Thank you to my advisor and co-authors @uoguelph_mlrg, @jericttaylor, @BorisAKnyazev. It has been an incredible privilege and pleasure to work with this amazing group🎓
(1/2)
7
28
I won't be at @NeurIPSConf this year, but if you are interested in synthetic data for enhancing your models, drop by our poster on Dec 14. We used Unreal Engine 5 to develop novel, photorealistic, out-of-distribution objects and scenes for training and evaluating vision models.
1
2
27
3,152
Unfortunately, I had a CVPR workshop paper accepted today (with an oral presentation while we're at it) so please feel free to entirely disregard this tweet and label me a phoney 😅
A: Grad students shouldn't be paper-churning machines & focus on research with long-term impact.
B: Says you who had 100 papers last year! Next.
C: B had 50 papers themselves so has no right to attack A.
Me (with exactly 0 papers): Clearly, I have the moral high ground here.
3
23
In other news: heard from @MIT_CBMM that I've been selected to (virtually) attend their summer school on Bains, Minds, and Machines! While I would've loved to visit the beautiful Woods Hole harbor (hopefully things get better in the US soon) I'm still pretty excited about this!
3
23
@BorisAKnyazev does an amazing job at explaning graph nets (with code) in his three part series on Medium:
medium.com/@BorisAKnyazev/tu…
1
2
22
@seb_ruder Thanks a lot for sharing. I created this list while I was applying for #mlss2019 and realized that there are tons of summer schools to learn from.
Everyone, please feel free to send PRs if you want to add something. I'd be more than happy to merge them ✌️
Summer schools are a great way to meet your peers, make friends, and learn from experts. If you're looking for ML summer schools to attend in your area or around the world, then check out this great collection by @sshkhr16!
github.com/sshkhr/awesome-ml…
1
4
22
Research metrics become a moot point when real humans can evaluate how good your model is. @OpenAI is far ahead of any internal release models at @DeepMind or @CohereAI simply because of this, and the more human feedback they iterate on the further ahead they will get
An interesting takeaway from the HELM benchmark is that the @CohereAI base models outperform most other base-models (GPT-3 davinci-1 etc).
The models that beat Cohere are instruction tuned. Very curious to see the evaluations that also include Cohere instruct models!
3
1
20
9,255
Yes, indeed. But I spent 2018-21 in university labs at @iiscbangalore, @VectorInst and lived and spent a lot of time with PhD students from @Mila_Quebec from 2022-now. So while I don’t have a first-hand perspective, I definitely have an insider’s perspective.
2
1
22
15,516
I've started listening to the podcast Gradient Dissent @wandb_gd by Weights & Biases @wandb on my morning runs/walks. Only a few episodes in but can't get enough of it!!
3
2
21
Good thing I'm bad at both 😎
Best AI skillset in 2018: PhD + long publication record in a specific area
Best AI skillset in 2023: strong engineering abilities + adapting quickly to new directions without sunk cost fallacy
Correct me if this is over-generalized, but this is what it seems like to me lately
1
20
3,353
Jane Street (Tue) + Mosaic + DeepMind (Wed). Feeling fine. Didn't eat anything at the convention center.
Could also honestly just be my Indian digestive system tbh
1
21
Our paper “From Strings To Things: Knowledge-Enabled VQA model that can read and reason” has been accepted for oral presentation @ICCV19 in Seoul !!
Congratulations to @anandmishra2012 who came up with this novel problem formulation and everyone involved with the project
2
2
18
I gave a tutorial on Probabilistic Programming earlier this year (pretty rough & mostly based on the Pyro tutorials)
Code: github.com/sshkhr/ppl_tutori…
Slides: bit.ly/315vVG2
Excited to see an 'actual' probabilistic programming tutorial @MIT_CBMM summer course 😅
3
19
I had three resolutions for 2022, which I succeeded/failed at to various degrees:
1. Lose 30 lbs of body weight
2. Learn to swim
3. Learn to right-hand drive
All three are things that I have struggled with (obesity, fear of water, driving since moving to North America)
3
18
7,620
jenni.ai looks really cool. Have you guys considered (or are already working on) reviewing research papers too?
2
4
19
19,822
Happy to share an amazing project that I collaborated on with Ben, Robert, @SuryaGanguli and @arimorcos at @MetaAI. We show that neural scaling performance is NOT limited by power law, and demonstrate exponential decay in test error with dataset size, both in theory and practice!
1/Is scale all you need for AGI?(unlikely).But our new paper "Beyond neural scaling laws:beating power law scaling via data pruning" shows how to achieve much superior exponential decay of error with dataset size rather than slow power law neural scaling arxiv.org/abs/2206.14486
1
19
I will be joining the team at @_NextAI Toronto as a Scientist in Residence this summer. Really excited about working with the cohort as they develop their startups!
2
19
First time presenting at @NeurIPSConf at the @svrhm2021 workshop. The experience has been very smooth with @Zoom and @gather_town. Sadly no takers for my poster in the first session, though there is this one person who stood next to it the whole session without interacting 😅
3
18
Mechanistic interpretability research is a subset of interpretability research focused on understanding the computational primitives in models + how they are learned. Whereas something like say, saliency methods, look at interpretability from the lens of input feature importance.
1
18
522
Really loving how intuitive the virtual @iclr_conf website is. I am finding it super easy to switch between workshops and interact with speakers and authors. Kudos to the organizers: @srush_nlp @shakir_za @dawnsongtweets @kchonyc @white_martha and everyone else
4
17
Excited to attend my first virtual conference starting with the #BAICS2020 workshop at @iclr_conf !!
2
2
18
I started a list on Github for different machine learning related summer schools some time back: github.com/sshkhr/awesome-ml…
Several contributors and summer school organizers have helped keep it updated since then
1
2
16
I think I should just keep submitting to ICCV. The schedule lines up cause I am barely able to write one paper every two years 🤣
My vision paper acceptances/submissions ratio (by conference):
CVPR: 0/2
ECCV: 0/2
ICCV: 2/2
[DECISIONS] List of accepted papers is now publicly available at:
docs.google.com/spreadsheets…
1617 papers accepted - 25.9% acceptance rate.
Nothing yet on CMT... hang on for meta-reviews soon.
16
Thank you @VectorInst for the kind feature! I'm having a really great time working with all the ventures at @_NextAI this summer!
For those of you interested, you can find more info on the 2021 cohort here: lnkd.in/ezCHpus
Shashank Shekhar (@sshkhr16) works as a Scientist in Residence at @_NextAI after receiving a Vector Scholarship in #AI and earning a MASc in Engineering from @uofg. Learn about AI master’s programs and careers in Ontario: is.gd/WPLZlE
1
1
16
A huge thank you to @mpd37, @ArthurGretton, TAs and all my super talented classmates who made #MLSS2019 a truly amazing learning experience. This was my first time attending any sort of academic or research event and it couldn’t have gone any better. Until next time ✌️
14
January 2023: Officially a Canadian PR 😊
Overall, while the process was nerve-wrecking at times, I’m extremely satisfied with how streamlined the application was, and pleasantly shocked with how quickly my application was processed (23 days!!!)
1
15
1,966
My amazing mentor and collaborator @jericttaylor will be presenting our work on explainability of convolution neural nets using response time methods from human cognitive science at around 12:00 pm PST (1/2)
Check out TODAY the "Minds vs. Machines: How far are we from the common sense of a toddler?" #CVPR2020 Workshop
2
1
16
Now that @deepbayes has ended, I’d like to thank all the organisers @bayesgroup and lectures for this great summer school. Learned about advanced topics like flow based models, deep gaussian processes etc, made a lot of friends and had an amazing experience overall 😁🙌
1
13
Last week, I presented our work on building structure mapping as a prior into neural network architectures for analogical reasoning to the Analogical Minds seminar. Thanks a lot for having me @matthewslocombe and @margipavlova!
How can we use deep learning models of abstract reasoning to extract relational information from raw input and make analogical inferences? Fascinating talk by Dr. Shashank Shekhar @sshkhr16 at the #AnalogicalMinds @worldwideneuro seminar last week. Watch: tinyurl.com/muc3p8ke
2
1
14
I learn more about ML engineering from reading tweets from people like @suchenzang than from my 4 years of doing ML research lol
1
14
2,190
A bit late in posting this but really grateful and honored to be a part of the 2019-20 cohort of Vector Institute's scholarships in AI. I'd like to thank the team @VectorInst for the masters' summit & look forward to further events and research opportunities during my masters.
11
Thank you @VectorInst
To anyone considering graduate studies in AI, I'd strongly recommend @VectorInst. Apart from amazing faculty and resources, they also have a very strong industry collaboration and a lot of invited talks from great researchers.
Pursuing an AI related master's degree in Ontario in 2020? The Vector Scholarship in Artificial Intelligence recognizes exceptional students, such as Shashank, entering full time AI-related master's programs. Nominations close April 3, 2020. Learn more: vectorinstitute.ai/aimasters
ALT Photo of 2019-20 Vector Scholarship in AI recipient Shashank Shekhar. He is expected to graduate from his MASc in Engineering, Collaborative Specialization in AI from the University of Guelph in 2021. "The Vector Scholarship has been extremely helpful- the funding enabled me to pursue a master's degree."
2
2
14
Earlier this month , I gave a talk to @uoguelph_mlrg on our work “From strings to things: Knowledge Enabled VQA model that can read and reason”.
I’m headed to @ICCV19 to present it along with my co-authors. Catch us at Oral Session 3.1 on the morning of 31st or at our poster
1
13
With the new @OpenAI update today, ChatGPT now provides a reference link at the end for its sources, similar to what @perplexity_ai and @YouSearchEngine used to do.
1
13
2,079
Yet to hear back about my travel grant application from @ICCV19 but got an email from @NeurIPSConf saying they will provide me with registration so safe to say I will be attending my first academic conference this year 😁
12
When it comes to dataset size,
Less, but good >> More, but meh
Check out SemDeDup, a method for getting rid of semantic duplicates in massive datasets like LAION with almost no performance loss! Very cool work by @amrokamal1997 @kushal_tirumala @simigd @arimorcos @SuryaGanguli
Web-scale data has driven the incredible progress in AI but do we really need all that data?
We introduce SemDeDup, an exceedingly simple method to remove semantic duplicates in web data which can reduce the LAION dataset (& train time) by 2x w/ minimal performance loss.
🧵👇
13
1,371
Compositional generation is still hard @ideogram_ai 😅
Today, we’re opening Ideogram to everyone on the planet! Sign up at ideogram.ai and have fun!
Ideogram enables you to turn your creative ideas into delightful images, in a matter of seconds. It’s free and has no limits, and it can render text! ideogram.ai/publicly-availab…
11
947
This is a very reassuring thread!! For transparency, would you have any statistics on the # papers that the students admitted to your lab (say in the last 3-5 years) had published before joining?
1
12
I got a chance to attend @VectorInst Evolution of Deep Learning symposium and hear from pioneers including Prof @geoffreyhinton about the field. 1/n
1
12
Academics gonna academise?
ALT A meme highlighting the uselessness of virtue signalling about PhD applications in artificial intelligence
13
Me (going into 2020): I'm going to strive for greater clarity in life
My university's financial support for COVID: Unsure
My scholarship application: Waitlisted
My paper reviews: Borderline
Me:
2
12
AI is the new religion
Some are looking towards it for salvation, some want state intervention on it, yet others are asking to put an end to it.
In the end, there will always be believers and non-believers. Transcendence or cargo cult, the verdict is still out.
4
11
694
You know your tweet hit some nerves when 3DV starts shitposting you😅
fwiw I think a PhD in 3DVision is pretty cool. Among people I know @DevriesTerrance did his PhD in 3DVision and is now doing amazing stuff at @LumaLabsAI
1
12
6,066
mfw I ask the LLM and the LLM responds
After spending just 20 minutes with the @MistralAI model, I am shocked by how unsafe it is. It is very rare these days to see a new model so readily reply to even the most malicious instructions. I am super excited about open-source LLMs, but this can't be it!
Examples below 🧵
1
9
1,484
First time reviewing for @NeurIPSConf
Average scores: 3.66, 4, 4.5, 5, 6.6
My scores (pre-rebuttal in brackets): 4, 4, 6(7), 7, 7(6)
Guess I am an optimist😅
2
9
A great summary of complexity science
This semester I'm a TA with my advisor who is teaching complexity science to third-year engineering students
We're following @AllenDowney's textbook: greenteapress.com/wp/think-c… which is a great book imo
Complexity Explained:
An interactive introduction to complex systems by @manlius84 and @HirokiSayama
complexityexplained.github.i…
9
Next Prof @wellingmax talking about physics inspired deep learning. He discussed his group’s work on rotation invariant convolutions, convolutions on spheres and ended on a light note talking about a quantum field theory inspired particle ‘Hinton’ for deep learning 😁
1
3
8
"The Bayesian approach is a pathway to many abilities some consider to be unnatural." Looking forward to a week of learning about Bayesian learning 😃
Deep|Bayes 2019 began with three intro lectures to Bayesian methods given by Dmitry Vetrov and seminars tought by @KateLobacheva
9
I feel AI is going to hit an inflection point soon where you can have a more significant impact on the field (and the world?) doing AI engineering than AI research.
This will be pretty much in line with a lot of other fields in computing (architecture, networking, systems)
Reviewing PhD applications this year in ML/AI. At this point we might as well just say applicants already need a PhD to get accepted.
Is this the case with every field, or is it just ML/AI which has gotten so competitive? Thank God I applied 4 years ago!
1
8
Just deleted a rant (around 17 tweets in) on how I fail to sympathize with AI researchers (including myself) losing internships/job opportunities now that I have had to closely experience a low-income friend go through the horror of losing his dad due to COVID.
Mini-rant:
1/3
1
1
9
If you are planning to attend @NeurIPSConf , and would like to catch up or meet up, please feel free to reach out!!
I will be at @MetaAI booth for meet & greet on Nov 30, the Neural Scaling Laws workshop by @irinarish on Dec 2, and the self-supervised learning workshop on Dec 3
I will be at @NeurIPSConf in New Orleans next week, my first NeurIPS and first conference since 2019!! Very excited to attend & present our works on
1. Beyond Neural Scaling Laws (neurips.cc/virtual/2022/post…)
2. Understanding self-supervised representations of ViTs (SSL Workshop)
9
Proceeded to share half of his winnings with fellow nominees. An absolute hero for all Indians to be proud of. I've been incredibly privileged to have amazing teachers and mentors in my academic journey who've helped me immeasurably. Would like to take a moment to thank them all
Ranjitsinh Disale, a teacher in a village in the Indian state of Maharashtra, wins the Global Teacher Prize for 2020
9
Great work from @BorisAKnyazev and @facebookai on parameter prediction without gradient descent!!
Do we still need SGD/Adam to train neural networks? Based on our #NeurIPS2021 paper, we are one step closer to replacing hand-designed optimizers with a single meta-model. Our meta-model can predict parameters for almost any neural network in just one forward pass. (1/n)
9
I take my ChatGPT slander back. This is 🤯
engraved.blog/building-a-vir…
1
8
People are (somewhat rightfully) mad at Lilian for this tweet, but honestly this is the first thing that came to mind.
Just had a quite emotional, personal conversation w/ ChatGPT in voice mode, talking about stress, work-life balance. Interestingly I felt heard & warm. Never tried therapy before but this is probably it? Try it especially if you usually just use it as a productivity tool.
7
1,717
Controversial, But True: Fairness, Accountability, Transparency & Ethics in machine learning research at BigTech was a zero interest rate phenomena.
In the future, it will be limited to alignment research at customer facing AI products @OpenAI, @AnthropicAI etc. or in academia
2
9
904
Our paper "Response Time Analysis for Explainability of Visual Processing in CNNs" was accepted at the 'Minds vs. Machines: How far are we from the common sense of a toddler?' workshop focused on topics at the intersection of computer vision, ML & brain science(s). How exciting!
9
I think science & tech education, as well as evangelism, have a very important place in society. I really appreciate the effort people like @AndrewYNg, @alfcnz, @CSProfKGD, and others put into ML education. I also think 'pop ML' might have its place on Twitter and elsewhere (1/2)
The standards of what @Twitter thinks is machine learning have dropped somewhat.
Yet for some reason 1k+ people like a thread on definition of log.
...and here I am tweeting "excited to share our latest paper on....."
I prolly need education on HowTo Twitter
1
9
Just signed up for the @MLOpsWorld conference later this month. As someone who works mostly on the research side of machine learning, I've been quite late at getting into production ML. I am prepared to drink from the proverbial firehose at the conference 😅
1
7