Robotics Research @Meta. Ex-Apple, PhD @berkeley_ai
San Francisco, CA
- Tweets 192
- Following 384
- Followers 1,972
- Likes 1,756
Pinned Tweet
Imitation learning has a data scarcity problem.
Introducing EgoDex from Apple, the largest and most diverse dataset of dexterous human manipulation to date — 829 hours of egocentric video + paired 3D hand poses across 194 tasks.
Now on arxiv: arxiv.org/abs/2505.11709 (1/4)
15
91
605
114,044
I’ve finished my PhD @berkeley_ai !
This wraps up 8 wonderful years and 3 degrees from @Berkeley_EECS, starting as a freshman undergrad in 2016. Thanks everyone and go bears! 🐻
I’ll still be in the Bay, I’m headed to Apple next as a Research Scientist - very excited about it!
31
4
324
38,877
Companies like @Waymo & @Amazon use remote humans to manage fleets of robots for self-driving & logistics. We introduce new formalism, algorithms, and open-source @NVIDIA Isaac Gym benchmarks for “Interactive Fleet Learning” (IFL).
w/ @ken_goldberg, @pabbeel, @berkeley_ai (1/8)
3
44
168
How can robots learn skills from many different teachers? Especially when they must succeed outside the training distribution?
Our #CoRL2023 paper tackles both multimodality and distribution shift with IIFL: Implicit Interactive Fleet Learning.
@berkeley_ai @UCBerkeley (1/7)
1
16
84
37,597
Tired of collecting interventions all day to train with DAgger?
Introducing IntervenGen from @NVIDIARobotics + @berkeley_ai. From just 10 corrective human interventions, IntervenGen generates 1000+ to cover broad robot mistake distributions 👇
arxiv.org/abs/2405.01472
🧵 1/
9
22
85
43,479
Thrilled and surprised to see @OpenAI cofounder and brilliant researcher @johnschulman2 reference my work in his highly anticipated keynote...! Appreciate the shoutout and thanks for the great talk! 😄 piped.video/watch?v=hhiLw5Q_…
2
5
67
9,612
Is your robot pestering you? Robots learning a new task should only ask humans for help when needed. Our new algorithm ThriftyDAgger will be presented at #CORL2021 from @AUTOLab_Cal @berkeley_ai
(1/8) Website w/ paper, code, videos: tinyurl.com/thriftydagger
2
14
55
In Seattle for the summer as a research scientist intern at the @nvidia Seattle Robotics Lab; very excited to continue my PhD research on scaling imitation learning w/ talented roboticists @AjayMandlekar @CaelanGarrett @ankurhandos and Dieter Fox. LMK if you’re in the city! ☀️
2
1
46
3,143
Between the AI researchers I already follow for work and the rest of Twitter posting AI hype every day (ironically, like a bot would), my feed has been unbearably homogeneous for months. Super low SNR and feels like closing a fridge dissatisfied. I miss the pre-chatGPT Twitter
2
3
28
5,731
Twitter’s anti-alcohol groupthink is the most bizarre out-of-touch-with-reality thing I see on this app
6
23
3,205
It was awesome to be a part of this!! X-embodiment Fleet Learning with an Avengers-level crossover of 173 roboticists in 34 labs around the world, and all data is open sourced!
RT-X: generalist AI models lead to 50% improvement over RT-1 and 3x improvement over RT-2, our previous best models. 🔥🥳🧵
Project website: robotics-transformer-x.githu…
2
23
3,315
Excited to release this today! The “bitter lesson” of LLMs has been that supervised learning at scale is all you need — IFL may be the recipe for scaling up supervised learning in robotics 🦾
Two BAIR blog posts in one week! 📈 New post today on robot fleet learning by @ryan_hoque :
bair.berkeley.edu/blog/2023/…
1
20
1,820
Seems like a good time to retweet this ;) Recent results are amazing but let's not conclude that robotics is solved... lots of great research remains to be done!
2
2
22
3,718
Please consider submitting your ML + robotics work to CoRL 2022! I'll be helping organize; hope to see you in New Zealand!
CoRL: 6th Conference on Robot Learning will be in Auckland, New Zealand, 14-18 Dec 2022 (where it'll be summertime ;):
@CoRL_Conf
Deadline 15 June: We welcome submissions addressing the theory and practice of machine learning for robots and automation.
robot-learning.org/
2
20
MimicGen is an exceptionally intuitive and effective way to scale up human data collection. Stop by the OOD workshop at CoRL tomorrow for a sneak peek of an extension of MimicGen I’ve been working on with @AjayMandlekar @CaelanGarrett @NVIDIAAI 👀
Tired of collecting demonstrations all day to train your robot?
Introducing MimicGen, an autonomous data generation system for robotics. Using just 200 human demos we generated a large multi-task dataset of 50K demos! #CoRL2023 #NVIDIAResearch 👇
mimicgen.github.io
🧵 1/
1
18
2,927
Delighted to share that Interactive Fleet Learning has been accepted for oral presentation at @corl_conf!! Looking forward to presenting our work at #CoRL2022 in New Zealand this December! openreview.net/forum?id=MoSC…
Companies like @Waymo & @Amazon use remote humans to manage fleets of robots for self-driving & logistics. We introduce new formalism, algorithms, and open-source @NVIDIA Isaac Gym benchmarks for “Interactive Fleet Learning” (IFL).
w/ @ken_goldberg, @pabbeel, @berkeley_ai (1/8)
2
19
Great to be in New Zealand for #CoRL2022 @corl_conf! 🤖 2 things: (1) stop by the “Learning from Humans” oral session tomorrow at 3:35 PM to catch my talk on fleet learning, (2) join the 1st ever *CoRL Puzzle Hunt* which @ashwinb96 @brthananjeyan and I have organized for you all!
2
17
The Berkeley Library has a proxy that lets you bypass most of the paywalls 🙂 guides.lib.berkeley.edu/ezpr…
1
17
An excerpt from Godel, Escher, Bach (1979, before NNs) — perhaps the “black box” (uninterpretable) aspect of deep neural nets, often cited as one of the cons, is actually unavoidable in any sufficiently powerful approximation of intelligence…!
1
14
1,661
We finally jumped on the LLM hype train in this project, applying PaLM to challenging robot search problems. Fun collaboration with @brian_ichter, one of the folks behind Google SayCan, led by @19kaushiks @satviks107 @RavenHuang4
Wouldn’t it be nice if ChatGPT could find your missing keys for you? Our latest research from @berkeley_ai + @GoogleAI suggests that robots can use large language models (LLMs) to find hidden objects faster. 🧵👇
1
16
1,824
ICYMI - gave a talk on our recent work w/ @berkeley_ai on interactive imitation learning @corl_conf in London last week! Check it out here: piped.video/zYDwnj_ghZM?t=6438
1
15
Well, I finally gave into FOMO and got on twitter, so prepare for a strange combination of work-related things and random musings masquerading as insight
2
14
There’s a great paper on this by @rtaori13 at Stanford: arxiv.org/abs/2209.03942
13
573
The world would be a better place if people kept *probability distributions* of their beliefs instead of absolute beliefs, and adjusted them over time proportional to evidence. Like 80% certainty for a political position, or 99.9% for a scientific law
2
11
Nice to see more work in scaling the num robots to num humans ratio in fleet learning!
In the last two years, large foundation models have proven capable of perceiving and reasoning about the world around us unlocking a key possibility for scaling robotics.
We introduce a AutoRT, a framework for orchestrating robotic agents in the wild using foundation models!
1
11
1,613
.@corl_conf was fantastic this year, thanks to all organizers and especially @minasliarokapis for hosting us! 🥝
10
987
@berkeley_ai+@GoogleAI collaboration incl. @19kaushiks, Shrey Aeron, Gabriel Deza, @adivganapathi, @almostsquare, @JohnnyChungLee, @andyzengtweets, Vincent Vanhoucke, @ken_goldberg
arXiv: arxiv.org/abs/2204.10297
Website with all data, code, models, logs: sites.google.com/berkeley.ed…
3
9
For those who couldn't make it all the way to New Zealand ;)... the video from my talk at CoRL on interactive fleet learning is now available on YouTube! piped.video/watch?v=USr_iICR…
Delighted to share that Interactive Fleet Learning has been accepted for oral presentation at @corl_conf!! Looking forward to presenting our work at #CoRL2022 in New Zealand this December! openreview.net/forum?id=MoSC…
9
2,536
This has been my favorite project so far in my PhD!! Great to collaborate w/ @Lawrence_Y_Chen, Satvik Sharma, @KDharmarajan123, @brthananjeyan, @pabbeel, @Ken_Goldberg @UCBerkeley
arXiv: arxiv.org/abs/2206.14349
Website: tinyurl.com/fleet-dagger (8/8)
1
2
9
We present the first systematic benchmarking of fabric manipulation algorithms on physical hardware, with 4 new learning algorithms and 4 baselines for T-shirt folding. All data collection, training, experiments conducted remotely—I’ve never seen the robot in person! (2/3)
1
2
7
We also execute 1000 real-world trials of physical block-pushing with a fleet of 4 ABB YuMi arms & 2 human supervisors performing teleoperation remotely over the Internet. (7/8)
1
1
6
We introduce the IFL Benchmark, an open-source Python benchmark and toolkit built on top of @NVIDIA Isaac Gym for rapid prototyping + standardized evaluation of IFL algorithms with 100s of robots in simulation. (4/8)
1
1
5
We propose Fleet-DAgger, a novel family of IFL algorithms including fleet adaptations of existing 1-robot, 1-human interactive learning algorithms like ThriftyDAgger and EnsembleDAgger. (5/8)
1
4
Douglas Hofstadter’s notion of a “strange loop”: a paradoxical loop where you think you’re making progress but somehow end up back at the start. He writes about how both Godel’s theorem and human consciousness are strange loops. Check out this video: piped.video/hQsnHkfs3sA
2
5
Robots make mistakes! Robotics companies use remote human supervision of robot fleets in a range of applications. Human interventions during task execution improve reliability + provide more data for the robots to improve w/ continual learning. (2/8)
1
1
5
Work done in collaboration with co-first author Gaurav Datta, Anrui Gu, Eugen Solowjow, @Ken_Goldberg @AUTOLab_Cal @berkeley_ai. See you in Atlanta! ✈️
arXiv: arxiv.org/abs/2306.15228
OpenReview: openreview.net/forum?id=VtUZ… (7/7)
2
538
A platform for remote driving was recognized as one of Time's Best Inventions of 2022 today... interactive fleet learning is the future!
Today, @TIME revealed its annual list of the Best Inventions, which features extraordinary innovations changing our lives. @PhantomAuto_ is proud to be included in #TIMEBestInventions of 2022 list for its Remote Operation Platform for Logistics.
phantm.co/3A2kQqc
5
Great work as usual from @priyasun_ and the team
We can tell our robots what we want them to do, but language can be underspecified. Goal images are worth 1,000 words, but can be overspecified.
Hand-drawn sketches are a happy medium for communicating goals to robots!
🤖✏️Introducing RT-Sketch: rt-sketch.github.io
🧵1/11
1
1
5
2,318
Great to collaborate with @AjayMandlekar @CaelanGarrett @Ken_Goldberg and Dieter Fox!
Paper: arxiv.org/abs/2405.01472
Website: sites.google.com/view/interv…
1
4
642
Key question: How can we optimally allocate M humans to N robots (M << N) for learning + execution? To our knowledge, we formalize multi-robot, multi-human interactive learning for the first time. (3/8)
1
1
4
Agree, but also something being purely fiat doesn’t mean it can be safely ignored—almost everything we consider to be true is a collective imagination (morality, Enlightenment ideals like the value of empirical science, the existence of corporations and nations, etc)
1
4
I feel like machines are robots that “graduated,” they do their jobs so well that we take their autonomy for granted and they don’t seem like robots anymore. What was once smart starts to look dumb. Something similar is happening with the Roomba right now.
1
2
204
Check out the HMC22 workshop tomorrow (in Paris or free online), where myself and @Ken_Goldberg will be presenting a keynote on our recent work and perspectives on human-robot collaboration for an interdisciplinary audience! algorithmicfutures.org/hmc22…
The program for our next workshop, #AFPLhmc22, is how live: algorithmicfutures.org/hmc22 -- and we have an amazing lineup of speakers, an interactive workshop on human-machine collaboration and #sustainability, and an interactive #art exhibit. Registration is open and free!
2
4
+1, I’d argue robotics (decision making under uncertainty) is much more in line with the AI problem than ML (statistical function approximation) is. ML is an approach to AI
4
In IFL Benchmark simulation environments with 100 robots & 10 humans, a novel algorithm outperforms baselines w.r.t. return on human effort, throughput, hard failures, & idle time. (6/8)
1
1
4
Idk, personally I think it’s kinda nice that our research doesn’t become obsolete every few months and we don’t have to train with a million GPUs
3
669
Highly recommend this book. A must-read for understanding the modern American zeitgeist and how to fix it
My book What's Our Problem? is now available.
The book introduces a new framework for thinking about our chaotic political environment. With 303 drawings, it's a toolbox for understanding our societies, our group dynamics, and our own minds.
Get it here: waitbutwhy.com/whatsourprobl…
3
884
It was a fun time. Locomotion and manipulation software was underwhelming (not new or SOTA), but impressive progress in a short time and may show results in the future if their warehouses are sufficiently structured. Good to see the industry and press interest in robotics
3
Exactly! Shameless plug — the closest analogue in academia to my knowledge is “interactive fleet learning”: arxiv.org/abs/2206.14349
3
130
The key insight of IIFL is a novel application of Jeffreys Divergence. Due to symmetry, the intractable partition function cancels out! I’m very excited about this technique: it can estimate uncertainty for *ANY* energy-based model (not just for interactive IL). (5/7)
1
2
368
Imitation learning is brittle outside the training distribution. Online interventions help but take significant human data and effort to show how to recover from all the possible mistakes a robot may make. Can we instead automatically generate them?
🧵 2/
1
2
426
It essentially allows authenticated users to establish a network connection with the robot server and starts a big observation-action loop. (Note it’s not publicly available at the moment) Setup/maintenance cost is imposed on the host in this model.
1
2
This research was supported by @NSF and developed in collaboration with @ashwinb96, @ENovoseller, @albertwilcoxiii, @Daniel_S_Brown, @Ken_Goldberg @UCBerkeley (8/8).
2
We build on NVIDIA MimicGen. By executing the robot policy during both *data collection* and *data generation,* the robot encounters novel mistake states, from which we can apply transformed recovery segments.
nitter.app/AjayMandlekar/status/1…
🧵 3/
Tired of collecting demonstrations all day to train your robot?
Introducing MimicGen, an autonomous data generation system for robotics. Using just 200 human demos we generated a large multi-task dataset of 50K demos! #CoRL2023 #NVIDIAResearch 👇
mimicgen.github.io
🧵 1/
1
2
523
We evaluate IntervenGen in contact-rich tasks and observe that it increases robustness up to 39x over existing baselines. IntervenGen with 10 source interventions outperforms a policy trained with 100 human interventions by 24%, with just 12% of the data collection effort.
🧵 4/
1
2
287
Case in point - almost every single tweet on my timeline today is a ChatGPT screenshot…
For those who say AI isn’t creative as it can’t have original thought—
Humans are no better. Creativity is the act of mimetic recombination. We combine the set of ideas we’re exposed to (“influences”) to form new ones
And like AI, humans often don’t realize when they’re copying
2
The policy adapted to dynamic pose changes in the environment without object tracking and was robust to both physical perturbations of the end effector as well as visual distractors in the scene.
🧵 7/
1
2
604
More evidence that intelligence is compression! arxiv.org/abs/2207.04630
1
2
230
It’s more the engineering for sure. Open world autonomous manipulation isn’t nearly reliable enough yet, our current paradigm is collect a bunch of data then train on it, but it’s not clear yet (1) where we will get all that data and (2) even if we do, if it’ll be 99.9% reliable
1
56
Scientific example: light is wave and particle (double slit experiment). Zen example: personal identity.
1
30
Simulation and physical robot experiments suggest that compared with baselines, ThriftyDAgger can both increase task success rate and reduce human supervision, both during training and during execution. (6/8)
1
1
