Associate Professor at MIT, SPARK Lab Director, Roboticist, interested in how machines see and understand the world (he/his/him)

Cambridge, MA
If you are applying for #gradschool and have last-minute questions about your application, I'm willing to offer office hours on zoom in early November. I do this with students at MIT and I want to make sure others get the same opportunity. write below if interested.
204
116
875
126,003
I just got tenure from #MIT! I feel fortunate & proud of the work done with many talented and awesome students, postdocs, and collaborators at the #MITSparkLab. I'm also grateful to my wife, family, friends, & mentors for their support. now I look forward to building what's next!
80
4
725
47,935
I've been promoted to Associate Professor at @MIT ! I learned so much during the last few years & look forward to keep pushing the state of the art with the awesome people in the #mitsparklab and our great collaborators!! a big THANK YOU to our collaborators/friends/supporters!
36
10
525
excited to release the course material (slides, lecture notes, coding+theory exercises, Unity-based simulator, ...) for the graduate course MIT 16.485 "Visual Navigation for Autonomous Vehicles" (VNAV), with topics spanning geometric vision, VIO, SLAM, 3D reconstruction, ...
10
91
451
Hello World! My wife Marina and I are so happy to welcome our twin daughters, Luna and Aurora Carlone, who were born on August 30, 2022! now let's get ready for some sleepless nights #baby #twins
35
293
If you are applying for #gradschool and have last-minute questions about your application, I'm willing to offer office hours on zoom this week. I do this with students I interact with at MIT and I want to make sure others get the same opportunity. write below if interested.
37
30
240
If you are applying for #gradschool and have last-minute questions about your application, I'm willing to offer office hours on zoom early next week. I do this with students I interact with at MIT and I want to make sure others get the same opportunity. write below if interested.
56
21
220
If you are applying for #gradschool and have last-minute questions about your application, I'm willing to offer office hours on zoom on November 2nd. I do this with students I interact with @ MIT and I want to make sure others get the same opportunity. write below if interested.
66
24
208
28,871
we just won an #Amazon Research Award for our work on real-time spatial perception systems and 3D scene graphs! The award will accelerate progress towards human-like perception for #robotics. check our recent work: piped.video/qZg2lSeTuvM, arxiv.org/abs/2201.13360 #mitSparkLab
3
17
185
#Khronos won the Outstanding Systems Paper Award at #RSS2024 !! very proud of Lukas, Marcus, and Yun! Khronos' code and paper are online: Video: piped.video/YsH6YIL5_kc?si=XXK6… Code: github.com/MIT-SPARK/Khronos Paper: arxiv.org/abs/2402.13817 #mitSparkLab #4DSceneUnderstanding
3
21
177
10,767
I'm ecstatic to announce that I'm one of the recipients of the RSS Early Career Award! Big congrats also to @leto__jean and Byron Boots! #mitSparkLab #robotics #robotPerception #RSS2020 #awards roboticsconference.org/progr…
14
7
168
with SegmentAnything+CLIP, we can build rich metric-semantic 3D maps for robotics. but what is the right granularity for objects & places in the map? is a backpack an object? or is the zipper on the backpack an object? piped.video/m-HJO10qhSQ [1/n]
2
24
162
20,265
this summer I decided to limit travel/meetings and had more time to think deeply and spend with my family. I also wrote a draft on how to use modern tools from statistics to get performance guarantees for outlier-robust perception in robotics & vision: arxiv.org/pdf/2208.10521.pdf
6
7
144
I've joined the editorial board of the International Journal of Robotics Research (IJRR) as a Senior Editor. I'm excited and honored to help reshape the future of IJRR, with a renewed commitment to providing informative and timely reviews. Send your best papers! #IJRR #robotics
7
2
139
9,161
I'm deeply honored and humbled to be a recipient of the NSF CAREER award! a big THANK YOU to #NSF and to the mentors, students, collaborators, friends, and family who have been supportive of #mitSparkLab!! The best is yet to come. #NSFCareer lucacarlone.mit.edu/
7
2
139
Active #SLAM (or more generally, active spatial perception) is a growing research area with new exciting opportunities powered by deep learning and metric-semantic mapping. This new paper surveys the state of the art and discusses open problems: arxiv.org/pdf/2207.00254.pdf
3
18
138
truly humbled to be selected as a #SloanFellow , joining an impressive list of fellows which includes 20 Nobel prize winners and 17 Fields Medal winners. I'm very grateful to the many people who have mentored and supported me along the way. And congrats to the other 2023 fellows!
We are delighted to announce the winners of this year’s Sloan Research Fellowship! These outstanding researchers are shining examples of innovation and impact—and we are thrilled to support them. Meet the winners here: sloan.org/fellowships/2023-F… 🎉 #SloanFellow #STEM #ScienceTwitter
13
2
131
17,689
our work on graduated non-convexity for robust geometric perception won the best paper award in Robotic Vision at ICRA! congrats to Heng, Pasquale, Vasileios! video: piped.video/F0AlzLA0DY8 paper: arxiv.org/abs/1909.08605 (picture courtesy of @pappasg69) #mitSparkLab #ICRA2020
2
13
128
cool work by Dominic Maggio and collaborators on real-time robot localization using Neural Radiance Fields (NeRF) with Monte Carlo Localization: piped.video/yIgl3kbpbXY (link to full paper in the video description) #mitSparkLab #NeRF #neuralFields #robotics #localization
4
18
128
Hydra has been accepted at #RSS2022!! Real-time 3D scene graphs with places, objects, rooms, etc.. amazing work by Nathan Hughes and Yun Chang! -- Is anybody interested in an open-source version? piped.video/qZg2lSeTuvM @RoboticsSciSys #mitsparklab #spatialAI #robotics
8
17
125
we are excited to release Hydra, a highly parallelized pipeline to build 3D scene graphs from sensor data in real-time. kudos to Nathan Hughes and Yun Chang for this amazing work. piped.video/qZg2lSeTuvM (includes link to paper) #mitSparkLab #mapping #sceneUnderstanding
5
20
122
This is one of the strongest papers I co-authored, with unique results on robust point cloud registration and a manifesto of certifiable perception: Paper: arxiv.org/abs/2001.07715 Code: github.mit.edu/SPARK/TEASER-… Video: piped.video/xib1RSUoeeQ Kudos to Hank and Jingnan! #mitSparkLab
2
27
120
Our "Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs" paper has been accepted on IJRR! The paper stresses the importance of high-level 3D scene understanding for robotics and the need to go beyond SLAM. arxiv.org/pdf/2101.06894.pdf piped.video/SWbofjhyPzI
2
14
104
if you are interested in working with me or other #MIT faculty, consider applying to these Postdoctoral Fellowship programs: - METEOR: computing.mit.edu/METEOR - PFPEE: engineering.mit.edu/the-mit-… best of luck! #postdoc #fellowships #mitSparkLab #mit
1
16
102
10,844
Our paper "Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-Time Systems" has been published in the Intl. Journal of Robotics Research (IJRR): journals.sagepub.com/doi/10.… In conjunction with the IJRR paper, we are pleased to announce ... [1/n]
1
16
99
8,366
HUGE!! we won the DARPA SubT Challenge Urban Circuit!! kudos to our great team CoSTAR led by JPL! #mitSparkLab #JPL #Caltech #MIT #KAIST #DARPA #SubTChallenge
5
13
101
Pre-release of part 1 of our handbook: from SLAM to Spatial Intelligence. Please provide comments and suggestions to make it better! and a big THANK YOU to our amazing contributors! #spatialAI #robotics #perception #slam #ComputerVision
We are in the process of editing a SLAM handbook, to be published by Cambridge University Press, with many stellar contributors. Part 1 is available as an online draft for public comments. Help us find bugs/problems!
1
10
96
6,403
This recent work shows how to build 3D scene graphs of arbitrary (including outdoor) environments without much training data: arxiv.org/pdf/2312.11713.pdf. Recipe: use LLMs to build an ontology of concepts, and Logic Tensor Networks to ground them into the real world. #mitsparklab
1
9
96
6,378
I had a blast organizing the MIT Robotics Seminars with @pulkitology last Fall! if you are interested, you can find the recordings at piped.video/c/MITRoboticsSem…. Recorded talks from last Fall: - Vladlen Koltun - piped.video/vNFTcD3QMn0 - Cynthia Sung - piped.video/tt8Hzjp7wJ8 [1/n]
3
10
91
7,905
how can you design more effective RL approaches for navigation? use high-level representations produced by modern perception pipelines! You can find our latest paper at arxiv.org/pdf/2108.01176.pdf, which uses 3D scene graphs for RL-based navigation. #mitSparkLab #graphNeuralNetworks
3
6
88
#AcademicTwitter: I'm looking for a strong postdoc who can help us advance Spatial Perception/AI for robotics. If you are interested (and have a background in 3D vision/SLAM/VIO/possibly UAVs) email me or share with anybody who might be interested. #mitSparkLab
1
37
87
Our paper on robust spatial perception is a finalist for the Best Paper Award in Robot Vision at ICRA 2020! The paper provides a general framework to extend newly developed global solvers for 3D vision to problems with outliers. #mitSparkLab piped.video/F0AlzLA0DY8
1
9
84
what's the next big research problem after SLAM? the design of spatial perception engines for high-level and real-time 3D scene understanding. learn more (paper accepted at RSS 2020): piped.video/watch?v=SWbofjhy… #mitSparkLab #perception #robotics #AI #mapping #SLAM #computerVision
19
83
I’m excited to be a recipient of an Amazon Research Award for the proposal "Real-time Spatial AI for Robotics" at MIT. #AmazonResearchAwards are great to boost robotics research -- check the @AmazonScience website: bit.ly/3IMH6Yl .. and congrats to the other recipients!
1
4
78
Today at 12:30pm EDT I'm going to talk about "The Future of Robot Perception: Recent Progress and Opportunities Beyond SLAM" at the #TartanSLAM seminar. If you are interested, tune in at: piped.video/j5g3efgdjRg
5
13
78
If you are applying for #gradschool and have last-minute questions about your application, I'll offer office hours on zoom on November 7, 12:00pm-1:30pm EST at mit.zoom.us/j/8900097908. bring your questions!
11
72
6,869
looks like Santa came early this year! #skydio
3
67
I'm having fun co-organizing the MIT Robotics Seminars with @pulkitology this year! Check out our amazing speakers from last spring (and before) and stay tuned for more cool talks in the Fall: piped.video/c/MITRoboticsSem… @MIT @MITAeroAstro @MITEECS @mitidss
2
4
61
3,515
Truly humbled to receive the AIAA Aeronautics and Astronautics Advising Award -- it's a blast to have the opportunity to work with such a cool and talented cohort of undergraduate students at #MIT!! #AIAA #mitSparkLab #advising
3
63
Kimera: from SLAM to Spatial Perception: arxiv.org/pdf/2101.06894.pdf. turns out that: (i) SLAM with meshes = pose graph optimization, (ii) 3D scene graphs are great for planning, (iii) Kimera runs on TX2, (iv) uHumans2 dataset! web.mit.edu/sparklab/dataset…. #mitSparkLab
13
65
great to see @Tesla's progress on world modeling and metric-semantic mapping for FSD. Still a long-way to go towards human-level spatial perception (e.g., persistency, high-level reasoning about context and relations, resolution, priors), but a huge step in the right direction!
3
6
63
𝟯𝗗 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗦𝗰𝗲𝗻𝗲 𝗚𝗿𝗮𝗽𝗵𝘀: 𝗔𝗰𝘁𝗶𝗼𝗻𝗮𝗯𝗹𝗲 𝗦𝗽𝗮𝘁𝗶𝗮𝗹 𝗣𝗲𝗿𝗰𝗲𝗽𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗣𝗹𝗮𝗰𝗲𝘀, 𝗢𝗯𝗷𝗲𝗰𝘁𝘀, 𝗮𝗻𝗱 𝗛𝘂𝗺𝗮𝗻𝘀 #mitSparkLab Video: piped.video/SWbofjhyPzI Paper: arxiv.org/abs/2002.06289
10
64
Join our #ICRA2022 workshop on "Robotic Perception and Mapping: Emerging Techniques" tomorrow! Ask questions about the future of robot perception to our speakers and panelists using: tinyurl.com/ropm2022
7
60
Wow! Our paper on “soft” drones is a best paper finalist in 3 categories at IROS 2021! Big congrats to Josh, Sam, and Nathan for this amazing work. Well deserved! iros2021.org/awards Paper: arxiv.org/pdf/2103.06465.pdf Video: piped.video/mqbj8mEyCdk #mitSparkLab
3
1
60
"Self-supervised Geometric Perception" accepted as oral at CVPR 2021!! great work by @hankyang94, Wei Dong, and Vladlen Koltun - preprint: arxiv.org/pdf/2103.03114.pdf #computerVision #MachinePerception
1
6
60
This new paper formalizes the need for metric-semantic hierarchical representations in large-scale robot perception and presents state-of-the-art algorithms to build #3DSceneGraphs in real-time: arxiv.org/pdf/2305.07154.pdf Video: piped.video/AEaBq2-FeY0 Code: github.com/MIT-SPARK/Hydra
2
16
59
11,366
we are organizing the workshop on "Robotic Perception and Mapping: Frontier Vision & Learning Techniques” at #iros2023 -- consider submitting papers, extended abstracts, posters, or videos. Deadline: Aug 20. sites.google.com/view/ropem/ #robotPerception #computerVision #deepLearning
11
59
3,528
our #KimeraMulti has been recognized with the T-RO best paper award! kudos to Yulun Tian and Yun Chang who have been leading this effort and to our ARL collaborator Carlos Nieto-Granda -- Open-access paper: ieeexplore.ieee.org/document… #spatialPerception #SLAM #multiRobot [1/2]
Team including IDSS core faculty Luca Carlone received the award for their paper “Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems.” bit.ly/3OOHCu9
1
3
59
7,693
TEASER has been accepted on the Transactions on Robotics!! many thanks to the reviewers for the great comments and big congrats to Yang Hank and Jingnan Shi! arxiv.org/pdf/2001.07715.pdf - This is just the teaser of our vision for #certifiablePerception! #mitSparkLab
3
61
In collaboration with Yun Chang, Yulun Tian, and Jonathan How, we have developed the first distributed #multiRobot #SLAM system that builds a metric-semantic 3D mesh of the environment in real-time. Paper: arxiv.org/pdf/2011.04087.pdf Video: piped.video/G8PktlQ82uw #mitsparklab
2
7
60
woohoo! our paper on “Control and Trajectory Optimization for Soft Aerial Manipulation” has just won the Track Best Paper award at the IEEE Aerospace Conference. big congrats to Josh Fishman!! stay tuned for more recent results! piped.video/NNpQxP0SPFk #mitSparkLab #softDrone
1
54
The @DARPA #SubTChallenge is over! Things did not go quite as planned but I'm very proud of the tremendous capabilities we developed and the amazing and talented people we met along the way! I learned a lot. what a ride!
3
1
53
At the end of a challenging semester, it was truly rewarding to attend many cool final presentations by the students in my graduate course "Visual Navigation for Autonomous Vehicles" at #MIT! I'm impressed by the resilience and talent of the students and TAs contributing to VNAV.
1
53
SPARK has 2 cool papers accepted at #ICRA2021: - ROBIN: a general tool to remove outliers in perception (arxiv.org/abs/2011.03659) - Kimera-Multi: a distributed multi-robot system for dense metric-semantic SLAM (arxiv.org/abs/2011.04087)#mitsparklab #robotperception #ComputerVision
6
49
We have open-sourced code and datasets for Kimera-Multi, our multi-robot metric-semantic SLAM system: Video: piped.video/7yYMRNMdKjY Code: github.com/MIT-SPARK/Kimera-… Datasets: github.com/MIT-SPARK/Kimera-… New experiments: arxiv.org/abs/2304.04362
1
9
53
6,286
Fast and certifiably robust point cloud registration with TEASER++. New demos and examples available here: github.com/MIT-SPARK/TEASER-…
9
50
how to communicate large maps under stringent communication constraints in multi-robot systems? Yun and Luca B. show it's possible to compress a 3D scene graph map while retaining useful information for navigation using "graph spanners". Video: piped.video/nKYXU5VC6A8
1
8
49
I'm ecstatic to see that our #RSS2021 paper is a Best Paper finalist! roboticsconference.org/progr… kudos to @jingnanshi and @hankyang94 ! check out our poster tomorrow at 10am EDT at #RSS2021. Paper and video: roboticsconference.org/progr… #misparklab
50
not a big fan of rankings, but I was quite amazed to see my name among the top 2% most-cited scientists across all fields in 2021, as reported in this study by @Stanford on @ElsevierConnect - Also glad to see many roboticists in the career/singleyr lists: elsevier.digitalcommonsdata.…
1
2
48
a cool feature of 3D scene graphs (as the ones produced by Hydra piped.video/qZg2lSeTuvM) is that they expose symbolic knowledge while grounding it into geometry. This allows interfacing with Large Language Models - early results: arxiv.org/pdf/2206.04585.pdf #mitSparkLab #LLM
3
5
47
happy to see I'm on the list of Outstanding Reviewers for #ICCV2021 (iccv2021.thecvf.com/outstand…)! I have been doing way too much reviewing this year, and it's great to get a token of appreciation once in a while. don't be reviewer #2!
50
It has been a challenging semester with hybrid teaching, 3 Master's theses, N papers, and much more. But what a great satisfaction: super-proud of Yun, Jingnan, Josh for their theses. This is the final race for the undergrad course 6.141/16.405: #robotics #autonomousdriving
2
3
49
for RL and GNN aficionados: in this preprint, we show how to feed 3D Scene Graphs to RL, and show that using high-level perceptual representations in RL leads to better performance in object search tasks. piped.video/watch?v=x4LM-g3-… #mitsparklab #RL #robotics
5
47
really enjoyed this talk from George Konidaris which is restating the central role of abstractions in robotics and AI. I think there is much more to unpack regarding the "perceptual abstractions", but I agree with his take on "abstractions hierarchies" piped.video/-hztAN4MdrA
5
46
I'm offering Office Hours on Zoom to answer questions about grad school applications on November 2nd, 4:30pm EST. If interested, register here to join the meeting: mit.zoom.us/meeting/register… #gradSchool #robotics #computerVision
7
45
5,493
very excited about this work from the #mitSparkLab on task-driven open-set 3d mapping!!
Helping robots zero in on the objects that matter. LIDS PI @lucacarlone1 and collaborators have developed a new method called Clio that enables robots to quickly map a scene and to make intuitive, task-relevant decisions. Learn more and read their paper: bit.ly/3Bxhnnq
1
2
43
3,346
what a sunny day - a dark cloud has been lifted and the sun is now shining bright. and we also have our first female Vice President - how cool is that?
40
Yesterday we had the autonomous racing competition for MIT 6.141/16.405 - for the first time it was virtual. Our incredible TAs setup a simulator in less than 5 weeks. So proud to see the determination, talent, and kindness of our students and TAs. piped.video/MItig9HHPyk
1
5
41
We're excited to announce the open-source release of #Khronos, our recent work on Spatio-temporal Metric-semantic SLAM! Video: piped.video/YsH6YIL5_kc?si=FuHu… Code: github.com/MIT-SPARK/Khronos Paper: arxiv.org/abs/2402.13817 [1/n]
1
8
42
3,360
Don't miss this SLAM Winter (or Summer?) School organized by our amazing colleagues at the University of Technology Sydney. July 5-9, 2021. Website: uts.edu.au/slam-winter-schoo…
9
39
2 papers accepted at @NeurIPSConf - one in collaboration with @davsca1. no bragging, just very proud of the work of my students and collaborators - @hankyang94, Francesco Milano, @antoniloq, @RosinolToni
1
4
39
very proud of my postdoc Hyungtae Lim for winning the Nothing Stands Still Challenge at #ICRA2024 in collaboration with Jingnan Shi, Yun Chang,& Lukas Schmid!! also great that the team winning 2nd place used github.com/MIT-SPARK/TEASER-… @hankyang94 #mitSparkLab @MITAeroAstro @mitidss
4
1
41
3,433
New paper on soft aerial manipulation from SPARK. Now you can hoard toilet paper from the safety of your apartment! (or do the right thing and relax by reading this paper arxiv.org/pdf/2004.04238.pdf). Full video: piped.video/NNpQxP0SPFk #mitSparkLab #MIT #drones #robots #toiletpaper
1
4
39
great guest lecture by John Leonard for MIT 16.485 Visual Navigation for Autonomous Vehicles piped.video/rm23cEvQNvE
10
41
we will record the MIT SPARK lab group meetings and release them on youtube every week. These are typically hands-on tutorials or seminars. This week, Jingnan gave a 2-part tutorial on our algorithms for 3D registration: piped.video/uwNdLdRozeA #mitSparkLab #robotics
8
38
humbling article about the reality of robotics for elderly care. tl;dr: robotics is hard
Japan wanted robots to help care for the elderly. What happened? trib.al/CdpvpSr
2
37
4,355
our work on Reinforcement Learning on 3D scene graphs has been accepted at #ICRA2022! we combine #RL + graph neural nets + modern hierarchical representations for #robotPerception and boost navigation performance! piped.video/watch?v=x4LM-g3-… arxiv.org/pdf/2108.01176.pdf #mitsparklab
2
4
38
cool update from Toni! I would argue that SLAM+LLM is already here (but in early stages, see arxiv.org/pdf/2209.05629.pdf), and definitely agree on diffusion models being at the horizon..
A small update on the history of SLAM, and its future :) Last year, NeRFs sparked a new and ongoing era of geo/photometric SLAM! What lies ahead? I think some exciting avenues will leverage LLMs, diffusion models, or use light fields as maps! Same problem, new tools🚀
1
2
36
missed the award ceremony at #ICRA2021 and hadn't realized I got an Outstanding Associate Editor Award! lots of reviewing this year, but I still think the review process is a great opportunity for intellectual growth and to give back to the community ieee-icra.org/files/ICRA2021…
37
The IEEE RAS Technical Committee on Computer & Robot Vision (@ieee_ras_crv) is excited to announce an open position for a "Student Representative” role, to support our activities (TC website: ieee-ras-crv.github.io/). Details below!
1
6
37
7,649
do you want to boost the robustness of your RANSAC/robust estimator? Check this preprint showing how to solve problems with >90% outliers using max clique and invariance: we prune outliers in milliseconds! arxiv.org/pdf/2011.03659.pdf #mitsparklab #robustPerception #computerVision
2
4
32
It was great to catch up and go for a hike with the cool people in the #mitsparklab after >1year of zoom meetings! with our new temporary mascot, Teddy! @hankyang94 @TalakRajat @jingnanshi
2
34
How can we make graph neural networks more expressive? check out our #NeurIPS2021 paper on Neural Trees. Presentation: Thursday, Dec 09, 11:30 AM - 01:00 PM (EST), Poster Session 6. neurips.cc/Conferences/2021/… Paper: proceedings.neurips.cc/paper… #mitsparklab
1
34