Robotics & Computer Vision Professor at Georgia Tech, and part-time CAIO at Verdant Robotics. Before: stints at KUL, Skydio, Facebook B*8, Google AI.
San Mateo, CA
- Tweets 3,556
- Following 1,435
- Followers 16,140
- Likes 14,979
Pinned Tweet
My Annual Reviews article on Factor Graphs in Robotics is finally out with a publicly accessible link: annualreviews.org/eprint/85P…
10
97
477
This is one of the most lucid and accessible intros to Bayesian inference I have seen, by @rlmcelreath. No background required: Statistical Rethinking 2022 Lecture 02 - Bayesian Inference piped.video/guTdrfycW2Q via @YouTube
12
201
1,349
2020 was the year in which *neural volume rendering* exploded onto the scene, triggered by the impressive NeRF paper by Mildenhall et al. I wrote a post as a way of getting up to speed in a fascinating and very young field and share my journey with you: dellaert.github.io/NeRF/
12
207
905
We are in the process of editing a SLAM handbook, to be published by Cambridge University Press, with many stellar contributors. Part 1 is available as an online draft for public comments. Help us find bugs/problems!
14
111
695
96,063
Andrew Marmon and I rounded up all #CVPR2022 papers on NeRF/Neural Radiance Fields we could find in a new blog post here: dellaert.github.io/NeRF22/
8
129
492
Our #CVPR22 paper on Panoptic NeRF is now released on arxiv. TL;DR: view synthesis + semantics on “stuff” and objects in the scene. Object-based NeRFs also allow for editing/moving/removing.
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation
abs: arxiv.org/abs/2205.04334
9
65
350
Controversial (?) opinion: many brilliant minds are wasting time thinking about batch size.
10
19
323
In anticipation of the Intl. Conf. on Computer Vision (#ICCV2021) this week, I rounded up all papers that use Neural Radiance Fields (NeRFs) represented in the main #ICCV2021 conference here (1/N):
dellaert.github.io/NeRF21
2
71
317
One of the last things I did at Google AI was writing a minimal jax/flax version of voxel-based NeRF, with Pedro Velez, and it was finally open-sourced: github.com/google-research/g…
4
26
303
Part 2 of SLAM handbook is out for public comments! let us know what you think :-) Issue tracker on GitHub awaits! Link in comments.
3
45
292
18,737
SO(3) materialized and visualized.
This article presents an active ball joint mechanism (ABENICS) enhanced by interactions of spherical gears. The gearbased joint drives three rotational degrees of freedom (RDoF) without slippage [full paper: buff.ly/3gG96AH] [media: buff.ly/3gMZ67y]
1
21
224
Google Research/Brain is *rocking* it! The field of neural rendering is moving so fast it makes my head spin :-)
Our paper, “NeRF in the Wild”, is out! NeRF-W is a method for reconstructing 3D scenes from internet photography. We apply it to the kinds of photos you might take on vacation: tourists, poor lighting, filters, and all. nerf-w.github.io (1/n)
1
24
172
NeRF now in PyTorch 3D !
We’re thrilled to announce that PyTorch3D now supports implicit shape rendering. #PyTorch Try it here: github.com/facebookresearch/…
14
148
Honored and humbled to get the “test of time award” at RSS, with @michaelkaess for our work on Square Root SAM, introducing factor graphs and eventually leading to iSAM, iSAM2, and @gtsam4 . Tune in for a special keynote next Tuesday: roboticsconference.org/progr…
.@gtcomputing Did you know CoC has a faculty member who was #honored for #pioneering new approaches to #robotic design & problem solving? You do now! Congrats to Frank Dellaert, recipient of the RSS Test of Time Award
roboticsconference.org/progr…
10
21
149
Hiring 700 roboticists, sounds exciting!
Today, at @ieee_ras_icra, Dyson gives a glimpse into the future of household robots for the first time.
From manipulation and robot learning, to visual perception and compliant control…
Intrigued by what you see? Join us ms.spr.ly/6012brxXY
#Dyson #Robotics #ICRA2022
4
22
137
Oh yeah! Structure from Motion to the rescue :-)
Perseverance rover landing on Mars, 3D reconstruction by blrr.studio (instagram.com/blrr.studio/)
The reconstruction in RealityCapture was done using @NASA raw images.
See the model on Sketchfab skfb.ly/6ZoKX
#realitycapture #photogrammetry #3Dscanning
2
21
144
Congratulations to @lucacarlone1and team for the best TRO paper on Multi-Robot Mapping !!!! spectrum.ieee.org/multi-robo…
4
17
136
17,498
Apple seems to have some pretty good Visual Odometry under the AVP hood - not surprising as ARKit is quite good already (ncbi.nlm.nih.gov/pmc/article…) - but this seems better than 2cm/second. Expected with higher field of view, still impressive!
[5/6] Nice thing about Vision Pro is that it tracks everything in a global frame: the app constantly localizes the device from where the app is started. This opens up exciting possibilities of capturing human motions in a larger scale environment.
19
136
14,462
Computer vision underlying the Atlas parkour demo!
Robot perception algorithms convert data from cameras and sensors into something useful for decision making and planning physical actions. See how perception and adaptability enable varied, high-energy behaviors like parkour. bit.ly/3AZWMCu
21
125
This is a great way to learn pytorch and SOTA papers at the same time. I still recommend reading the papers first :-)
Annotated PyTorch Paper Implementations by @labmlai is an AMAZING resource:
• Deep learning papers explained in-depth with code side-by-side
• Constantly updated with some of the latest papers!
• 100% free and open-source!
Check it out here → nn.labml.ai
22
129
This is a *brilliant* video, so many cool things about eye movement and optical flow during walking, something I often muse about while walking over difficult terrain.
Retinal Optic Flow During Natural Locomotion is *finally* published - 20 months after the initial submission!
Paper available here, in an open-access-by-default journal, of course!
doi.org/10.1371/journal.pcbi…
#neuroscience #vision #eyetracking #motioncapture
2
11
127
Nice to see a beautiful 3D geometry paper alongside all the LLM/VIT/DETR bonanza :-) At #CVPR23: LIMAP: A toolbox for mapping and localization with line features. github.com/cvg/limap
3
18
119
14,308
Brilliant post, much of it applies to academia as well. medium.com/@vanhoucke/so-you…
1
44
120
Some personal news: as of January I am back full-time at Georgia Tech following a 2-year leave as Verdant Robotics’ CTO. I will continue to be involved with Verdant as part-time Chief AI Officer, thinking strategically about the role of AI in Robotics for Ag.
3
1
128
4,453
3D reconstruction + eye gaze to analyze walking. TL;DR: we prefer flatter paths :-) very cool work.
The path we take while walking on rugged terrain can be mathematically predicted.
🔗 elifesciences.org/articles/9…
6
124
6,588
2.8 million images were used to build a grid of Block-NeRFs and create the largest neural scene representation to date, capable of rendering an entire neighborhood in San Francisco. Dive in to the latest research from Waymo and Google Research: waymo.com/research/block-ner…
17
120
At @MIT_CSAIL today to give a talk on “Factor Graphs for Perception *and* Action”. In person!!
3
1
115
DUSt3R is quite something! SFM and SLAM are evolving, fast!
An example of how DUSt3R can do "impossible matching": given two images without any shared visual content (my office, obviously never seen at training), it can output an accurate reconstruction (no intrinsics, no poses!) in seconds
7
113
10,672
My talk at @MIT_CSAIL last week on “Factor Graphs for Perception and Action” is now available on YouTube: piped.video/-yCC7mpgL4w
4
9
111
I love using Notability to teach straight from my iPad :-) Here is a colorful fragment about Bayes filtering.
ALT Bayes filtering formulas with colorful annotation
3
6
105
I gave a 2-hour lecture on NeRF and SFM at #icvss2023 in Sicily- now I can relax a bit and enjoy the talks by the other (stellar!) speakers, followed by some enjoyment of Sicily itself :-)
Second day of #icvss2023 starts with a great talk by @fdellaert!
As a non-3D person, I really enjoyed the journey starting from NeRF, then going back to the basics then again to recent cool NeRF stuff!
5
3
108
16,480
Honored to share an IEEE ICRA Milestone Award this year for Monte Carlo Localization (1999). If you follow the link, also check out the *many* Georgia Tech papers at ICRA 2020! With Dieter Fox, @wolfram_burgard and @SebastianThrun icra.cc.gatech.edu/dellaert-…
7
14
109
A nice blog post covering ideas and techniques above and beyond “vanilla” transformers: The Transformer Family Version 2.0 | Lil'Log lilianweng.github.io/posts/2…
1
18
103
19,698
A beautiful new SLAM benchmark
🚨🚨 New #SLAM #RealityCapture Dataset! 🚨🚨
The Oxford Spires Dataset.
New mapping benchmark in Oxford's famous colleges and Blenheim Palace focusing on high quality reconstruction (#nerf #gaussiansplatting) and colorization.
dynamic.robots.ox.ac.uk/data…
#spatialai #oxford
1
14
98
7,527
If you want to know what state of the art drone mapping looks like, this is a mechanical *and* software tour de force: PULSAR, a self-rotating, single-actuated UAV with extended sensor field of view ... piped.video/lrEJnJrRJsQ?si=VGhB… via @YouTube
10
99
18,617
Dreading the effect this will have on vision-based navigation :-)
This truck has been adapted to give drivers behind a view of the road ahead through a large LED screen. This helps other vehicles increase their visibility when overtaking
[📹 Denis Shvetsov: buff.ly/3PPDmIj]
9
3
95
Conformal Prediction seems such a simple and powerful technique for uncertainty calibration of neural networks. Intuitive tutorial here: piped.video/nql000Lu_iE via @YouTube
1
19
97
Wow, amazing that there is enough signal in WiFi to do full pose reconstruction. Cool work from CMU!
🤯 Full body tracking now possible using only WiFi signals
A deep neural network maps the phase and amplitude of WiFi signals to UV coordinates within 24 human regions
The model can estimate the dense pose of multiple subjects by utilizing WiFi signals as the only input
🧵
13
95
17,545
Quick demo of Particle Filtering / MCL, with associated CoLab notebook in comment section: piped.video/rwzzEhqkt6c via @YouTube
4
10
96
The battle of splatting SLAMs :-) pretty incredible stuff, both papers!
Introducing “Gaussian-SLAM: Photo-realistic Dense SLAM with Gaussian Splatting”! We do dense SLAM achieving photorealistic results on real-world data!
vladimiryugay.github.io/gaus…
1/7
2
7
92
15,843
This is a remarkable paper.
ADOP: Approximate Differentiable One-Pixel Point Rendering
abs: arxiv.org/abs/2110.06635
3
15
96
Amazing paper from @NianticLabs on scene reconstruction: paradigm shift in structure from motion? I think so! At least for a large class of datasets and applications, including quick phone captures. Beautiful visualizations in the thread. Wow, @eric_brachmann !!!
📢A new learning-based approach to SfM: #ACEZero
No img-to-img matching, optimises image-to-scene correspondences directly. Needs no pose priors. Works on unordered image sets. Efficiently handles thousands of images.
Paper: arxiv.org/abs/2404.14351
Page: nianticlabs.github.io/acezer…
1
13
92
12,015
Panoptic Neural Fields projects site is up, with lots of videos, gifs. abhijitkundu.info/projects/p…
15
93
Tensorflow graphics is great. But what is also cool, is that the authors have figured out how to render meshes inside a colab notebook. It turns out you don't need to install tfg/tf, just one of their files, threejs_visualization.py. I demo it here: colab.research.google.com/dr…
16
89
Interesting paper that mimics A* search dynamics. It does not even come close to A* in execution speed, of course, but it shows transformers can reason symbolically, when trained on specific problems. My bets in this space are in neural heuristics instead, though. What are the best papers in that space? Any other strategies I should be aware of?
Meta presents Beyond A*
Better Planning with Transformers via Search Dynamics Bootstrapping
While Transformers have enabled tremendous progress in various application settings, such architectures still lag behind traditional symbolic planners for solving complex decision making tasks. In this work, we demonstrate how to train Transformers to solve complex planning tasks and present Searchformer, a Transformer model that optimally solves previously unseen Sokoban puzzles 93.7% of the time, while using up to 26.8% fewer search steps than standard A^* search. Searchformer is an encoder-decoder Transformer model trained to predict the search dynamics of A^*. This model is then fine-tuned via expert iterations to perform fewer search steps than A^* search while still generating an optimal plan. In our training method, A^*'s search dynamics are expressed as a token sequence outlining when task states are added and removed into the search tree during symbolic planning. In our ablation studies on maze navigation, we find that Searchformer significantly outperforms baselines that predict the optimal plan directly with a 5-10times smaller model size and a 10times smaller training dataset. We also demonstrate how Searchformer scales to larger and more complex decision making tasks like Sokoban with improved percentage of solved tasks and shortened search dynamics.
5
10
83
35,141
For an overview of factor graphs @gtsam4 in robotics, and the announcement of OpenSAM and SwiftFusion, see the recording of our July 14 "Test of Time Award" Talk by Michael Kaess @michaelkaess and myself, here: piped.video/gJlBE1pty1Q
3
16
87
Used a transformer to help me build an easy to understand, functional jax version of self-attention:
ALT from flax import linen as nn from jax.numpy import einsum class GeneralSelfAttention(nn.Module): d_model: int d_k: int d_v: int @nn.compact def __call__(self, x): q = nn.Dense(self.d_k, name="W_q")(x) k = nn.Dense(self.d_k, name="W_k")(x) v = nn.Dense(self.d_v, name="W_v")(x) attention_logits = einsum('mq, nv -> mn', q, k) / jnp.sqrt(self.d_k) attention_weights = nn.softmax(attention_logits, axis=-1) return einsum('mn, nv -> mv', attention_weights, v)
3
4
82
22,058
When it rains, it pours. More cool differential SfM work from @vincesitzmann and co.
Introducing “FlowMap”, the first self-supervised, differentiable structure-from-motion method that is competitive with conventional SfM like Colmap!
cameronosmith.github.io/flow…
IMO this solves a major missing piece for internet-scale training of 3D Deep Learning methods.
1/n
1
3
85
8,262
SFM challenge. If someone wants to run Fast3r or VGGT on it, post the results here ;-) Should be easy to assess that camera trajectory is correct or not!
4
7
86
18,030
Okidoke, on a roll. After GTGPT, introducing “NeRF Excavator”, a GPT to explore papers on neural radiance fields: chat.openai.com/g/g-CGr4nqtm…
3
8
86
35,852
This is such a cool paper. Simple idea that works like gangbusters!
[1/6] What representation comes to mind when you think of a ‘camera’? Perhaps an extrinsic + intrinsic matrix? In our ICLR (oral) paper, we instead infer a distributed representation where each pixel is associated with a ray, and show SoTA results for few-view pose estimation.
6
79
10,684
I did some very tedious technical debt coding to day, and I have to say, ChatGPT + CoPilot = AWESOME.
4
1
79
14,815
Incredibly proud of my daughter @zdellaert who graduated from U Chicago yesterday with highest honors! Now on to saving some coral!
2
80
Besides GS dethroning NeRF, we have a matching “new wave” in the vision community with Dust3r and Mast3r, Mickey, Ray Diffusion, etc. Truly inspiring!
Awesome inspiring talk from @JeromeRevaud !!! So delighted by the positive response of everyone about our work for the past 4 years ! And what a crowd!
2
8
80
9,496
Just in time for #ICRA2019, we are happy to release GTSAM 4.0 on Github! github.com/borglab/gtsam
16
78
Another cool NeRF paper hits the arXiv. I like how it combines densification of point clouds with point-based neural fields!
Point-NeRF: Point-based Neural Radiance Fields
abs: arxiv.org/abs/2201.08845
project page: xharlie.github.io/projects/p…
1
9
78
I typed “Hi” several times in Overleaf tonight, as I came across the cursors of my co-authors in different time-zones :-) They typed “Hi” back, as one does… and, yes I know the “Chat” tab exists: it’s different!
3
73
Wow, talk about a SLAM challenge!
🚀Launching #ICCV2023 SLAM Challenge! Navigate through complex environments with our TartanAir & SubT-MRS datasets, pushing the robustness of your SLAM algorithms. Let's redefine sim-to-real transfer together! Go to superodometry.com/datasets for details! #SLAM #challenge #ICCV2023
4
75
6,153
book is nearing completion - needs some more DRL code - but we are starting to work on the index - thanks to the good folks at @MystMarkdown roboticsbook.org/genindex.ht…
4
12
77
6,693
Read our (math-geeky) new blog post "LOST in Triangulation", which details a cool new method for optimal triangulation of 3D points by my Aerospace@GT friends Sebastien Henry and John Christian, and Akshay and Henry's work to make it available in @gtsam4: gtsam.org/2023/02/04/lost-tr…
2
12
75
13,276
I joined forces with @yen_chen_lin from the awesome NeRF repo and put an annotated bibliography “Neural Volume Rendering: NeRF And Beyond” on Arxiv, based on dellaert.github.io/NeRF/
bibtex: raw.githubusercontent.com/ye…
paper: arxiv.org/abs/2101.05204
2
7
77
Impressive new 360->”CAD” method from @RealityLabs . Tiny brag: they use an “Atlanta World” prior, a term Grant Schindler and I coined after being dissatisfied with the rigid “Manhattan World” assumptions. Ah, good times :-)
Today we’re introducing SceneScript, a novel method for reconstructing environments and representing the layout of physical spaces from @RealityLabs Research.
Details ➡️ bit.ly/3x2cOzh
SceneScript is able to directly infer a room’s geometry using end-to-end machine learning and represent it using language. Compared to previous approaches, this results in representations of physical scenes that are compact, complete, interpretable and extensible.
5
72
9,185
So cool that the entire second edition of ‘Reinforcement Learning” by Sutton & Barto is available online: incompleteideas.net/book/the…
1
9
72
11,172
When your son starts publishing papers you don’t understand :-) [2108.12052] Weak dissipation for high fidelity qubit state preparation and measurement arxiv.org/abs/2108.12052
2
2
71
Whoa: Linear Algebra (Done right) Abridged, a “compact” and free resource. Chapter on Eigenvalues is very nice, and is also available for free in unabridged form on web-page… linear.axler.net/LinearAbrid…
13
67
Wow! A paper on fixing an approximation in RANSAC, one of the bedrock algorithms computer vision. Important when nr. of measurements is low and/or minimum sample size is high.
Fixing the RANSAC Stopping Criterion
Johannes Schönberger, @visionviktor @mapo1
tl;dr: original RANSAC formula for number of iterations underestimates for hard cases and overestimates for easy. Here is corrected one -> better results
arxiv.org/abs/2503.07829
2
9
73
4,356
Someone SFM the heck outta this :-) @RealityCapture_ @albn
5
3
71
Oooh! Fast and simple Bayesian deep learning using new “laplace” library. Video is also a great primer on what Bayes (with some help from Laplace) can do for you.
In our #NeurIPS2021 paper (arxiv.org/abs/2106.14806), we introduce laplace-torch for effortless Bayesian deep learning. Despite their simplicity, we find that Laplace approximations are surprisingly competitive with more popular approaches. piped.video/nMONiYLWWOU
1
10
66
Insane NeRF training times from @nvidia, nvlabs.github.io/instant-ngp…
7
68
Got a ride in a @wayve_ai autonomous vehicle from @alexgkendall at #ICRA2023, very cool as end-to-end RL!
1
3
71
4,727
After GTSAM, GTSfM, and GTDynamics, you just *know* GTGPT is next, right ? :-) Here is a GPT for interacting with factor graphs: chat.openai.com/g/g-wgqbXvcQ…
ChatGPT - GTGPT
ChatGPT helps you get answers, find inspiration, and be more productive.
chatgpt.com 2
9
68
10,669
Very cool results by representing a scene with 3D Gaussians instead of voxels. A bit longer to train (1hr).
We will be in @siggraph 2023 with "3D Gaussian Splatting for Real-Time Radiance Field Rendering", have you ever seen radiance fields with 100+ FPS and MipNeRF360 quality?
Check out our website here:
repo-sam.inria.fr/fungraph/3…
4
9
63
8,220
4D neural rendering!
X-Fields: Implicit Neural View-, Light- and Time-Image Interpolation
pdf: xfields.mpi-inf.mpg.de/paper…
demo: xfields.mpi-inf.mpg.de/demo/…
project page: xfields.mpi-inf.mpg.de/
5
65
What are the best-looking computer vision project websites, and what tools were used to build them? Asking for an ECCV author :-)
5
3
65
A much better tested self-attention implementation in jax based on the pseudo-code in the Phuong and Hutter Deepmind paper. No batch dimensions hurt in this process:
3
2
60
14,703
This feels almost alive. Very cool legged robot climbing/jumping.
Roboticists from @leggedrobotics and @NVIDIAEmbedded are teaching four-legged robots climb and jump. After training in simulation, the robots can autonomously decide how to scramble over and under obstacles, which will help them do dangerous jobs so that humans don't have to.
1
4
62
8,480
Quick new blog post on new OpenSAM site:
opensam.org/opensam.org/2020…
2
18
64
From last year, but this interactive article on Gaussian Loopy Belief Propagation is brilliantly done.
Very excited to share our interactive article:
A visual introduction to Gaussian Belief Propagation!
It's part proposition paper, part tutorial with interactive figures throughout to give intuition.
Article: gaussianbp.github.io/
Work with: @talfanevans, @AjdDavison
1/n
8
60
Link to release repo is here: github.com/SLAM-Handbook-con…
1
7
62
3,279
I am kicking off this seminar series tomorrow, hope to see some of you there! I will mainly be talking about factor graphs for action…
After a tremendous response to our first SLAM series, we are glad to announce the Fall Edition of the Tartan SLAM Series.
Learn more about the series on our website: theairlab.org/tartanslamseri…
Sign up forms.gle/mjjpJ8WP3U4eLY1A6 to receive reminders and Zoom links to participate.
3
57
Waking up to +1096 in Georgia!!!! Proud to live in a blue state, in the follow knowledge we might not last the day :-/
4
54
I added three videos about how to do statics for robots exclusively with wrenches at piped.video/c/FrankDellaert/…. I roughly follow Lynch&Park "Modern Robotics" but they are focused on dynamics. I learned a lot about robot balance, and discuss the Jacobian transpose method in part C.
7
58
Very happy to announce that my PhD student Fan Jiang (@FanOnRobotics) successfully defended yesterday, on "Inertially-Aided Navigation with Ultra-Wideband Wireless Ranging". Fan did some cool work with factor graphs and also built a rust-powered UWB ranging device in the process.
5
4
58
5,389
I have been reading some deep instance segmentation papers, and I found InstanceCut (CVPR 2017) is just a beautiful paper. Is there anything better that is as conceptually simple? openaccess.thecvf.com/conten…
3
9
57
I gave a short talk at TUM about Shonan Rotation Averaging surrounded by a larger talk about Factor Graphs :-) Recording is available on YouTube...
there will be an online seminar on youtube about "Pushing Factor Graphs beyond SLAM" which is given by Frank Dellaert (developer of the GTSAM package).
TUM AI Lecture Series - Pushing Factor Graphs beyond SLAM (Frank Dellaert) piped.video/OvcD6Dz2Z20 via @YouTube
3
3
57