Can we learn whole-body mobile manipulation directly from human demonstrations? Introducing Whole-Body Mobile Manipulation Interface (HoMMI) Egocentric + UMI, 0 teleop -> bimanual & whole-body manipulation, long-horizon navigation, active perception hommi-robot.github.io

Everything you love about generative models — now powered by real physics! Announcing the Genesis project — after a 24-month large-scale research collaboration involving over 20 research labs — a generative physics engine able to generate 4D dynamical worlds powered by a physics simulation platform designed for general-purpose robotics and physical AI applications. Genesis's physics engine is developed in pure Python, while being 10-80x faster than existing GPU-accelerated stacks like Isaac Gym and MJX. It delivers a simulation speed ~430,000 faster than in real-time, and takes only 26 seconds to train a robotic locomotion policy transferrable to the real world on a single RTX4090 (see tutorial: genesis-world.readthedocs.io…). The Genesis physics engine and simulation platform is fully open source at github.com/Genesis-Embodied-…. We'll gradually roll out access to our generative framework in the near future. Genesis implements a unified simulation framework all from scratch, integrating a wide spectrum of state-of-the-art physics solvers, allowing simulation of the whole physical world in a virtual realm with the highest realism. We aim to build a universal data engine that leverages an upper-level generative framework to autonomously create physical worlds, together with various modes of data, including environments, camera motions, robotic task proposals, reward functions, robot policies, character motions, fully interactive 3D scenes, open-world articulated assets, and more, aiming towards fully automated data generation for robotics, physical AI and other applications. Open Source Code: github.com/Genesis-Embodied-… Project webpage: genesis-embodied-ai.github.i… Documentation: genesis-world.readthedocs.io… 1/n

Can we collect robot data without any robots? Introducing Universal Manipulation Interface (UMI) An open-source $400 system from @Stanford designed to democratize robot data collection 0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)

Introducing GEN-0, our latest 10B+ foundation model for robots ⏱️ built on Harmonic Reasoning, new architecture that can think & act seamlessly 📈 strong scaling laws: more pretraining & model size = better 🌍 unprecedented corpus of 270,000+ hrs of dexterous data Read more 👇

Check out UMI! 3 things I learned in this project: 1. Wrist-mount cameras can be sufficient for challenging manipulation tasks with the right hardware design. 2. Cross-embodiment policy is possible with the right policy interface. 3. BC can generalize if the data is right.

Q-learning is not yet scalable seohong.me/blog/q-learning-i… I wrote a blog post about my thoughts on scalable RL algorithms. To be clear, I'm still highly optimistic about off-policy RL and Q-learning! I just think we haven't found the right solution yet (the post discusses why).

The @SloanFoundation picked five @Columbia scientists for research fellowships this year. Congrats @jcolinhill, @GKaragiorgi, @KaczaLab, @SongShuran + @henryquantum! bit.ly/33pMFfg @CUSEAS @ColumbiaQuantum @ColumbiaCompSci @NevisLabs

Can robots leverage their entire body to sense and interact with their environment, rather than just relying on a centralized camera and end-effector? Introducing RoboPanoptes, a robot system that achieves whole-body dexterity through whole-body vision. robopanoptes.github.io/

What if the form of visuomotor policy has been the bottleneck for robotic manipulation all along? Diffusion Policy achieves 46.9% improvement vs prior StoA on 11 tasks from 4 benchmarks + 4 real world tasks! (1/7) website : diffusion-policy.cs.columbia… paper: arxiv.org/abs/2303.04137

Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models abs: arxiv.org/abs/2207.11514 project page: semantic-abstraction.cs.colu…

Time to democratize humanoid robots! Introducing ToddlerBot, a low-cost ($6K), open-source humanoid for robotics and AI research. Watch two ToddlerBots seamlessly chain their loco-manipulation skills to collaborate in tidying up after a toy session. toddlerbot.github.io/

Big congrats to the winners (and the finalists!) of the #RSS2022 awards: - Best paper award: roboticsconference.org/progr… - Best systems paper award: roboticsconference.org/progr… - Best student paper award: roboticsconference.org/progr…

Congratulations to the 2021 Microsoft Research Faculty Fellows! This fellowship recognizes innovative, promising new faculty whose exceptional talent for innovation identifies them as emerging leaders in their fields. Learn about their research interests: aka.ms/AAcu80l

Congrats to our @ColumbiaCompSci Prof Shuran Song @SongShuran, who's won an @NSF CAREER award to enable #Robots to learn on their own and adapt to new environments. bit.ly/3smzRj2 @ColumbiaScience @Columbia

How to prevent behavior cloning policies from drifting OOD on long horizon manipulation tasks? Check out Latent Policy Barrier (LPB), a plug-and-play test-time optimization method that keeps BC policies in-distribution with no extra demo or fine-tuning: project-latentpolicybarrier.…

Can we automate task-specific mechanical design without task-specific training? Introducing Dynamics-Guided Diffusion Model for Robot Manipulator Design, a data-driven framework for generating manipulator geometry designs for given manipulation tasks. w. Huy Ha, @SongShuran

Today we're excited to share a glimpse of what we're building at Generalist. As a first step towards our mission of making general-purpose robots a reality, we're pushing the frontiers of what end-to-end AI models can achieve in the real world. Here's a preview of our early results in autonomous general-purpose dexterous capabilities – fast, reactive, smooth, precise, bi-manual coordinated sensorimotor control.

Here’s something you didn’t know LLMs can do – reconstruct articulated objects! Introducing Real2Code – our new real2sim approach that scalably reconstructs complex, multi-part articulated objects. arxiv.org/abs/2406.08474

When will robots help us with our household chores? TidyBot++ brings us closer to that future. Our new open-source mobile manipulator makes it more accessible and practical to do robot learning research outside the lab, in real homes!

Want a robot that learns household tasks by watching you? EquiBot is a ✨ generalizable and 🚰 data-efficient method for visuomotor policy learning, robust to changes in object shapes, lighting, and scene makeup, even from just 5 mins of human videos. 🧵↓

How do we unlock the full dexterity of robot hands with data, even beyond what teleoperation can achieve? DEXOP captures natural human manipulation with full-hand tactile & proprio sensing, plus direct force feedback to users, without needing a robot👉dex-op.github.io/

🔊 Audio signals contain rich information about daily interactions. Can our robots learn from videos with sound? Introducing ManiWAV, a robotic system that learns contact-rich manipulation skills from in-the-wild audio-visual data. See thread for more details (1/4) 👇

Can we collect robot dexterous hand data directly with human hand? Introducing DexUMI: 0 teleoperation and 0 re-targeting dexterous hand data collection system → autonomously complete precise, long-horizon and contact-rich tasks Project Page: dex-umi.github.io

Video generation is powerful but too slow for real-world robotic tasks. How can we enable both video and action generation while ensuring real-time policy inference? Check out our work on the Unified Video Action Model (UVA) to find out! unified-video-action-model.g… (1/7)

@SongShuran summarized the dynamics in many academic labs so well 😂

TidyBot++ is "batteries included" We've open sourced everything: • Hardware design • Low-level controller • Phone teleoperation interface • Policy learning pipeline Project page: tidybot2.github.io Docs: tidybot2.github.io/docs Get started building your own TidyBot++ today!

✈️🤖 What if an embodiment-agnostic visuomotor policy could adapt to diverse robot embodiments at inference with no fine-tuning? Introducing UMI-on-Air, a framework that brings embodiment-aware guidance to diffusion policies for precise, contact-rich aerial manipulation.

How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.

I’ve been training dogs since middle school. It’s about time I train robot dogs too 😛 Introducing, UMI on Legs, an approach for scaling manipulation skills on robot dogs🐶It can toss, push heavy weights, and make your ~existing~ visuo-motor policies mobile!

How do we learn motor skills directly in the real world? Think about learning to ride a bike—parents might be there to give you hands-on guidance.🚲 Can we apply this same idea to robots? Introducing Robot-Trains-Robot (RTR): a new framework for real-world humanoid learning.

Can robots learn to manipulate with both care and precision? Introducing Adaptive Compliance Policy, a framework to dynamically adjust robot compliance both spatially and temporally for given manipulation tasks from human demonstrations. Full detail at adaptive-compliance.github.i…

Humans can design tools to solve various real-world tasks, and so should embodied agents. We introduce PaperBot, a framework for learning to create and utilize paper-based tools directly in the real world. paperbot.cs.columbia.edu/

Introducing 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 -- Hardware! A low-cost, open-source, mobile manipulator. One of the most high-effort projects in my past 5yrs! Not possible without co-lead @zipengfu and @chelseabfinn. At the end, what's better than cooking yourself a meal with the 🤖🧑‍🍳

Normally, changing robot policy behavior means changing its weights or relying on a goal-conditioned policy. What if there was another way? Check out DynaGuide, a novel policy steering approach that works on any pretrained diffusion policy. dynaguide.github.io/ 🧵

Adaptive Compliance Policy just won the best paper award at the ICRA Contact-Rich Manipulation workshop! Huge thanks to the team and everyone who supported us at the workshop. adaptive-compliance.github.i… contact-rich.github.io/

After 18 months in stealth, dozens of prototypes, millions of real-home demonstrations, and one final all-nighter, we’re thrilled for you to say hello to Memo

It's honestly been such a dream of mine to combine my two passions: dancing and robotics.

We present🌊AquaBot🤖: a fully autonomous underwater manipulation system powered by visuomotor policies that can continue to improve through self-learning to perform tasks including object grasping, garbage sorting, and rescue retrieval. aquabot.cs.columbia.edu more details👇

TRI's latest Large Behavior Model (LBM) paper landed on arxiv last night! Check out our project website: toyotaresearchinstitute.gith… One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the technology, and to share a lot of details for how we're achieving it. piped.video/watch?v=BEXFnru5…

Also congratulations to @SongShuran -- delighted to be joining the list with her. :) technologyreview.com/innovat…

What if you could control new hand designs without a new policy? Introducing GET-Zero, an embodiment-aware policy that can zero-shot control a wide range of hand designs with a single set of network weights. get-zero-paper.github.io

Happy Valentines Day! 🌹 Enjoy a special Valentine's day themed policy (sound on!) from the AquaBot team 👬❤️🦾 Visit aquabot.cs.columbia.edu/ to learn more about our recent ICRA publication!

Your bimanual manipulators might need a Robot Neck 🤖🦒 Introducing Vision in Action: Learning Active Perception from Human Demonstrations ViA learns task-specific, active perceptual strategies—such as searching, tracking, and focusing—directly from human demos, enabling robust visuomotor policies under visual occlusions. 🧵👇

Can robots acquire real-world manipulation skills without real-world robot training data? Introducing Im2Flow2Act: — a framework that leverages object flow to bridge human video and simulation data, enabling scalable skill acquisition. Project Website: im-flow-act.github.io/

Weights drop ⚠️ We released our pre-trained model for the cup arrangement task trained on 1400 demos! We aim to enable anyone to deploy UMI on their robot to arrange any "espresso cup with saucer" they buy on Amazon. github.com/real-stanford/uni…

Everyone says they want general-purpose robots. We actually mean it — and we’ll make it weird, creative, and fun along the way 😎 Recruiting PhD students to work on Computer Vision and Robotics @umdcs for Fall 2026 in the beautiful city of Washington DC!

Cooking in kitchens is fun. BUT doing it collaboratively with two robots is even more satisfying! We introduce MOSAIC, a modular framework that coordinates multiple robots to closely collaborate and cook with humans via natural language interaction and a repository of skills.

We made a step-by-step video tutorial for building the UMI gripper! Please leave comments on @YouTube if you have any question piped.video/x3ko0v_xwpg

🤖 How can robot policies zero-shot generalize to any new environment and any new object? Introducing our new project: 🚀Data Scaling Laws in Imitation Learning for Robotic Manipulation🚀—bringing us closer to the dream of having robots work as waiters in hot pot restaurants! 🍲

ICRA2025で主著1件を発表します。 形状と固さを比較的自由に変更出来るグリッパーを提案しました。 ご興味ある方はぜひ。 Shunya Hara, Osamu Fukuda, Mitsuru Higashimori, “Juzu Type Gripper That Can Change Both Shape and Firmness”.

ToddlerBot 2.0 is released🥳! Now Toddy can also do cartwheels🤸! We have added so many features since our first release in February; see github.com/hshi74/toddlerbot… for more details. Threads🧵(1/n)

How can we put robotics on the same scaling trend as large language models while not compromising on rich low-level manipulation and control?

We recently launched umi-data.github.io as a community-driven effort to pool UMI-related data together. 🦾 If you are using a UMI-like system, please consider adding your data here. 🤩🤝 No dataset is too small; small data WILL add up!📈

Andy Zeng's trademark toss.

Introducing DexWild -- a scalable approach to diverse "in the wild" data collection for dexterous robotic hands! This data can be used to co-train policy for any downstream robotic hands on any body form factor (humanoids, AMR with arms, etc). 🚀🤖

Introducing Mobi-π: Mobilizing Your Robot Learning Policy. Our method: ✈️ enables flexible mobile skill chaining 🪶 without requiring additional policy training data 🏠 while scaling to unseen scenes 🧵↓

🤖 Can robots reason about their mistakes by reflecting on past experiences? (1/n) We introduce REFLECT, a framework that leverages Large Language Models for robot failure explanation and correction, based on a summary of multi-sensory data. See below for details and links👇

Really cool invited talk by @SongShuran - “Making Video Model Useful for Robots”

Excited to share our new work ReLIC, a framework for versatile loco-manipulation through flexible interlimb coordination. We combine reinforcement learning and model-based control to let robots dynamically assign limbs 🦾🦿 for manipulation or locomotion based on task demands.

Congratulations to #CoRL2021 best systems paper finalist, "FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding", Huy Ha, Shuran Song. openreview.net/forum?id=0QJe… #robotics #learning #award #research

Deformable objects are common in household, industrial and healthcare settings. Tracking them would unlock many applications in robotics, gen-AI, and AR. How? Check out MD-Splatting: a method for dense 3D tracking and dynamic novel view synthesis on deformable cloths. 1/6🧵

Hats off to @ToyotaResearch for exciting results with “Large Behavior Models” and Diffusion Policies: piped.video/watch?v=w-CGSQAO…

Spent just over an hour collecting UMI data in Stanford, sent it to @hgupt3 in CMU to run drone I've never seen before, and it just works 🤯 I was expecting to iterate on the data a few more times but embodiment guidance is so robust! umi-on-air.github.io/ to learn more!

🚨 🚨 Another new work showcasing bitter lesson 2.0 🚨 🚨 Introducing MOO: robot-moo.github.io We leverage vision-language models (VLMs) to allow robots to manipulate objects they've never interacted with, and in new environments, while learning end-to-end policies. 🧵

From our demo floor at AI@, check out Code as Policies at work. This helper robot is able to compute and execute a task given via natural language. Read more → goo.gle/3U5CmCg

#CVPR2022 call for volunteers is now up! cvpr2022.thecvf.com/call-vol…

RTR is not just for adaptation. We also designed a challenging "swing-up" task, training the policy entirely from scratch in the real world. The humanoid learned this motion with the help and reward from the robot arm in less than 20 minutes.

Excited to share our latest progress on legged manipulation with humanoids. We created a VR interface to remote control the Draco-3 robot 🤖, which cooks ramen for hungry graduate students at night. We can't wait for the day it will help us at home in the real world! #humanoid

Semantic abstraction -- give CLIP new 3D reasoning capabilities, so your robots can find that “ rapid test behind the Harry Potter book.” 😉 w. Huy Ha

We spent a lot of effort on the documentation and hope that people can easily reproduce our work (including hardware!). We disscussed our hardware choices and how we fixed all kinds of harware problems so you don't have to. Please check it out! github.com/real-stanford/umi…

TOMORROW: Spring 2022 GRASP SFI: Shuran Song, Shuran Song,(@SongShuran ) Columbia University, “The Reasonable Effectiveness of Dynamic Manipulation for Deformable Objects” 3/16 @ 3:00 - 4:00pm - Levine 512 & Zoom. See you there! grasp.upenn.edu/events/sprin…

RL gets specific to the robot it is trained on. Can a policy be trained to control many agents? Turns out, training (shared) policy for each motor instead of whole robot not only achieves SOTA at train but also transfers to unseen agents w/o fine-tuning! huangwl18.github.io/modular-…