ALT Voltron Framework – Balancing language conditioning and generation to shape visual representation learning.

ALT Our investigation of different axes for developing visually-conditioned language models (VLMs) spans four different axes. [Top Left] 1. Optimization procedure - should we freeze model components, or do we need multi-stage training? [Bottom Left] 2. Image processing and visual representations - how should we choose pretrained representations? [Top Right] 3. Language Models - how do base or instruct-tuned LMs affect performance? Does co-training on language-only data help? [Bottom Right] 4. Scaling Properties - are we undertraining our models? What type of added data helps?

ALT Learning by Decomposition: First, a user tried to complete a high-level task ("wash the coffee mug") with a model pre-trained on simple instructions (e.g. "go to the cup." "pick it up," etc.). Over the course of interaction, users use high-level language ("wash the coffee mug") that the robot doesn't understand, and have to simplify to get the robot to complete the task. After succeeded at a task, users "teach" these high-level instructions by decomposing them in terms of the low-level actions the robot was able to execute. This "interact, then decompose" framework is pretty general, and lets us update our system efficiently between tasks, so users can use what they just taught immediately, when completing a new task.

ALT Left panel: a robot manipulator trying to insert a book into a shelf, but getting stuck. Right panel: a human user identifying that the robot is stuck, and issuing a correction – "No, to the right!"

ALT Workshop on Learning from Diverse, Offline Data in NYC on June 27th, as part of Robotics: Science & Systems 2022.

ALT Visually Guided Latent Actions: We build models that leverage perception (via a learned object detector) to learn a low-dimensional structured "latent actions" for task-conditional control.

ALT #ChatGPT by @OpenAI Screenshot. User: Let's role play: You're the "brain" behind an assistive robot that may not be perfect, and I'm a human trying to work with you. We're going to try and make my life easier, and you're going to help me figure out what our physical robot can and cannot do. Please acknowledge that you understand, and describe who you are." ChatGPT: "I understand. I am Assistant, a large language model trained by OpenAI. I am not a physical robot, but I am here to help you understand what a hypothetical assistive robot might be capable of.... Please let me know if there is anything specific you would like to know about the capabilities of an assistive robot."

ALT Our incredible speakers for the L-DOD Workshop at RSS 2022. Left-to-right: Kristen Grauman, Abhinav Gupta, Cathy Wu, Benjamin Sapp, Eric Jang, Sergey Levine, and Davide Scaramuzza.

ALT Yonatan Bisk will be presenting his talk "Language Should be Embodied - But what does that mean?" this Thursday 11/19 at 10 AM PT for the Stanford NLP Seminar. The talk is open to the public: please register at nlp.stanford.edu/seminar.