Introducing Dynamism v1 (DYNA-1) by @DynaRobotics – the first robot foundation model built for round-the-clock, high-throughput dexterous autonomy. Here is a time-lapse video of our model autonomously folding 850+ napkins in a span of 24 hours with • 99.4% success rate — zero human intervention • 60% human throughput speed • 4.3/5 quality ratings (set by the client) A thread on our motivation, insights and results:

Introducing DrEureka🎓, our latest effort pushing the frontier of robot learning using LLMs! DrEureka uses LLMs to automatically design reward functions and tune physics parameters to enable sim-to-real robot learning. DrEureka can propose effective sim-to-real configurations for several robots and tasks, and we even got a bit creative with it: Let’s make a robot dog walk and balance on a yoga ball! Check out these fun videos, and follow the thread for a deep dive!

Excited to finally share Generative Value Learning (GVL), my @GoogleDeepMind project on extracting universal value functions from long-context VLMs via in-context learning! We discovered a simple method to generate zero-shot and few-shot values for 300+ robot tasks and 50+ datasets using SOTA VLMs like Gemini (Try out the demo on our website on your robot video today!) I worked a lot on leveraging foundation models as guidance for robots in my PhD, and to me, this result forges a new frontier in how we can use foundation models for robot learning, given its broad applicability independent of embodiment and task types. Quite excited about how we can build on this work as a community!

We just did World’s first on-stage autonomous demo of long-horizon dexterous VLA 🚨 No training. No setup. Performance out of the box. Live demo is hard and unpredictable, but we felt great about our model’s generalization, and it went pretty well! 💯 Zero-shot. 100% success.

We have raised $120M to accelerate our mission of building and delivering high-performance general-purpose robots to the physical world. Within one year, we have made research breakthroughs, showing that it is possible to achieve real-world reliability with large VLAs, and demonstrated commercial and deployment traction, with our DYNA-1 models running live in sites at SF, LA, and Sacramento. This is just the beginning, and I never felt more optimistic about a future where AI-powered robots can positively impact human productivity. From when I started PhD where robot policies barely worked even in highly controlled settings to now deploying DYNA robots with the confidence of out-of-box model performance, when I think about the trajectory of robotics, it’s astonishing how quickly we’ve gone from if it works in the lab to it works in the world. The next frontier isn’t about proving robots can move—it’s about proving they can reliably help in real-world environments, at scale, across industries. That’s what we’re building at @DynaRobotics The impact of this shift will be massive: - Unlocking productivity across logistics, manufacturing, and beyond. - Expanding what small teams and businesses can achieve. - Freeing humans to focus on higher-level creativity, problem-solving, and connection. The mission is bigger than any single deployment. It’s about ushering in an era where general-purpose robots are as ubiquitous and trusted as computers or smartphones. We’re just getting started, and I couldn’t be more excited for what’s ahead. Join us!🚀🤖

I recently gave a talk at MIT's Embodied Intelligence seminar, covering some of my recent works and perspectives on "Foundation Model Supervision for Robot Learning". Recording is up on Youtube: piped.video/watch?v=JfZYtpEi… Hope you like it and find it useful!

Sharing some exciting DYNA-1 result: zero-shot environment generalization We put DYNA-1 under test in a completely different environment from our training distribution – with an entirely different background (@DynaRobotics banner) and metal table. The table has a reflective and smooth surface, creating a wildly different visual appearance as well as interaction dynamics. The model is able to proceed as usual, adeptly folding and recovering from its own mistakes. By focusing on task mastery, we achieve robust generalization out of the box

Excited to launch @DynaRobotics with a team of incredible researchers, engineers and company builders! At Dyna, our mission is to bring affordable general-purpose AI robots to real production environments.

Super excited to share Eureka, our "spin" on how to use LLMs to teach low-level dexterity skills! Eureka is an open-ended reward design agent that can write and evolve superhuman reward functions for a large suite of robots and tasks, including challenging pen spinning tricks!

Excited to share VIP, a self-supervised visual reward and representation pre-trained on diverse human videos! VIP’s frozen reward and rep. can solve diverse unseen robot tasks using TrajOpt, online RL, and enables real-world few-shot offline RL! sites.google.com/view/vip-rl 🧵:

Excited to share our #ICML2023 paper ✨LIV✨! Extending VIP, LIV is at once a pre-training, fine-tuning, and (zero-shot!) multi-modal reward method for (real-world!) language-conditioned robotic control. Project: penn-pal-lab.github.io/LIV Code & Model: github.com/penn-pal-lab/LIV 🧵:

Humbled to share that I was selected as an Apple Scholar in AIML PhD Fellowship! Very grateful to Apple, my advisors @dineshjayaraman @obastani as well as all my mentors and collaborators for their support! machinelearning.apple.com/up…

Giving a talk tomorrow at the Foundation Models for Interactive Robot Learning workshop (lfmrss2025.weebly.com/) at RSS! Will cover some @DynaRobotics results too! What would people like to see?

Excited to share my first paper as an "advisor" :D We show that pre-trained visual representations enable a simple, fast, no-training subgoal decomposition method for long-horizon robotic manipulation! Paper: arxiv.org/abs/2310.08581 Website: zcczhang.github.io/UVD/ (🧵1/n)

I am attending #CORL2023 and presenting two new papers at various workshops! Excited to make new friends and catch up! Please reach out if you are attending and would like to chat about anything robot learning :)

We have been stress testing our model flywheel and seeing strong results in many challenging tasks! Task 1: chopping veggie🧑‍🍳 A test of coordinated tool use that involves asymmetric task and dynamic feedback to make consistent cuts. We used just 70 trajectories to get the results you see in the video. Task 2: cup stacking 🎉 A test of high-precision control, which requires precise and delicate positioning at every step. Mistakes at any step are catastrophic! The arms we use have high control error, but the model makes up for it. These are quite distinct tasks that are difficult in different ways than some of our earlier demos like laundry/napkin folding. Very glad to see by injecting dextereous control into pre-training in an integrated way, we see substantial boost in post-training robustness and efficiency!

It's so satisfying to watch our models just do the task on demand; human like and no downtime

I am very excited about our new paper (CoRL 2024 Oral) on using LLM code generation to automate environment curricula. It is often hypothesized that intelligent motor control is driven by the need to habituate in and adapt to varied environments. While there's a beautiful literature on unsupervised env design, scaling such idea to robotics has been very difficult so far given the high-dimensional env configuration space. Instead, robotics engineers still use their domain knowledge to manually design environments and their curricula thereof. In Eurekaverse, we demonstrated possibility of automating all that using LLMs and showed a compelling use case on challenging robot parkour! @willjhliang did an amazing job leading the work, and he's presenting it this week at #CoRL2024! Make sure to check out the poster and our oral presentation!

😲 Even I am surprised sometimes for what our models are capable of

We did a fun and timely halloween experiment benchmarking our VLA models' robust reasoning capabilities! 🎃 There's a lot of interest in reasoning for VLA models, but I personally felt most tasks the community benchmark on (1) do not require meaningful reasoning capabilities, or (2) are somewhat unrealistic and do not represent tasks in real-world scenarios. So we decided to use object counting and manipulation as a real benchmark; it's quite common and realistic, but I haven't seen much work in this area. End-to-end Imitation learning would fail because of combinatorially many permutations you can ask to the robot. Our VLA model can count and follow language commands fairly robustly -- all in an end-to-end architecture without external memory modules or counting logic. The model also robustly handles external disturbances to the scene (like shuffling the candy baskets). It's a small cute experiment we did to benchmark reasoning, but it's pretty fun so thought we'd share!

the best part of this is that the model is totally operating *zero-shot*; we didn't prep anything particular for corl. we just brought a robot (a new one that we never ran models on), loaded the same model i demo'd live on stage during actuate, and let it run for the entire conf

It's really unfortunate to hear the FAIR layoff news.. Meta friends: if you are interested in working on frontier real-world robotics, @DynaRobotics is hiring! Please DM me!

I'm attending #ICRA2025 this week! Happy to chat about Dyna and all things embodied AI. DMs are open!

“Don’t practice until you get it right. Practice until you can’t get it wrong.” We have developed a general recipe for robust and autonomous robot foundation models for real-world applications. The linchpin in our recipe is an accurate reward model (RM) that scores every robot interaction with precision. Building on our prior research, we have delivered the first scalable foundation reward model for robotics. This model outperforms previous approaches and can reliably estimate task progress on challenging dexterity tasks, like napkin folding. This capability unlocks a host of production-critical capabilities, such as (1) autonomous exploration, (2) intentional error recovery, (3) high-quality dataset creation and curation, and much more.

Sharing some exciting @DynaRobotics results tomorrow! Stay tuned :D

Excited to start my internship at @MetaAI in their Menlo Park, CA office! I will be working with @yayitsamyzhang, @shagunsodhani, and @Vikashplus on RL and robot learning topics. If you are in the area and want to chat/hang out, please let me know!!

Thanks for having me! I gave a talk recently at MIT's Embodied Intelligence seminar. Recording should be up soon!

Thanks Chris! Dyna’s core philosophy: ship general and robust models and let everyone see. No hype, no cherry picking, just results

Autonomous deployment in real production sites is the real litmus test of general purpose robots

When we founded Dyna, we began with a first-principles question that shaped our focus: What single technical hurdle must we clear to unlock unlimited demand for robots? After talking to hundreds of customers, the answer is so clear yet so underemphasized in current robotics discourse: PERFORMANCE — high throughput and high-quality output, consistently. Delivering robot performance has been our singular north star ever since; no one wants a robot that only kind of works slowly. We have tested DYNA-1 with several distinct 24-hr trials under different natural environment variations and found DYNA-1 robustly complete 700+ napkins in all trials. My favorite part of these timelapse videos is the 6-7am period when the sunrises and the model just continues performing the task as usual!

Honored to be part of the cohort!

Our cute Dynasaurs running around the clock; no downtime, they just work🦖🤖 Kudos to my teammates who did all the work here; it's a testament to how robust our models are. I didn't have to do any work to get this up and running for days!

This is so impressive! I can't imagine the amount of progress we will unlock as a community with low-cost, highly capable robots. Congrats to the Unitree Team!

Learning policies in simulation and transferring to the real world (or Sim-To-Real in short) is a promising strategy for robots to learn complex skills. However, humans need to tune the simulator carefully so that the policies work robustly in the real world: this is difficult, slow, and tedious. DrEureka aims to automate this by having LLMs design crucial components of sim-to-real transfer: Reward Design and Domain Randomization. Here are a few bonus several-minute-long uncut videos of DrEureka yoga ball walking policy in-the-wild on Penn’s campus, enjoy!

I ll be hanging out at the LeRobot Hackathon this weekend, come say hi!

Due to popular requests, we have now uploaded our pre-trained LIV model on HuggingFace for easier downloads! This is my first time doing it, and the experience was quite smooth @_akhaliq huggingface.co/jasonyma/LIV

Using LLMs to program a robot dog to play with balls.. sounds familiar ;) jokes aside, AI assisted development of robot capabilities will be the future. We are still so early

Delighted to announce that our work on a unified framework for offline imitation from observations and examples has been accepted to ICML 2022! #icml #ICML2022

DYNA-1 now folds napkins for paying customers, and we’re unlocking more skills to ship into more commercial environments in the coming weeks & months. Mastering napkin folding won’t transform daily life, but it’s a pivotal step toward making embodied AI commercially viable. This is a dream come true. After years of PhD research aimed at making robots genuinely useful in the real world, nothing felt as close as to what we accomplished in the last few months at Dyna. As we embark on this journey, we are excited to start sharing some of our results and research more widely with the robotics community! Alongside real-world robustness, we are also pushing the boundaries of cutting-edge large-scale robot learning. Join us in building robots for the real world. We look forward to hearing from you! Check out our blog post with our results: dyna.co/research

Agreed; it's my first time attending a developer conference instead of an academic conference. Really got me thinking deeper about deployment-in-the-loop model iteration. Also thanks for the photo feature :D

We are presenting LIV today at #ICML2023! Exhibit Hall 1, #827 2:00pm - 3:30pm HST The future of robotics is multi-modal, and LIV demonstrates how multi-modal value pre-training from diverse human videos can bootstrap language-conditioned robot skill learning. See you there!

A unique challenge we run into at Dyna is how can we make best use of the large amount of data autonomously collected by DYNA-1 during deployment? In continuous deployment settings, robot data does not naturally come with episodic boundaries. We have also developed an approach that can automatically segment the streaming data and provide accurate progress estimation and language labeling to enhance the model's task understanding.

Super excited to share that GoFAR has been accepted to #NeurIPS2022 and flagged for an award! This is a new foundation for the theory and practice of (offline) goal-conditioned RL, check it out!

In contrast, We find that standard recipes for training robot foundation models are insufficient for real-world PERFORMANCE. On a demanding production task like restaurant-grade napkin folding, state-of-the-art models saturate at ≈ 80% single-episode success even after hundreds of hours of domain-specific data. At that level, the chance of 30 flawless consecutive executions is (0.8)³⁰ ≈ 0.1 %—functionally zero for 24/7 operations. We observe the same failure mode in-house: after one to two hours the policy drifts into unfamiliar states and cannot self-recover. The time-lapse below shows our strongest base model collapsing despite an initially perfect run. Flashy demos hide this brittleness; sustained autonomy demands ≥ 99% step-level reliability and robust fault-recovery, not just high single-episode accuracy. So what is our approach?

Excited to give this talk! Also lmk if you are going to be there; DMs are open

Honored to see Eureka on this list along side many amazing works! eureka-research.github.io/

Looking forward to sharing some of my thoughts on how programmatic outputs from large foundation models can accelerate robot learning!

We (@dineshjayaraman , @akrishna42 , and I) recently went on the Economist podcast to discuss DrEureka (eureka-research.github.io/dr…) and other recent works from our lab as well as trends on robot foundation models! Give it a listen if you are interested!

By scaling our RM-in-the-loop training, DYNA-1 has leapt forward in just a few weeks: • Week 1: Base model can complete single success, but falls apart after 5 minutes • Week 2: Ran 1 hours unaided, but compounding errors make recovery impossible • Week 3: Ran 8 hours, but executed only 6-7 napkins per hour (~10 mins per fold) • Week 4: Completed our first 24-hour run—but ~200 folds at low quality and speed • Week 5: Completed 24+ hours with ~350 folds at decent production-grade quality. • Week 6: Sustained 24+ hours with ~850 folds and high production-grade quality From stop-and-go to round-the-clock excellence, our continual learning recipe drives rapid, tangible gains.

I am attending #RSS this week! Participating in the Pioneers workshop as well as giving a talk at the Foundation Models for Interactive Robot Learning workshop. Happy to meet up and chat if you are around!

Excited to be speaking at #Actuate2025!

I am attending #ICML2023 next week in Hawaii! Excited to make new friends and re-connect with old ones! Please reach out if you are attending and would like to chat about anything related to research or ML! My particular interests include foundation models, RL, and robotics!

We are organizing Workshop on Goal-Conditioned Reinforcement Learning (GCRL) at #NeurIPS 2023! Submission Deadline: October 4th, 2023 Website: goal-conditioned-rl.github.i…

It was super fun catching up with @micoolcho and @chris_j_paxton and talking about our Generative Value Learning (GVL) work on @RoboPapers! We are presenting this work next month at ICLR 2025 as a Spotlight paper!

Big congrats to my mentors and collaborators @DrJimFan and @yukez on the new group! Embodied AI and robotics research just kicked up a gear ;)

At a technical level, DrEureka, following our prior work Eureka (eureka-research.github.io/), uses LLM-guided evolutionary search to generate safety-aware reward functions in code that can be used to train policies in sim. Then, leveraging LLMs’ capability as hypothesis generators, DrEureka uses the LLM to choose (1) which physics parameters to randomize, and (2) what ranges they should be randomized over based on reward-aware physics prior (RAPP) over DR parameters. Finally, using the synthesized reward and DR parameters, it trains policies for real-world deployment.

happy 4th! it was really fun to play with this human robot interaction🤖🇺🇸

First, check out our project website for the paper, interactive demos, and getting your robot video labeled by GVL today! You can even listen to an AI podcast about our paper, or ask Gemini questions about our paper too! We (especially @xf1280) put in a lot of effort in getting these demos up. Let us know how you find these new ways to engage with paper! generative-value-learning.gi…

DYNA-1 achieved an unprecedented level of robustness for robot foundation models. But at Dyna, we hold ourselves to an even higher standard: production-grade quality (grades 4 or 5 out of 5 point scale). While 98% of folds reach near-perfect quality (grade ≥3), only 75% hit our rigorous quality bar. What’s the difference? Less than ⅓ inch precision on the initial fold separates perfection (grade 5) from near-perfection (grade 3). Our customers demand perfection—not near perfection—and we deliver. Tiny differences define commercial-grade quality at Dyna. This level of precision also raises the bar of our research, as every research idea is rigorously vetted to ensure measurable and significant real-world performance improvement.

Excited to share some of my recent works on pre-training for robotics with the MILA community!

Over this learning process, DYNA-1 iteratively becomes much better at handling extremely difficult and out-of-distribution situations. Napkin folding is particularly challenging because: Single-pull precision: Extracting exactly one napkin from a tall stack demands fine control and rapid feedback; otherwise the gripper drags out multiple napkins, causing misfolds and chaos (as you can see in the attached videos). Flattening: When a multi-pull leaves napkins crumpled, the policy must (1) detect that multiple sheets were removed, (2) locate corners folded inward, and (3) separate & flatten overlapped layers before refolding. All of which are nontrivial dexterous endeavors. Rapid self-recovery: Once in an out-of-distribution state, the robot must untangle the mess and resume folding fast enough to keep throughput intact. Every extra second spent on edge cases erodes throughput, so the policy needs to find the quickest remedy. DYNA-1’s ability to handle chaotic scenarios even surprised us, and is the fundamental reason why it can go on for 24-hr with 99+% completion rate. There are too many robustness snippets to list, but here are a few of our favorites:

…and what about task generalization? By focusing on dexterity and real-world robustness, we’re seeing strong positive transfer to other tough commercial tasks, such as laundry folding and, at a client’s request, cup-filling. DYNA-1 can autonomously fold many shirts of different sizes and materials in a row and also fill ingredient cups with utmost precision. Cup-filling is perhaps the hardest “no-reset” task we’ve encountered: delicate pickup, precise placement, handover, tool use—one slip ends the run. Though not perfect yet, DYNA-1 can clear every step while our internal baselines fail to move beyond the first step consistently.

Thanks @_akhaliq! LIV is now on arXiv: arxiv.org/abs/2306.00958 Check it out if you are interested in the space of (RL-based) vision-language pre-training for robotics! Happy to answer any questions about the paper :)

Happy to announce that UVD is announced as the best paper at the CORL LEAP Workshop!

DrEureka is co-led by @willjhliang and me with collaborators from @Penn and @NVIDIA: @johnnywang_16, @sam_wang23, and our advisors @yukez @DrJimFan @obastani @dineshjayaraman Check out our project website for the paper and more videos: eureka-research.github.io/dr… Code: github.com/eureka-research/D…

Check out what I have been working on the past few months! SMODICE is a simple and versatile offline IL algorithm that is compatible with learning from observations, mismatched experts, and even just examples! Detailed tweet coming soon 📅

Attending #CVPR2024 this week in Seattle! Looking forward to making new friends and catching up with everyone!

We have added example code for visualizing **animated** zero-shot VIP reward curve on robot videos! Try it out (on your own robot videos) here: github.com/facebookresearch/…

Check out our #L4DC paper on learning policy-aware dynamics model for reinforcement learning! The idea is very simple: focus model learning on the current policy’s visitation distribution. We theoretically show why this is desirable and extend the dual RL paradigm to MBRL, resulting in a simple, practical algorithm, TOM!

We found that DYNA-1 can achieve zero-shot environment generalization for long-horizon dexterity. While we have seen foundation models that can generalize to new environments for simple pick-and-place skills by training on diverse environments and objects, such results remain elusive for bi-manual fine-grained dexterity. The video below shows DYNA-1 folding napkins at a customer site with no additional training, but it’s worth noting that we did observe noticeable performance loss (a topic we will be conducting more research on). With additional on-site training, DYNA-1 quickly improves and becomes adept at continuous folding at the customer site. This milestone represents a significant step towards our vision of delivering performance, out of the box.

I am attending #NeurIPS2022 next week to present several works on offline RL and pre-training for robotics! Would love to meet people and discuss anything, in particular, RL and robot learning topics! DM me if you are attending and want to chat or grab coffee 😀

We at @GRASPlab hosted @ericjang11 last week for a thought-provoking talk on Humanoid robots and 1X! The full recording is up now, check it out!

Come to our workshop on Goal-Conditioned RL tomorrow at #NeurIPS2023!

I won't be at #RSS2024, but @willjhliang will be presenting our DrEureka work! Go talk to him to get the latest scoop on using foundation models for dexterous robot skill learning!

This was a very fun project and we learned a lot about how to use LLMs to enable robot skill learning! There are many challenges and potential future directions. For example, how to combine DrEureka with real-world execution feedback and using vision to provide feedback on reward and DR generation. And of course, not having to use a leash in the real world would be nice, but safety was important (no robot was hurt in the demos!). To cap off, here are some failures and blooper videos :)

Jim has been a fantastic mentor for me! Apply if you are interested in foundation models for decision making and AI Agents!

Reward learning is a fundamental challenge in RL. In VIP, we address this by pre-training a value function on action-free human videos, and the pre-trained VIP value function can zero-shot transfer to unseen robot tasks! Find out more about VIP at the Deep RL workshop tomorrow!

sim2real for manipulation tasks is really hard! Great to see this work come out

The robotics industry has received much excitement lately. However, current technical challenges significantly hinder the widespread viability of robots. Most robotic foundation models today operate at just 10-30% of human-level speeds, impacting productivity on high-speed production lines. They also lack generalization capabilities, requiring extensive data collection even for minor environmental or task variations. Additionally, robot hardware remains expensive and insufficiently durable, limiting deployments to scenarios where ROI is easily justified... And these are just a few of the issues.

We are hiring, and my DMs are open! dyna.co/careers

I'd like to thank all my collaborators for making this a super fun and rewarding project: @JoeyHejna @ayzwah @ChuyuanFu @shahdhruv_ @jackyliang42 @drzhuoxu @SeanKirmani @sippeyxp @DannyDriess @xiao_ted @JonathanTompson @obastani @dineshjayaraman @Stacormed @tingnan1986 @DorsaSadigh @xf1280 . Many of them are currently at @corl_conf , make sure to talk to them about our paper! I am particularly grateful to @xf1280 for his mentorship and guidance throughout this project; I benefited a lot from his expertise and insights on frontier VLMs for robotics!

Papers covered: Generative Value Learning (GVL): arxiv.org/abs/2411.04549 Language-Image Value Learning (LIV): arxiv.org/abs/2306.00958 Value Implicit-Pretraining (VIP): arxiv.org/abs/2210.00030 Eurekaverse: arxiv.org/abs/2411.01775 DrEureka: arxiv.org/abs/2406.01967 Eureka: arxiv.org/abs/2310.12931 Universal Visual Decomposer (UVD): arxiv.org/abs/2310.08581 Goal-Contrastive Rewards (GCR): arxiv.org/abs/2410.19989

If our mission excites you, please reach out! We are looking for incredible talents in both AI and robotics! DM me or visit our site for open roles! dyna.co

Excited to have @ericjang11 visit @GRASPlab @PennEngineers!!

Giving contributed talks on VIP at the Offline RL, Foundation Model for Decision Making, and Deep RL workshops at #NeurIPS2022. Come check out how we can pre-train a value function on passive human data and zero-shot transfer to robotics manipulation!

We highlight quadruped yoga ball walking. This task is particularly hard because (1) it is a novel task for which the LLM could not have seen human-generated reward functions or DR, and (2) simulation cannot model the deformable surface of an air-inflated ball, making good sim-to-real absolutely necessary. It was not clear at all when we started that this could work, so we are very excited about the result! Here is a comparison against other controllers and we clearly see that alternatives quickly fail and are quite unsafe to deploy.

UVD code is open-source now: github.com/zcczhang/UVD/ Get your long-horizon demonstrations segmented in few seconds! We support various SOTA pre-trained reps (VIP, R3M, LIV, VC-1, ...) as well as policy backbones (MLP, GPT).

We are organizing a workshop on goal-conditioned RL! Please do consider submitting/participating if relevant :) more details to follow.

We believe mastery of one beats mediocrity in many. We’re laser-focused on achieving general-purpose capabilities by perfecting one task at a time. Today, we’re bringing cost-effective, easy-to-deploy robotics solutions to businesses of all sizes.

How do you know you have product-market fit? When your robot looks like crap but delivers so much value that customers happily pay for it.

Also check out the @FortuneMagazine coverage from @sharongoldman ; thanks Sharon for breaking the news: fortune.com/2025/03/25/exclu…

This is my internship project at @NVIDIAAI! I had a blast working on it and learned a lot from the experience. I am really grateful to my mentors @AnimaAnandkumar @DrJimFan @yukez for their guidance on the project!

Congrats Tony!! Very impressive :)

The answer is yes, and the method is simple yet intriguing! We propose formulating value learning as an autoregressive prediction task over *shuffled* sequence of the input video. Why? Think about a standard video showing a task unfolding in chronological order. We empirically find that this actually makes it harder for the VLM to estimate progress because it might just latch onto the order of the frames instead of the underlying changes that signify actual progress towards completing the task. By shuffling, we force the VLM to work “harder” to figure out the correct order based on the visual cues of task progress, and doing so significantly improves the faithfulness of the value predictions! In a way, GVL poses value predictions as an ‘’temporal unshuffling’’ puzzle to the VLM; it has all the pieces, but it has to figure out how those pieces fit together in a way that makes sense based on progress towards a goal.

After conversations with hundreds of potential customers, we discovered that people don’t care about fancy designs—they just want robots to perform reliably and deliver clear ROI. Our approach? Start simple and excel.

Nice work on how to go beyond BC by learning to search using world and reward models! The nice thing is that you don't need to collect any additional data.

Another video of our VLA's robust reasoning capabilities against real-time disturbances:

We are organizing a workshop on task specification at #RSS2024! Consider submitting your latest work to our workshop and attending!