Assistant Professor @imperialcollege MTS @amilabs
London
- Tweets 730
- Following 1,428
- Followers 4,002
- Likes 13,685
(1/2) New CVPR paper on speech-to-gesture prediction! Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound and synthesize a corresponding video of the speaker.
18
281
835
📢 New paper alert! How does one adapt a pre-trained visual model to novel downstream tasks without task-specific finetuning or any model modification? Inspired by #prompting in NLP, our new paper investigates Visual Prompting. (1/5)
10
106
693
model = torch.compile(model) is magic.
With only one line of code, I get ~40% speed up per training iteration.
20
18
465
87,786
Animals are intelligent agents that plan and act to accomplish complex goals. Can we try learning from them? We present EgoPet, a new ego centric video dataset of animals scraped from YouTube and TikTok.
13
73
407
182,430
CLIP is arguably the leading pretraining paradigm in computer vision. In a new preprint, we show that vision-only SSL models trained on web data can match CLIP on VQA tasks, despite not using language.
Paper: arxiv.org/abs/2504.01017
Project Page: davidfan.io/webssl/
3
57
310
19,297
Life update: Wrapping up my PhD and graduating in two weeks from @TelAvivUni and @berkeley_ai!
Next up: moving to NYC to start a postdoc at @AIatMeta, where i will be working with @ylecun. 🚀
Also, looking to meet some new and old friends in NYC area, DM me :)
20
5
312
36,900
Navigation World Models won the Best Paper Honorable Mention Award at #CVPR2025 ☺️
It is my first postdoc paper since joining Yann's lab at @AIatMeta, so I am very excited. It was also extremely fun working with @GaoyueZhou, @dans_t123, @trevordarrell (and @ylecun)
Fun story:
Congratulations to the #CVPR2025 Honorable Mentions for Best Paper!
@GoogleDeepMind, @UCBerkeley, @UMich, @AIatMeta, @nyuniversity, @berkeley_ai, #AllenInstituteforAI, @UW, #UniversityCollegeLondon, @UniversityLeeds, @ZJU_China, @NTUsg, @PKU1898, @Huawei Singapore Research Center
27
22
283
74,760
Happy to share our new work on Navigation World Models! 🔥🔥
Navigation is a fundamental skill of agents with visual-motor capabilities. We train a single World Model across multiple environments and diverse agent data.
w/ @GaoyueZhou, Danny Tran, @trevordarrell and @ylecun.
5
58
276
83,570
a recipe to reproduce #Genie3:
1️⃣ collect a large egocentric video dataset and apply VGGT to get camera poses. Add more data from 3D reconstructed scenes.
2️⃣ train a Navigation World Model with long context → amirbar.net/nwm
3️⃣ distill to an efficient model for RT.
9
20
196
26,739
also- it is a distraction. long horizon planning in pixel space doesn’t make sense.
On world model / egocentric visual dynamics model, also on building robotic simulation, also on building robotic genAI models:
Being visually realistic doesn't mean being physically accurate and semantically correct.
9
3
151
42,766
FAIR is probably the only lab outside of academia where research projects can start like this.
[7/8] This side project started in October when @TongPetersb, @_amirbar, and I were thinking about the rise of CLIP as a popular vision encoder for MLLMs. The community often assumes that language supervision is the primary reason for CLIP's strong performance.
However, we realized that the pretraining data distribution and scale differ a lot. For example, CLIP models are often trained on billion-scale image-text pairs from the web, while SSL models are often trained on million-scale or hundred million-scale data from an ImageNet-like distribution. Thus, we really need apples-to-apples comparisons to study this question.
We hope that our work will inspire a return to more controlled experimentation whenever possible!
3
6
111
15,809
Excited to share that our paper on Navigation World Models was selected for an Oral presentation at CVPR!
Code & models:
github.com/facebookresearch/…
huggingface.co/facebook/nwm
Happy to share our new work on Navigation World Models! 🔥🔥
Navigation is a fundamental skill of agents with visual-motor capabilities. We train a single World Model across multiple environments and diverse agent data.
w/ @GaoyueZhou, Danny Tran, @trevordarrell and @ylecun.
3
6
101
8,298
heading to #ICCV2025, anyone up for a ☕️?
also, my team at FAIR has an internship opening on world modeling, planning, and their robotics applications. DM me if you’re interested.
6
9
103
12,639
Navigation World Models was accepted to #CVPR2025 🎉 Congrats to co-authors @GaoyueZhou, Danny Tran, @trevordarrell and @ylecun
See you in Nashville!
Happy to share our new work on Navigation World Models! 🔥🔥
Navigation is a fundamental skill of agents with visual-motor capabilities. We train a single World Model across multiple environments and diverse agent data.
w/ @GaoyueZhou, Danny Tran, @trevordarrell and @ylecun.
1
7
93
6,310
How does visual in-context learning work? We find "task vectors" -- latent activations that encode task-specific information and can guide the model to perform a desired task without providing any task examples. 🧵 (1/n)
Finding Visual Task Vectors
Find task vectors, activations that encode task-specific information, which guide the model towards performing a task better than the original model w/o the need for input-output examples
arxiv.org/abs/2404.05729
2
14
89
18,857
(1/2) say you want to plan your way back from honolulu to nyc using a WM in pixels-
1. the world is stochastic and partially observable. you can plan how to pack your suitcase and leave the hotel room, but 99% of your pixel-level plan afterwards is useless
9
4
85
53,806
Thanks for featuring our work @ak92501 :) Also, we've just released the code for running most of the experiments github.com/amirbar/DETReg.
DETReg: Unsupervised Pretraining with Region Priors for Object Detection
pdf: arxiv.org/pdf/2106.04550.pdf
abs: arxiv.org/abs/2106.04550
project page: amirbar.net/detreg/
unsupervised pretraining approach for object detection with transformers using region priors
1
18
69
(2/2) We also release our full dataset and will make the code available. Joint work with with @shiryginosar, Gefen Kohavi, Caroline Chan, Andrew Ownes and Jitendra Malik. For more details see project page: people.eecs.berkeley.edu/~sh…. @berkeley_ai @ZebraMedVision
1
9
63
heading to Nashville to attend @CVPR tomorrow. looking forward to meeting old & new friends and chat about #WorldModels
5
1
55
3,352
Excited to share DETREg, our new work on unsupervised pretraining for object detection with transformers. Compared to previous works, the key idea in DETReg is attempting to learn object detection in the unsupervised pre-training stage. @berkeley_ai , @TelAvivUni, @NVIDIAAI
1
8
50
If you are at CVPR, come visit our poster today! Thursday 2:30-5:00, poster 127.
Excited to share DETREg, our new work on unsupervised pretraining for object detection with transformers. Compared to previous works, the key idea in DETReg is attempting to learn object detection in the unsupervised pre-training stage. @berkeley_ai , @TelAvivUni, @NVIDIAAI
1
6
43
Need a strong feature extractor for your upcoming NeurIPS paper? we got you 😉
We are open-sourcing all the models in Web-SSL, from ViT-L to ViT-7B!
It was super fun to train and play with these massive ViTs.
Models: huggingface.co/collections/f…
Github: github.com/facebookresearch/…
Huge credit to @DavidJFan for putting these models together!
39
3,641
Mom, I’m on a podcast!
Thanks for having me on your podcast, @samcharrington
Today we're joined by @_amirbar, PhD candidate at @berkeley_ai and @TelAvivUni to discuss his research on visual-based learning and his paper, “EgoPet: Egomotion and Interaction Data from an Animal’s Perspective.”
We dig into:
🔹Caption-based dataset limitations
🔹The ‘Learning problem’ in robotics
🔹Visual Interaction Prediction (VIP), Vision to Proprioception Prediction (VPP), Locomotion Prediction (LP), and more!
🎧 / 🎥 Listen or watch the full episode on our page: twimlai.com/go/692.
📖 CHAPTERS
===============================
00:00 - Introduction
02:36 - Research interests
09:42 - Research projects
20:02 - EgoPet
27:31 - EgoPet dataset
29:25 - Visual Interaction Prediction (VIP) vs object recognition
31:09 - Findings on the model performance trained on EgoPet dataset
32:29 - Benchmark tasks (VIP, VPP, LP)
37:50 - Future directions
3
3
37
2,838
Paper: arxiv.org/abs/2404.09991
Project Page: amirbar.net/egopet/
Data: github.com/bakhtiararya/EgoP…
Code: github.com/DannyTran123/egop…
Kudos to the authors Arya Bakhtiar, Danny Tran, @antoniloq, @jathushan, @ylecun, @amirgloberson and @trevordarrell.
1
2
35
72,449
We're organizing the 1st workshop on Prompting in Vision at #CVPR2024 in Seattle. We have an amazing line of speakers, and we accept 8-pages paper submissions. Stay tuned!
Website: prompting-in-vision.github.i…
OpenReview: openreview.net/group?id=thec…
7
34
3,133
Check out PEVA 🌎, our recent attempt to build a world model for human body control.
What would a World Model look like if we start from a real embodied agent acting in the real world?
It has to have: 1) A real, physically grounded and complex action space—not just abstract control signals. 2) Diverse, real-life scenarios and activities.
Or in short:
It has to be annoyingly complex—in both the action and vision space—to even get close to real life.
We did an initial attempt: Whole-Body Conditioned Egocentric Video Prediction.
In collaboration with @dans_t123 , @_amirbar, @ylecun , @trevordarrell and @JitendraMalikCV.
(For more details, check: arxiv.org/abs/2506.21552)
What we did is very simple: Predict Egocentric Video from human Actions (PEVA) - Given the past video and a future action represented by relative 3D body pose, PEVA predicts how the world looks next—from the first-person view.
By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, it learns how physical actions shape perception.
1
2
31
2,771
Paper link: arxiv.org/abs/2209.00647, We will release the dataset & code soon in the project page: yossigandelsman.github.io/vi…
Joint work with collaborators @YGandelsman , @trevordarrell @amirgloberson and Alyosha Efros from @berkeley_ai & @TelAvivUni. (5/5)
4
3
26
Interestingly, compositionality & spatial understanding isn't just learned with *almost infinite data*. This suggests that there is room for inductive biases and more structured models.
1
1
28
(1/n) so why train a large world model in an end-to-end manner on large data for navigation indeed? navigation is a first step in a larger vision to build a model capable of simulating many tasks by planning in a single framework (e.g, think manipulation and more).
Devil's advocate mode on: Navigation World Models have existed for a long time... they're called maps! And there are plenty of good algorithms out there which enable robots to build them / render views from them / localise within them / use them for planning. #SLAM #SpatialAI :)
1
5
27
8,295
Working on prompting in computer vision? Consider submitting your paper to our #CVPR2024 workshop.
Paper submission deadline: March 15th.
Website: prompting-in-vision.github.i…
1
4
26
12,636
basically this - hunyuan-gamecraft.github.io/
a recipe to reproduce #Genie3:
1️⃣ collect a large egocentric video dataset and apply VGGT to get camera poses. Add more data from 3D reconstructed scenes.
2️⃣ train a Navigation World Model with long context → amirbar.net/nwm
3️⃣ distill to an efficient model for RT.
2
3
27
5,232
if you like torch.compile, wait till you hear about FlexAttention. if you're using attention masks (e.g, causal masking), expect another 30%-40% boost.
best to test using the recent torch nightly.
model = torch.compile(model) is magic.
With only one line of code, I get ~40% speed up per training iteration.
6
23
4,049
Introducing IMProv, our new multimodal prompting model trained on 200k Semantic Scholar figures & LAION 400m. Shows cool in-context learning capabilities for computer vision. Kudos to @Jerry_XU_Jiarui @YGandelsman @jw2yang4ai
@JianfengGao0217 @trevordarrell @xiaolonw. 🧵👇
Can a machine solve diverse computer vision tasks even on the ones it is not trained on?
Introducing IMProv: It performs multimodal in-context learning for solving generic computer vision tasks. It formulates all tasks as an image inpainting problem.
arxiv.org/abs/2312.01771
1
6
25
5,579
EgoPet contains 84 hours of video footage of mainly cats and dogs but also other exotic animals like turtles, eagles and snakes.
1
2
21
2,800
Our new work "Compositional Video Synthesis with Action Graphs" is out! We focus on goal-oriented #video #generation and introduce the new task of *Action Graph to Video*.
Project page: roeiherz.github.io/AG2Video
Abstract: arxiv.org/abs/2006.15327
@BAIR @NVIDIAAI @TAU @Nvidia
1
5
24
(2/2)
2. you’d need to generate hours of video for a single plan, which is highly inefficient.
so you must lift the level of abstraction from pixels.
3
23
2,666
Join us to the first workshop on Prompting in Vision at @CVPR on June 17th, starting on 9am. We have an amazing line of speakers, poster session, and a panel moderated by @trevordarrell.
Looking for a good prompt? Join our workshop at #CVPR2024 on June 17! 🚀✨
@CVPR @_amirbar @liuziwei7 @YGandelsman @SharonYixuanLi @hyojinbahng @LINJIEFUN @amirgloberson @zhang_yuanhan @BoLi68567011 @JingkangY
4
22
5,801
Our code & pretrained models:
github.com/facebookresearch/…
New paper from FAIR+NYU:
Q: Is language supervision required to learn effective visual representations for multimodal tasks?
A: No.
⬇️⬇️⬇️
1
2
19
1,625
These are actually hard to catch as it requires collaboration between different organizing committees and a manual review. If major AI conferences converge to openreview this can probably be automated.
1
20
7,358
Together with the dataset, we provide a benchmark and evaluation suite of two in domain tasks (interaction prediction, locomotion prediction) and a downstream robotic task (quadruped vision to proprioception prediction).
1
2
19
2,573
Ping me if you're attending #NeurIPS2023 and want to grab some coffee or chat about visual prompting and large vision models! I'm also in the job market (graduating summer 2024).
19
992
One potential explanation why random masking does not scale: different sentences have different information density. Using constant 15% masking might not extend to larger and more diverse datasets. Next token prediction is just simpler.
2
20
4,679
We find that video models pre-training on EgoPet perform better on these tasks compared to video models trained on other larger ego datasets like Ego4D.
1
1
18
2,742
fun fact -- EgoPet is heavily inspired by @ylecun's take that we're still far from cat level intelligence.
Animals are intelligent agents that plan and act to accomplish complex goals. Can we try learning from them? We present EgoPet, a new ego centric video dataset of animals scraped from YouTube and TikTok.
1
18
2,516
VLMs are the new cool kid, but what representations make instruction tuning and in-context learning work?
TL;DR: No matter how you define the task (image examples, text examples, or instructions), VLMs convert it into a shared cross-modal task representation. More details 👇
In a new preprint, we show that VLMs can perform cross-modal tasks...
...since text ICL 📚, instructions 📋, and image ICL 🖼️ are compressed into similar task representations.
See “Task Vectors are Cross-Modal”, work w/ @trevordarrell, @_amirbar.
task-vectors-are-cross-modal…
1
1
19
2,857
Very cool work. A different way to think about it is via equivariance. Upsampling of the features should be consistent w.r.t augmentations T: F(T(x))=T(F(x))
FeatUp
A Model-Agnostic Framework for Features at Any Resolution
Deep features are a cornerstone of computer vision research, capturing image semantics and enabling the community to solve downstream tasks even in the zero- or few-shot regime. However, these features
2
18
3,510
Enjoyed catching up with friends and colleagues at #NeurIPS2023! Wishing everyone safe travels.
1
15
810
Happy to share the code for our #CVPR paper
"Learning Individual Styles of Conversational Gesture"
github.com/amirbar/speech2ge… @shiryginosar
1
2
18
Working on prompting in computer vision? Consider submitting your paper to our #CVPR2024 workshop.
Paper submission deadline: March 15th.
Website: prompting-in-vision.github.i…
1
4
15
6,570
אני מזדהה עם התחושות שלך. אבל.. חשוב לי גם להגיד שבתור סטודנט בברקלי, מעוז השמאל הפרוגרסיבי, הפידבקים שאני מקבל מהפרופסורים ומסטודנטים לדוקטורט הוא של תמיכה בישראל. יש גם הפגנות נגדנו, בעיקר של סטודנטים לתואר ראשון (רובם ככל הנראה לא כזה יודעים על מה הם מפגינים).
2
17
1,464
Given input-output image example(s) of a new task a new input image, the goal is to produce the output image, consistent with the given examples. Posing this problem as simple image inpainting, literally just filling in a hole in a concatenated grid-like visual prompt image (2/5)
2
1
16
The secret ingredient to get this to work is the training data. To obtain image data that better resembles our visual prompts, we curated 88k unlabeled figures from paper sources on Arxiv. (3/5)
1
14
Thanks for featuring our work! We’ll answer questions on alphaXiv throughout the week.
alphaxiv.org/abs/2412.03572
Navigation World Models
A controllable video generation model that predicts future visual observations for navigation tasks based on past observations and actions.
Problem:
Visual-motor agents struggle with planning flexible navigation trajectories, especially in dynamic or unfamiliar environments.
Method:
Introduces a Conditional Diffusion Transformer (CDiT) scaled to 1 billion parameters, trained on egocentric videos from human and robotic agents, enabling trajectory simulation and evaluation.
Insights: Dynamic trajectory planning benefits from learned visual priors, allowing adaptation to new constraints and environments, even imagining trajectories in unseen spaces.
Results: Achieves superior trajectory planning by simulating or ranking options, improving navigation performance in both familiar and unfamiliar environments.
Author @_amirbar is on alphaXiv this week to discuss the paper!
2
14
1,996
Had an amazing time meeting all co-authors in person at #CVPR2022 ! @trevordarrell @GalChechik @amirgloberson @xinw_ai @colorado_reed @roeiherzig @vadimkantorov Anna Rohrbach.
5
14
The code for our paper Compositional Video Synthesis with Action Graphs is now available here: github.com/roeiherz/AG2Video
ICML 2021 camera ready:
arxiv.org/abs/2006.15327
project page:
roeiherz.github.io/AG2Video/
Our work *Compositional Video Synthesis with Action Graphs* is out!
We introduce *Action Graphs*, a natural and convenient structure representing the dynamics of actions between objects over time and show we can synthesize goal-oriented videos on 2 datasets.
#TAU #BAIR #NVIDIA
1
4
14
Come visit our EgoPet poster #92 today at 4:30pm! Unfortunately I couldn’t attend Milan, but say hi to the great @antoniloq !!!
Today at 4:30p at #ECCV2024 in Milan, I'll present EgoPet, the first large collection of egocentric videos from animals' perspective! If you're curious about what we can learn from animals, come to poster #92!
Project Website: amirbar.net/egopet/
1
13
1,082
We train a large 3B LLaMA-like transformer on over 50 computer vision datasets with the sole objective to predict the next VQGAN token. We visually prompt the model in test time and observe very interesting completions. Congrats to @YutongBAI1002 and @younggeng. 🧵👇
How far can we go with vision alone?
Excited to reveal our Large Vision Model! Trained with 420B tokens, effective scalability, and enabling new avenues in vision tasks! (1/N)
Kudos to @younggeng @Karttikeya_m @_amirbar, @YuilleAlan Trevor Darrell @JitendraMalikCV Alyosha Efros!
12
1,260
Burning babies, raping women, butchering innocent party goers, not to mention kidnapping over 100 civilians -- these are war crimes. Evacuating the civil population is a measure to protect them from being used as human shield for Hamas. But you know this.
1
13
380
For more information, see our paper and project page:
Project Page: amirbar.net/nwm
Preprint: arxiv.org/abs/2412.03572
work is a collaboration between @AIatMeta and @berkeley_ai 😊
1
1
11
1,577
גם לי קרה לא מזמן, זה קטע? רופאת משפחה אמרה שלפי ה bmi אני צריך להוריד במשקל. ולא עזר להסביר שאני מתאמן קבוע במכון ואחוז שומן סבבה 🤦♂️
2
9
1,434
Very excited to share that we've received an FDA clearance for our intracranial hemorrhage triage algorithm! Trained on over 250,000 CT scans, we hope to augment radiologists in their day-to-day practice and ensure patients receives the best care. @ZebraMedVision
1
2
11
Our Large Vision Model (LVM) paper code and interactive demo are finally live here:
github.com/ytongbai/LVM
huggingface.co/spaces/Emma02…
How far can we go with vision alone?
Excited to reveal our Large Vision Model! Trained with 420B tokens, effective scalability, and enabling new avenues in vision tasks! (1/N)
Kudos to @younggeng @Karttikeya_m @_amirbar, @YuilleAlan Trevor Darrell @JitendraMalikCV Alyosha Efros!
1
1
11
1,897
The papers I was assigned to review for #CVPR2024 feel distant from my expertise compared to #NeurIPS2023. Wondering if the matching system in CVPR is worse or if I've shifted away from being a vision person. 🤔
3
11
5,480
MetaMorph extends instruction tuning (e.g, LLaVA-like models) to image generation, showing very appealing results 👇
How far is an LLM from not only understanding but also generating visually?
Not very far!
Introducing MetaMorph---a multimodal understanding and generation model.
In MetaMorph, understanding and generation benefit each other. Very moderate generation data is needed to elicit visual generation from an LLM, when trained jointly with visual understanding.
1
9
1,234
We explore the nature of I.V Contrast using deep learning methodologies, driven by data and guided by clinical insights. Joint work with Raouf Muhamedrahimov, Jonathan Laserson, Ayelet Akselrod-Ballin, Eldad Elnekave openreview.net/forum?id=SylD…
1
1
10
NWM intersects with multiple communities (video generation, 3D vision, robotics, RL, representation learning...) and it seemed to equally piss off everyone. I remember I told @GaoyueZhou and @dans_t123 - "lower expectations, this paper is a 99% reject".
1
11
3,668
WORLDMEM: Adding memory to world models
Thanks for sharing! @_akhaliq
For more information:
📜ArXiv: arxiv.org/abs/2504.12369
🤗 Hugging Face: huggingface.co/papers/2504.1…
🌐 xizaoqu.github.io/worldmem/
🧑💻 GitHub: github.com/xizaoqu/WorldMem
🚀 Demo: huggingface.co/spaces/yslan/…
9
849
(1/n) New work on Scene Graph to Image generation! We focus on input scenes that are more complex than those previously tackled and show improved performance on three different datasets: Visual Genome, COCO, CLEVR.
pdf: arxiv.org/abs/1912.07414
1
4
10
Using static graphs that do not allow a convenient way to debug the code. By the time they moved to dynamic graphs it was too late.
2
10
859
Our Navigation World Model can simulate trajectories by generating video. This capability unlocks planning: simply find the sequence of actions that leads from the input image to a target goal. In unknown environments, our model can hallucinate navigation trajectories.
1
10
1,796
Interested in our work but busy doing the dishes?
No problem! you can listen to the generated #notebooklm podcast, which is 95% accurate 😉
notebooklm.google.com/notebo…
VLMs are the new cool kid, but what representations make instruction tuning and in-context learning work?
TL;DR: No matter how you define the task (image examples, text examples, or instructions), VLMs convert it into a shared cross-modal task representation. More details 👇
2
9
695
great example on how to stick to your research agenda despite temporary distractions.
Paper is rejected, but a followup paper that completely depends on the rejected paper is accepted #NeurIPS
2
10
1,044
(Un)surprisingly @ylecun didn't mind 😅 and so did @trevordarrell which was a bit reassuring. Anyway--it's nice to see the outcome is an award.
if you're interested to hear more, come tomorrow (Sat) to Oral Session 4B (ExHall A2, 1:00-2:15) and visit poster #396 (Hall D, 5-7pm)
1
9
1,509
You’re nailing it @iclr_conf
7
1,465
Our first 510(k) for a deep learning based algorithm!
businesswire.com/news/home/2…
2
2
8
Had a great time yesterday giving a talk on the challenges for AI in Radiology at the Hebrew University computer vision seminar. @CseHuji @ZebraMedVision.
1
8
Jews and Palestinians, whether they like it or not are semites. Closely related “cousins”. Anything else is your projection.
Handcuffing, raping then burning your victims… the cruelty of Hamas does not exist in nature 💔
3
7
685
A lot of our work is also built on the work of others. Our Conditional Diffusion Transformer model (CDiT) extends DiT by @billpeeb and @sainingxie, and much of the data and inspiration is based on the works of our @berkeley_ai colleagues Noriaki Hirose and @shahdhruv_
1
8
1,674
חלאס עם ההתממות אבישי. חוקי נבצרות, מתנות, ביטול הפרת אמונים, חוק דרעי, פוליטיזציה של מינוי שופטים ועוד ועוד. שינוי כללי המשחק באופן קיצוני ללא הסכמה רחבה במדינה ללא חוקה וכשראש הממשלה תחת חקירה מפר את הסכם ניגוד העניינים שלו.
8
114