what happens in the residual stream of gemma3? l2 norm of activation explodes at the end of every transformer block after x=x+res. key architectural difference between gemma2 and 3 is softcapping vs qknorm. 1b is not even multimodal (fig reps gemma2-2b vs 3-1b). what's wrong?

Want strong SSL, but not the complexity of DINOv2? CAPI: Cluster and Predict Latents Patches for Improved Masked Image Modeling.

if you train a model exclusively on JPEG images, will performance drop on other image file formats?

🚨 RELEASE ALERT ‼️ github.com/facebookresearch/… THIS CHANGES EVERYTHING $META just dropped a game-changing codebase! Now everyone can do LLM research! 😱 🧵10 best things people are already building with lingua 🔥👇

wait. distributed training with pure pytorch is not that bad. why did we all collectively get gaslit into using accelerate...

This simple pytorch trick will cut in half your GPU memory use / double your batch size (for real). Instead of adding losses and then computing backward, it's better to compute the backward on each loss (which frees the computational graph). Results will be exactly identical

🚨New doctor in the house!🚨 Congrats to @TimDarcet for his tremendous work (DINOv2, registers, CAPI) & successful PhD defense followed by ~2 hrs of questions -- he's got stamina! Congrats to his incredible team of advisors from Inria & Meta: @julienmairal @p_bojanowski M. Oquab

Oh I am a big fan of self supervised learning. Also ssl has never been benchmark maxing on imagenet afaik. I am mainly complaining about the supervised classification imagenet hill climb

Introducing DINOv3 🦕🦕🦕 A SotA-enabling vision foundation model, trained with pure self-supervised learning (SSL) at scale. High quality dense features, combining unprecedented semantic and geometric scene understanding. Three reasons why this matters…

lfg it's fitting

Just FYI, computer vision papers submitted to IEEE that include this image of Ms. Forsén will no longer be considered for publication

Statistics from @ICSE2024. Authors submitting, *each*, 33, 27, 24, ... papers. Interactive dashboard: app.powerbi.com/view?r=eyJrI…

Announcing the #ICLR2024 Outstanding Paper Awards: blog.iclr.cc/2024/05/06/iclr… Shoutout to the awards committee: @eunsolc, @katjahofmann, @liu_mingyu, @nanjiang_cs, @guennemann, @optiML, @tkipf, @CevherLIONS

Oh wow, FID is fragile...

I've come to really not like submitit honestly In practice what I was doing in my codebase is just writing my own sbatch files, and every exp is a different script (which is commited to the repo)

2024 update

Our method reveals model-specific patterns too: DinoV2 (left) shows specialized geometric features (depth, perspective), while SigLIP (right) captures unique text-aware visual concepts: This opens new paths for understanding model differences! (7/9)

Could you give a summary for all the lazy readers who won't open the link?

Every time a colleague of mine does not lock their laptop, I add something to their .bashrc. alias vim='nano' is a good one, but moving file to a random folder is even funnier. rm is too evil, don't do it!

Outstanding Finalist 2: “DINOv2: Learning Robust Visual Features without Supervision," by Maxime Oquab, Timothée Darcet (@TimDarcet), Théo Moutakanni (@TheoMoutakanni) et al. 5/n

DINOv2+registers=♥️ We are releasing code and checkpoints for DINOv2 augmented with registers and a slightly better training recipe. No more of those pesky artifacts! Simple one-liner, try it out: dinov2_vitg14_reg = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14_reg')

Never thought I'd see a transpose with 8 numbers

Fewer than 1 in 5 doctors can correctly answer a basic question about statistics

2. Loss? “DINO head”: good results, too unstable Idea: preds and targets have diff. distribs, so EMA head does not work on targets → need to separate the 2 heads So we just use a clustering on the target side instead, and it works

The acceptance rate among aspiring ICLR2024 first authors who submitted >= 4 papers was 15%! Contrast that with the base acceptance rate that year: 30.5%. Unsettling.

As expected, that was popular. Here is my attempt at consolidating all the answers into a list. - Prenorm: normalization in the residual blocks before the attention operation and the FFN respectively - GQA (Group Query Attention): more Q than (K, V)

nice work !! Did you roll out your own fsdp implementation? github.com/facebookresearch/…

uiua goes unbelievably hard wtf array-orientated, stack based, glyph programming language and now I wanna make the game of life in it this weekend

LLMs are great, but their internals are less explored. I'm excited to share very interesting findings in paper “Massive Activations in Large Language Models” LLMs have very few internal activations with drastically outsized magnitudes, e.g., 100,000x larger than others. (1/n)

fuck your fancy personal page template im rawdoggin the html and you wont even make me use css

Open science is how we continue to push technology forward and today at Meta FAIR we’re sharing eight new AI research artifacts including new models, datasets and code to inspire innovation in the community. More in the video from @jpineau1. This work is another important step towards our goal of achieving Advanced Machine Intelligence (AMI). What we’re releasing: • Meta Spirit LM: An open source language model for seamless speech and text integration. • Meta Segment Anything Model 2.1: An updated checkpoint with improved results on visually similar objects, small objects and occlusion handling. Plus a new developer suite to make it easier for developers to build with SAM 2. • Layer Skip: Inference code and fine-tuned checkpoints demonstrating a new method for enhancing LLM performance. • SALSA: New code to enable researchers to benchmark AI-based attacks in support of validating security for post-quantum cryptography. • Meta Lingua: A lightweight and self-contained codebase designed to train language models at scale. • Meta Open Materials: New open source models and the largest dataset of its kind to accelerate AI-driven discovery of new inorganic materials. • MEXMA: A new research paper and code for our novel pre-trained cross-lingual sentence encoder with coverage across 80 languages. • Self-Taught Evaluator: a new method for generating synthetic preference data to train reward models without relying on human annotations. Access to state-of-the-art AI creates opportunities for everyone. We’re excited to share this work and look forward to seeing the community innovation that results from it. Details and access to everything released by FAIR today ➡️ go.fb.me/hgtkel

Vision Transformers Need Registers paper page: huggingface.co/papers/2309.1… Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.

DINOv2: Learning Robust Visual Features without Supervision Maxime Oquab, Timothée Darcet, Théo Moutakanni et al.. Action editor: Abhishek Kumar. openreview.net/forum?id=a68S… #supervised #visual #features

Absolutely gold article. Changed the way I see Layer Norm

You misread. There had been multiple LLM projects within FAIR for years. Some were open sourced as research prototypes (e.g. OPT175B, Galactica, BlenderBot...). In mid-2022, FAIR started a large LLM project called Zetta, which was still going in late 2022 when ChatGPT came out. A small group at FAIR-Paris was working on theorem proving. They needed an LLM for their own purpose and thought Zetta was too big and not ready. They developed their own model, which eventually became Llama-1. What happened internally between Zetta and Llama is somewhat similar to what just happened between DeepSeek and the big US players: a small team of talented folks innovated and beat the large teams.

Depth anything V2 one shots this problem btw. All it requires is an algorithm to create a coherent world via past imagery and depth calculations.

What's a vision model I can use to count toy pieces like this? GPT-4o tells me things like, "um, about 20", or counts incorrectly. Bonus points if it's easy to use via API and can work with plain english prompts

Another banger by @TheoMoutakanni : RayDINO, a DINO for chest X-ray. Excellent results on a ton of benchmarks with the frozen model, with great generalization and low bias. Check it out! arxiv.org/abs/2405.01469

Damn expectation-maximisation of a GMM got hands (it's the easiest algo in stat learning im just bad)

BRAG ABOUT SOMETHING YOU’RE PROUD OF ACCOMPLISHING IN 2023 ✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨✨

I don’t get why Apple hasn’t incorporated Whisper. They should be funding a team of, say, 5 researchers systematically iterating on it. Apple should have the world’s best voice-to-text and TTS models.

🚀 DinoV3 just became the new go-to backbone for geoloc! It outperforms CLIP-like models (SigLip2, finetuned StreetCLIP)… and that’s shocking 🤯 Why? CLIP models have an innate advantage — they literally learn place names + images. DinoV3 doesn’t.

Think before you speak: Training Language Models With Pause Tokens - Performing training and inference on LMs with a learnable pause token appended to the input prefix - Gains on 8 tasks, e,g, +18% on SQuAD arxiv.org/abs/2310.02226

We release AIMv2, the second iteration of the AIM family of large autoregressive vision encoders. This time we bring multimodality into the game 🔥 Paper: arxiv.org/abs/2411.14402 Repo: github.com/apple/ml-aim Model Gallery: huggingface.co/collections/a…

In AI research there is tremendous value in intuitions on what makes things work. In fact, this skill is what makes “yolo runs” successful, and can accelerate your team tremendously. However, there’s no track record on how good someone’s intuition is. A fun way to do this is “betting”, where researchers try to predict the results of an experiment, or whether an approach would ultimately be successful. When I was at Google Brain in 2022, I made a bet for what accuracy a 540B-parameter LLM would get on mate-in-one in chess after finetuning. I had great fun asking my friends to participate—their predictions ranged from 10% to 80% (I think it ended up being around 30%). I particularly enjoyed a few bets with @LiamFedus (now my manager at OpenAI). Back in the day when we were writing a paper on emergent abilities, we bet on whether he would be able to predict the final accuracy of a task based on the log-prob trends from smaller models, and I won that one. More recently, we had a bet on how much data would be needed for a model to reach a certain performance, and I lost that bet by an order of magnitude. It was a nice ego check for me. (Bro tip: if you bet a dinner, specify the price range before you lose) Having a track record holds you to be accountable for intuitions and helps you remember when you were wrong. The best researchers excite their peers about only a few things, and some of those things work well in a big way. You don’t want to be excited about everything, but then only a small portion of those things actually work. Finally, I think there is also a lot of value in correctly predicting that and research direction won't go well—these “negative bets” aren’t typically rewarded in today’s culture, but I believe there is a lot of value in saving your team time.

checkout scholar-inbox.com

Next week on Wednesday, February 7th, our Geo-Regional Asia Group is excited to welcome Timothée Darcet, PhD student, building large vision models at @Meta AI (FAIR) & @Inria to present "Vision Transformers need Registers." Learn more: cohere.com/events/c4ai-Timot…

DINOv2, a SOTA ViT trained by @Meta on 142 million images, is now part of 🤗 Transformers! It's one of the strongest vision backbones at the moment, so I created a tutorial on training a linear classifier on top of it for semantic segmentation, using DINOv2's frozen features 1/2

In case some of you were (like me) curious about this stat for AI conferences: here it is for ICLR2024

Gradient Checkpointing is the single most effective way of reducing GPU memory footprint. This thing is fantastic! Am I missing something, or is it that good?

Notable models that use non-imagenet norm are Dust3r, OpenIBL, many image matching models, and some (many?) remote sensing models. This is an issue when you create a fair codebase to benchmark multiple models (where ideally you can simply swap the model to compute the results).

Is there a good reason we use softmax losses in contrastive learning, instead of just doing MSE? ie L = ||xi-xi'||² - lambda sum_k ||xi-xk'||² I'd guess the optimization dynamics are maybe friendlier, but does anyone have a good pointer? Both for CLIP and SSL btw