Accepted papers at TMLR@TmlrPub
Joined March 2022
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, Lucas Beyer openreview.net/forum?id=4nPs…

How to train your ViT? Data, Augmentation, and Regularization in...

Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image...

openreview.net
2
41
291
ZerO Initialization: Initializing Neural Networks with only Zeros and Ones Jiawei Zhao, Florian Tobias Schaefer, Anima Anandkumar openreview.net/forum?id=1AxQ…

ZerO Initialization: Initializing Neural Networks with only Zeros...

Deep neural networks are usually initialized with random weights, with adequately selected initial variance to ensure stable signal propagation during training. However, selecting the appropriate...

openreview.net
1
16
125
DINOv2: Learning Robust Visual Features without Supervision Maxime Oquab, Timothée Darcet, Théo Moutakanni et al.. Action editor: Abhishek Kumar. openreview.net/forum?id=a68S… #supervised #visual #features

DINOv2: Learning Robust Visual Features without Supervision

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could...

openreview.net
1
17
109
38,303
A Simple Convergence Proof of Adam and Adagrad Alexandre Défossez, Leon Bottou, Francis Bach, Nicolas Usunier openreview.net/forum?id=ZPQh…

A Simple Convergence Proof of Adam and Adagrad

We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with bounded gradients....

openreview.net
20
111
Greedy Bayesian Posterior Approximation with Deep Ensembles Aleksei Tiulpin, Matthew B. Blaschko openreview.net/forum?id=P1Du…

Greedy Bayesian Posterior Approximation with Deep Ensembles

Ensembles of independently trained neural networks are a state-of-the-art approach to estimate predictive uncertainty in Deep Learning, and can be interpreted as an approximation of the posterior...

openreview.net
12
85
Emergent Abilities of Large Language Models Jason Wei, Yi Tay, Rishi Bommasani et al. openreview.net/forum?id=yzkS…

Emergent Abilities of Large Language Models

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that...

openreview.net
27
79
Modular Deep Learning Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, Edoardo Ponti. Action editor: Karthik Narasimhan. openreview.net/forum?id=z9Ek… #modular #modularity #hierarchical

Modular Deep Learning

Transfer learning has recently become the dominant paradigm of machine learning. Pre-trained models fine-tuned for downstream tasks achieve better performance with fewer labelled examples....

openreview.net
11
77
25,344
A geometrical connection between sparse and low-rank matrices and its application to manifold lea... Lawrence K. Saul openreview.net/forum?id=p8gn… #sparse #manifold #dimensional

A geometrical connection between sparse and low-rank matrices and...

We consider when a sparse nonnegative matrix $\mathbf{S}$ can be recovered, via an elementwise nonlinearity, from a real-valued matrix~$\mathbf{L}$ of significantly lower rank. Of particular...

openreview.net
1
13
75
17,080
Representation Alignment in Neural Networks Ehsan Imani, Wei Hu, Martha White openreview.net/forum?id=fLIW…

Representation Alignment in Neural Networks

It is now a standard for neural network representations to be trained on large, publicly available datasets, and used for new problems. The reasons for why neural network representations have been...

openreview.net
18
67
Understanding convolution on graphs via energies Francesco Di Giovanni, James Rowbottom, Benjamin Paul Chamberlain et al.. Action editor: Guillaume Rabusseau. openreview.net/forum?id=v5ew… #convolutions #graphs #convolutional

Understanding convolution on graphs via energies

Graph Neural Networks (GNNs) typically operate by message-passing, where the state of a node is updated based on the information received from its neighbours. Most message-passing models act as...

openreview.net
1
10
63
15,766
The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning Anders Johan Andreassen, Yasaman Bahri, Behnam Neyshabur, Rebecca Roelofs openreview.net/forum?id=Qs3E…

The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning

Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single...

openreview.net
12
61
Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections Gabriel Loaiza-Ganem, Brendan Leigh Ross, Rasa Hosseinzadeh, Anthony L. Caterini, Jesse C. Cresswell. Action editor: Serguei Barannikov. openreview.net/forum?id=a90W…

Deep Generative Models through the Lens of the Manifold Hypothesis:...

In recent years there has been increased interest in understanding the interplay between deep generative models (DGMs) and the manifold hypothesis. Research in this area focuses on understanding...

openreview.net
8
55
4,964
Structured Uncertainty in the Observation Space of Variational Autoencoders James Langley, Miguel Monteiro, Charles Jones, Nick Pawlowski, Ben Glocker openreview.net/forum?id=cxp7…

Structured Uncertainty in the Observation Space of Variational...

Variational autoencoders (VAEs) are a popular class of deep generative models with many variants and a wide range of applications. Improvements upon the standard VAE mostly focus on the modelling...

openreview.net
9
45
Self-Supervision is All You Need for Solving Rubik’s Cube Kyo Takano. Action editor: Marc Lanctot. openreview.net/forum?id=bnBe… #rubik #cube #deepcubea

Self-Supervision is All You Need for Solving Rubik’s Cube

Existing combinatorial search methods are often complex and require some level of expertise. This work introduces a simple and efficient deep learning method for solving combinatorial problems with...

openreview.net
11
47
199,337
Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent Hikaru Umeda, Hideaki Iiduka. Action editor: Ali Ramezani-Kebrya. openreview.net/forum?id=sbmp… #batch #sgd #gradient

Increasing Both Batch Size and Learning Rate Accelerates Stochastic...

The performance of mini-batch stochastic gradient descent (SGD) strongly depends on setting the batch size and learning rate to minimize the empirical loss in training the deep neural network. In...

openreview.net
1
16
47
3,452
TLDR: Twin Learning for Dimensionality Reduction Yannis Kalantidis, Carlos Eduardo Rosar Kos Lassance, Jon Almazán, Diane Larlus openreview.net/forum?id=86fh…

TLDR: Twin Learning for Dimensionality Reduction

Dimensionality reduction methods are unsupervised approaches which learn low-dimensional spaces where some properties of the initial space, typically the notion of “neighborhood”, are preserved....

openreview.net
9
43
Equivariant Mesh Attention Networks Sourya Basu, Jose Gallego-Posada, Francesco Viganò, James Rowbottom, Taco Cohen openreview.net/forum?id=3Iqq…

Equivariant Mesh Attention Networks

Equivariance to symmetries has proven to be a powerful inductive bias in deep learning research. Recent works on mesh processing have concentrated on various kinds of natural symmetries, including...

openreview.net
1
9
43
Wavelet Networks: Scale-Translation Equivariant Learning From Raw Time-Series David W. Romero, Erik J Bekkers, Jakub M. Tomczak, Mark Hoogendoorn. Action editor: Jeremias Sulam. openreview.net/forum?id=ga5S… #wavelet_networks #wavelet #transforms

Wavelet Networks: Scale-Translation Equivariant Learning From Raw...

Leveraging the symmetries inherent to specific data domains for the construction of equivariant neural networks has lead to remarkable improvements in terms of data efficiency and generalization....

openreview.net
10
40
10,736
A Note on "Assessing Generalization of SGD via Disagreement" Andreas Kirsch, Yarin Gal openreview.net/forum?id=oRP8…

A Note on "Assessing Generalization of SGD via Disagreement"

Several recent works find empirically that the average test error of deep neural networks can be estimated via the prediction disagreement of models, which does not require labels. In particular...

openreview.net
1
5
39
Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning Théo Vincent, Daniel Palenicek, Boris Belousov, Jan Peters, Carlo D'Eramo. Action editor: Pablo Castro. openreview.net/forum?id=Lt2H… #reinforcement #iterative #iterations

Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep...

The vast majority of Reinforcement Learning methods is largely impacted by the computation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn...

openreview.net
11
37
18,125
Finding and Fixing Spurious Patterns with Explanations Gregory Plumb, Marco Tulio Ribeiro, Ameet Talwalkar openreview.net/forum?id=whJP…

Finding and Fixing Spurious Patterns with Explanations

Image classifiers often use spurious patterns, such as “relying on the presence of a person to detect a tennis racket,” which do not generalize. In this work, we present an end-to-end pipeline for...

openreview.net
3
11
39
A Kernel Perspective on Behavioural Metrics for Markov Decision Processes Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, Mark Rowland. Action editor: Gergely Neu. openreview.net/forum?id=nHfP… #kernels #reinforcement #markov

A Kernel Perspective on Behavioural Metrics for Markov Decision...

We present a novel perspective on behavioural metrics for Markov decision processes via the use of positive definite kernels. We define a new metric under this lens that is provably equivalent to...

openreview.net
1
8
37
26,820
Causal Parrots: Large Language Models May Talk Causality But Are Not Causal Matej Zečević, Moritz Willig, Devendra Singh Dhami, Kristian Kersting. Action editor: Frederic Sala. openreview.net/forum?id=tv46… #causal #ai #inference

Causal Parrots: Large Language Models May Talk Causality But Are...

Some argue scale is all what is needed to achieve AI, covering even causal models. We make it clear that large language models (LLMs) cannot be causal and give reason onto why sometimes we might...

openreview.net
9
33
8,320
When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted ... Sean McGrath, Parth Mehta, Alexandra Zytek, Isaac Lage, Himabindu Lakkaraju. Action editor: Laurent Charlin. openreview.net/forum?id=pbs2… #predictive #predicting

When Does Uncertainty Matter?: Understanding the Impact of...

As machine learning (ML) models are increasingly being employed to assist human decision makers, it becomes critical to provide these decision makers with relevant inputs which can help them decide...

openreview.net
10
34
6,161
Towards fully covariant machine learning Soledad Villar, David W Hogg, Weichi Yao, George A Kevrekidis, Bernhard Schölkopf. Action editor: Jean Barbier. openreview.net/forum?id=gllU… #symmetries #symmetry #invariances

Towards fully covariant machine learning

Any representation of data involves arbitrary investigator choices. Because those choices are external to the data-generating process, each choice leads to an exact symmetry, corresponding to the...

openreview.net
7
38
6,560
Better Theory for SGD in the Nonconvex World Ahmed Khaled, Peter Richtárik. Action editor: Raman Arora. openreview.net/forum?id=AU4q… #optimal #convexity #sgd

Better Theory for SGD in the Nonconvex World

Large-scale nonconvex optimization problems are ubiquitous in modern machine learning, and among practitioners interested in solving them, Stochastic Gradient Descent (SGD) reigns supreme. We...

openreview.net
4
34
38,161
How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, Tanmoy Chakraborty. Action editor: Swarat Chaudhuri. openreview.net/forum?id=uHLD… #reasoning #attention #ontological

How to think step-by-step: A mechanistic understanding of...

Despite superior reasoning prowess demonstrated by Large Language Models (LLMs) with Chain-of-Thought (CoT) prompting, a lack of understanding prevails around the internal mechanisms of the models...

openreview.net
12
36
2,484
Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Superv... Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, Pascal Vincent. Action editor: Jinwoo Shin. openreview.net/forum?id=ZgXf… #imagenet #regulari

Guillotine Regularization: Why removing layers is needed to improve...

One unexpected technique that emerged in recent years consists in training a Deep Network (DN) with a Self-Supervised Learning (SSL) method, and using this network on downstream tasks but with its...

openreview.net
10
33
8,298
Diffusion Models for Constrained Domains Nic Fishman, Leo Klarner, Valentin De Bortoli, Emile Mathieu, Michael John Hutchinson. Action editor: Rianne van den Berg. openreview.net/forum?id=xuWT… #diffusion #denoising #riemannian

Diffusion Models for Constrained Domains

Denoising diffusion models are a novel class of generative algorithms that achieve state-of-the-art performance across a range of domains, including image generation and text-to-image tasks....

openreview.net
4
33
4,447
Layerwise Bregman Representation Learning of Neural Networks with Applications to Knowledge Disti... Ehsan Amid, Rohan Anil, Christopher Fifty, Manfred K Warmuth. Action editor: Stephan Mandt. openreview.net/forum?id=6dsv… #normalization #layerwise #trained

Layerwise Bregman Representation Learning of Neural Networks with...

We propose a new method for layerwise representation learning of a trained neural network that conforms to the non-linearity of the layer's transfer function. In particular, we form a Bregman...

openreview.net
9
36
7,249
Soft Diffusion: Score Matching with General Corruptions Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alex Dimakis, Peyman Milanfar. Action editor: Jonathan Scarlett. openreview.net/forum?id=W98r… #denoising #diffusion #diffusions

Soft Diffusion: Score Matching with General Corruptions

We define a broader family of corruption processes that generalizes previously known diffusion models. To reverse these general diffusions, we propose a new objective called Soft Score Matching....

openreview.net
1
7
35
17,641
VN-Transformer: Rotation-Equivariant Attention for Vector Neurons Serge Assaad, Carlton Downey, Rami Al-Rfou', Nigamaa Nayakanti, Benjamin Sapp openreview.net/forum?id=EiX2… #rotation #equivariance #neurons

VN-Transformer: Rotation-Equivariant Attention for Vector Neurons

Rotation equivariance is a desirable property in many practical applications such as motion forecasting and 3D perception, where it can offer benefits like sample efficiency, better generalization,...

openreview.net
7
34
4,436
Identifying latent distances with Finslerian geometry Alison Pouplin, David Eklund, Carl Henrik Ek, Søren Hauberg. Action editor: Bamdev Mishra. openreview.net/forum?id=Q2Gi… #geodesics #metrics #riemannian

Identifying latent distances with Finslerian geometry

Riemannian geometry provides us with powerful tools to explore the latent space of generative models while preserving the underlying structure of the data. The latent space can be equipped it with...

openreview.net
5
35
39,374
A Short Survey on Importance Weighting for Machine Learning Masanari Kimura, Hideitsu Hino. Action editor: Satoshi Hara. openreview.net/forum?id=IhXM… #importance #weights #supervised

A Short Survey on Importance Weighting for Machine Learning

Importance weighting is a fundamental procedure in statistics and machine learning that weights the objective function or probability distribution based on the importance of the instance in some...

openreview.net
8
35
5,351
Why should autoencoders work? Matthew Kvalheim, Eduardo Sontag. Action editor: Jeffrey Pennington. openreview.net/forum?id=uGVF… #autoencoders #decoding #homeomorphic

Why should autoencoders work?

Deep neural network autoencoders are routinely used computationally for model reduction. They allow recognizing the intrinsic dimension of data that lie in a $k$-dimensional subset $K$ of an input...

openreview.net
6
32
2,685
Concept-Driven Continual Learning Sin-Han Yang, Tuomas Oikarinen, Tsui-Wei Weng. Action editor: Andrew Lampinen. openreview.net/forum?id=HSW4… #forgetting #continual #memory

Concept-Driven Continual Learning

This paper introduces two novel solutions to the challenge of catastrophic forgetting in continual learning: Interpretability Guided Continual Learning (IG-CL) and Intrinsically Interpretable...

openreview.net
1
2
34
3,837
A Variational Perspective on Generative Flow Networks Heiko Zimmermann, Fredrik Lindsten, Jan-Willem van de Meent, Christian A Naesseth. Action editor: Jakub Tomczak. openreview.net/forum?id=AZ4G… #generative #flow #variational

A Variational Perspective on Generative Flow Networks

Generative flow networks (GFNs) are a class of probabilistic models for sequential sampling of composite objects, proportional to a target distribution that is defined in terms of an energy...

openreview.net
6
31
5,755
Diagnosing and Fixing Manifold Overfitting in Deep Generative Models Gabriel Loaiza-Ganem, Brendan Leigh Ross, Jesse C Cresswell, Anthony L. Caterini openreview.net/forum?id=0nEZ…

Diagnosing and Fixing Manifold Overfitting in Deep Generative Models

Likelihood-based, or explicit, deep generative models use neural networks to construct flexible high-dimensional densities. This formulation directly contradicts the manifold hypothesis, which...

openreview.net
5
33
Bayesian Optimization with Informative Covariance Afonso Eduardo, Michael U. Gutmann. Action editor: Pierre Alquier. openreview.net/forum?id=JwgV… #optimization #bayesian #prior

Bayesian Optimization with Informative Covariance

Bayesian optimization is a methodology for global optimization of unknown and expensive objectives. It combines a surrogate Bayesian regression model with an acquisition function to decide where to...

openreview.net
3
28
6,135
Neural Collapse: A Review on Modelling Principles and Generalization Vignesh Kothapalli. Action editor: Jeffrey Pennington. openreview.net/forum?id=QTXo… #classifier #generalization #deep

Neural Collapse: A Review on Modelling Principles and Generalization

Deep classifier neural networks enter the terminal phase of training (TPT) when training error reaches zero and tend to exhibit intriguing Neural Collapse (NC) properties. Neural collapse...

openreview.net
11
28
9,551
Decoder Denoising Pretraining for Semantic Segmentation Emmanuel Asiedu Brempong, Simon Kornblith, Ting Chen et al. openreview.net/forum?id=D3WI…

Decoder Denoising Pretraining for Semantic Segmentation

Semantic segmentation labels are expensive and time consuming to acquire. Hence, pretraining is commonly used to improve the label-efficiency of segmentation models. Typically, the encoder of a...

openreview.net
1
5
29
Holistic Evaluation of Language Models Percy Liang, Rishi Bommasani, Tony Lee et al.. Action editor: Karthik Narasimhan. openreview.net/forum?id=iO4L… #language #dialects #trustworthiness

Holistic Evaluation of Language Models

Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation...

openreview.net
1
7
29
15,714
Attending to Graph Transformers Luis Müller, Mikhail Galkin, Christopher Morris, Ladislav Rampášek. Action editor: Fredrik Johansson. openreview.net/forum?id=Hhbq… #graphs #graph #molecular

Attending to Graph Transformers

Recently, transformer architectures for graphs emerged as an alternative to established techniques for machine learning with graphs, such as (message-passing) graph neural networks. So far, they...

openreview.net
4
29
3,275
IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled ... Jay Gala, Pranjal A Chitale, A K Raghavan et al.. Action editor: W Ronny Huang. openreview.net/forum?id=vfT4… #corpus #multilingual #corpora

IndicTrans2: Towards High-Quality and Accessible Machine...

India has a rich linguistic landscape, with languages from 4 major language families spoken by over a billion people. 22 of these languages listed in the Constitution of India (referred to as...

openreview.net
9
28
14,610
The Vendi Score: A Diversity Evaluation Metric for Machine Learning Dan Friedman, Adji Bousso Dieng. Action editor: Antonio Vergari. openreview.net/forum?id=g97O… #diversity #diverse #similarity

The Vendi Score: A Diversity Evaluation Metric for Machine Learning

Diversity is an important criterion for many areas of machine learning (ML), including generative modeling and dataset curation. However, existing metrics for measuring diversity are often...

openreview.net
10
28
5,596
Auto-Lambda: Disentangling Dynamic Task Relationships Shikun Liu, Stephen James, Andrew Davison, Edward Johns openreview.net/forum?id=KKeC…

Auto-Lambda: Disentangling Dynamic Task Relationships

Understanding the structure of multiple related tasks allows for multi-task learning to improve the generalisation ability of one or all of them. However, it usually requires training each pairwise...

openreview.net
4
29
Predicting Out-of-Domain Generalization with Neighborhood Invariance Nathan Hoyen Ng, Neha Hulkund, Kyunghyun Cho, Marzyeh Ghassemi. Action editor: Vincent Dumoulin. openreview.net/forum?id=jYkW… #classifier #classification #generalization

Predicting Out-of-Domain Generalization with Neighborhood Invariance

Developing and deploying machine learning models safely depends on the ability to char- acterize and compare their abilities to generalize to new environments. Although recent work has proposed a...

openreview.net
9
29
13,233
Clustering units in neural networks: upstream vs downstream information Richard D Lange, David Rolnick, Konrad Kording openreview.net/forum?id=Euf7…

Clustering units in neural networks: upstream vs downstream...

It has been hypothesized that some form of "modular" structure in artificial neural networks should be useful for learning, compositionality, and generalization. However, defining and quantifying...

openreview.net
1
1
24
Not All Causal Inference is the Same Matej Zečević, Devendra Singh Dhami, Kristian Kersting. Action editor: Mingming Gong. openreview.net/forum?id=ySWQ… #causality #causal #inferences

Not All Causal Inference is the Same

Neurally-parameterized Structural Causal Models in the Pearlian notion to causality, referred to as NCM, were recently introduced as a step towards next-generation learning systems. However, said...

openreview.net
1
5
23
4,403
Temperature check: theory and practice for training models with softmax-cross-entropy losses Atish Agarwala, Samuel Stern Schoenholz, Jeffrey Pennington, Yann Dauphin. Action editor: Ruoyu Sun. openreview.net/forum?id=LBA2… #softmax #entropy #models
4
27
7,614
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback Stephen Casper, Xander Davies, Claudia Shi et al.. Action editor: Marcello Restelli. openreview.net/forum?id=bx24… #ai #reinforcement #language

Open Problems and Fundamental Limitations of Reinforcement Learning...

Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large...

openreview.net
6
25
2,770
Variational Classification: A Probabilistic Generalization of the Softmax Classifier Shehzaad Zuzar Dhuliawala, Mrinmaya Sachan, Carl Allen. Action editor: Frederic Sala. openreview.net/forum?id=EWv9… #softmax #classifiers #classification

Variational Classification: A Probabilistic Generalization of the...

We present a latent variable model for classification that provides a novel probabilistic interpretation of neural network softmax classifiers. We derive a variational objective to train the model,...

openreview.net
4
26
2,933
Patches Are All You Need? Asher Trockman, J Zico Kolter. Action editor: David Ha. openreview.net/forum?id=rAnB… #attention #convolutional #vision

Patches Are All You Need?

Although convolutional neural networks have been the dominant architecture for computer vision for many years, Vision Transformers (ViTs) have recently shown promise as an alternative. Subsequently...

openreview.net
4
27
6,688
Graph Neural Networks for Temporal Graphs: State of the Art, Open Challenges, and Opportunities Antonio Longa, Veronica Lachi, Gabriele Santin et al.. Action editor: Shinichi Nakajima. openreview.net/forum?id=pHCd… #temporal #graphs #graph

Graph Neural Networks for Temporal Graphs: State of the Art, Open...

Graph Neural Networks (GNNs) have become the leading paradigm for learning on (static) graph-structured data. However, many real-world systems are dynamic in nature, since the graph and node/edge...

openreview.net
4
26
4,680
Loss-to-Loss Prediction: Scaling Laws for All Datasets David Brandfonbrener, Nikhil Anand, Nikhil Vyas, Eran Malach, Sham M. Kakade. Action editor: Sarath Chandar. openreview.net/forum?id=1Avb… #predict #scaling #predicting

Loss-to-Loss Prediction: Scaling Laws for All Datasets

While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should change as we...

openreview.net
6
29
2,597
Metalearning Continual Learning Algorithms Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber. Action editor: Alessandro Sordoni. openreview.net/forum?id=IaUh… #forgetting #continual #learned

Metalearning Continual Learning Algorithms

General-purpose learning systems should improve themselves in open-ended fashion in ever-changing environments. Conventional learning algorithms for neural networks, however, suffer from...

openreview.net
2
28
1,048
Sparse Coding with Multi-layer Decoders using Variance Regularization Katrina Evtimova, Yann LeCun openreview.net/forum?id=4GuI…

Sparse Coding with Multi-layer Decoders using Variance Regularization

Sparse representations of images are useful in many computer vision applications. Sparse coding with an $l_1$ penalty and a learned linear dictionary requires regularization of the dictionary to...

openreview.net
5
27
The Robustness Limits of SoTA Vision Models to Natural Variation Mark Ibrahim, Quentin Garrido, Ari S. Morcos, Diane Bouchacourt. Action editor: Dumitru Erhan. openreview.net/forum?id=QhHL… #autoencoders #vision #supervised

The Robustness Limits of SoTA Vision Models to Natural Variation

Recent state-of-the-art vision models have introduced new architectures, learning paradigms, and larger pretraining data, leading to impressive performance on tasks such as classification. While...

openreview.net
6
26
4,035
Controllable Generative Modeling via Causal Reasoning Joey Bose, Ricardo Pio Monti, Aditya Grover openreview.net/forum?id=Z44Y… #NewPaper #PaperPost

Controllable Generative Modeling via Causal Reasoning

Deep latent variable generative models excel at generating complex, high-dimensional data, often exhibiting impressive generalization beyond the training distribution. However, many such models in...

openreview.net
4
24
9,532
Optimal Transport for Domain Adaptation through Gaussian Mixture Models Eduardo Fernandes Montesuma, Fred Maurice NGOLE MBOULA, Antoine Souloumiac. Action editor: Vincent Dumoulin. openreview.net/forum?id=DCAe… #adaptation #transport #unsupervised

Optimal Transport for Domain Adaptation through Gaussian Mixture...

Machine learning systems operate under the assumption that training and test data are sampled from a fixed probability distribution. However, this assumptions is rarely verified in practice, as the...

openreview.net
5
27
1,905
Statistical Mechanics of Min-Max Problems Yuma Ichikawa, Koji Hukushima. Action editor: Eduard Gorbunov. openreview.net/forum?id=qZqU… #adversarial #gans #optimal

Statistical Mechanics of Min-Max Problems

Min-max optimization problems, also known as saddle point problems, have attracted significant attention due to their applications in various fields, such as fair beamforming, generative...

openreview.net
4
26
3,391
Learning a Decision Tree Algorithm with Transformers Yufan Zhuang, Liyuan Liu, Chandan Singh, Jingbo Shang, Jianfeng Gao. Action editor: Yu Bai. openreview.net/forum?id=1Kzz… #metatree #tree #trees

Learning a Decision Tree Algorithm with Transformers

Decision trees are renowned for their ability to achieve high predictive performance while remaining interpretable, especially on tabular data. Traditionally, they are constructed through recursive...

openreview.net
9
24
1,796
Variational Inference on the Final-Layer Output of Neural Networks Yadi Wei, Roni Khardon. Action editor: Jasper Snoek. openreview.net/forum?id=mTOz… #rademacher #variational #bayesian

Variational Inference on the Final-Layer Output of Neural Networks

Traditional neural networks are simple to train but they typically produce overconfident predictions. In contrast, Bayesian neural networks provide good uncertainty quantification but optimizing...

openreview.net
2
25
2,585
High Fidelity Visualization of What Your Self-Supervised Representation Knows About Florian Bordes, Randall Balestriero, Pascal Vincent openreview.net/forum?id=urfW…
4
26
Pathologies of Predictive Diversity in Deep Ensembles Taiga Abe, E. Kelly Buchanan, Geoff Pleiss, John Patrick Cunningham. Action editor: Neil Houlsby. openreview.net/forum?id=TQfQ… #diversity #ensembles #diverse

Pathologies of Predictive Diversity in Deep Ensembles

Classic results establish that encouraging predictive diversity improves performance in ensembles of low-capacity models, e.g. through bagging or boosting. Here we demonstrate that these intuitions...

openreview.net
1
3
26
3,098
Weight Expansion: A New Perspective on Dropout and Generalization Gaojie Jin, Xinping Yi, Pengfei Yang, Lijun Zhang, Sven Schewe, Xiaowei Huang openreview.net/forum?id=w3z3…

Weight Expansion: A New Perspective on Dropout and Generalization

While dropout is known to be a successful regularization technique, insights into the mechanisms that lead to this success are still lacking. We introduce the concept of weight expansion, an...

openreview.net
6
26
Approximations to the Fisher Information Metric of Deep Generative Models for Out-Of-Distribution... Sam Dauncey, Christopher C. Holmes, Christopher Williams, Fabian Falck. Action editor: Daniel Roy. openreview.net/forum?id=Ecuw… #autoencoders #generative #likeli

Approximations to the Fisher Information Metric of Deep Generative...

Likelihood-based deep generative models such as score-based diffusion models and variational autoencoders are state-of-the-art machine learning models approximating high-dimensional distributions...

openreview.net
5
23
1,745
Approximating 1-Wasserstein Distance with Trees Makoto Yamada, Yuki Takezawa, Ryoma Sato, Han Bao, Zornitsa Kozareva, Sujith Ravi openreview.net/forum?id=Ig82…

Approximating 1-Wasserstein Distance with Trees

The Wasserstein distance, which measures the discrepancy between distributions, shows efficacy in various types of natural language processing and computer vision applications. One of the...

openreview.net
3
24
On the link between conscious function and general intelligence in humans and machines Arthur Juliani, Kai Arulkumaran, Shuntaro Sasai, Ryota Kanai openreview.net/forum?id=LTyq…

On the link between conscious function and general intelligence in...

In popular media, there is often a connection drawn between the advent of awareness in artificial agents and those same agents simultaneously achieving human or superhuman level intelligence. In...

openreview.net
10
23
Learning Graph Structure from Convolutional Mixtures Max Wasserman, Saurabh Sihag, Gonzalo Mateos, Alejandro Ribeiro. Action editor: Makoto Yamada. openreview.net/forum?id=OILb… #graphs #graph #deconvolution

Learning Graph Structure from Convolutional Mixtures

Machine learning frameworks such as graph neural networks typically rely on a given, fixed graph to exploit relational inductive biases and thus effectively learn from network data. However, when...

openreview.net
6
25
4,685
A probabilistic Taylor expansion with Gaussian processes Toni Karvonen, Jon Cockayne, Filip Tronarp, Simo Särkkä. Action editor: Roman Garnett. openreview.net/forum?id=2Tne… #gaussian #taylor #posterior

A probabilistic Taylor expansion with Gaussian processes

We study a class of Gaussian processes for which the posterior mean, for a particular choice of data, replicates a truncated Taylor expansion of any order. The data consist of derivative...

openreview.net
2
22
2,379
Are you using test log-likelihood correctly? Sameer Deshpande, Soumya Ghosh, Tin D. Nguyen, Tamara Broderick. Action editor: Michael Gutmann. openreview.net/forum?id=n2Yi… #likelihoods #likelihood #comparisons

Are you using test log-likelihood correctly?

Test log-likelihood is commonly used to compare different models of the same data or different approximate inference algorithms for fitting the same probabilistic model. We present simple examples...

openreview.net
3
22
10,241
A Unified Survey on Anomaly, Novelty, Open-Set, and Out of-Distribution Detection: Solutions and ... Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks et al. openreview.net/forum?id=aRtj… #NewPaper #PaperPost

A Unified Survey on Anomaly, Novelty, Open-Set, and Out...

Machine learning models often encounter samples that are diverged from the training distribution. Failure to recognize an out-of-distribution (OOD) sample, and consequently assign that sample to an...

openreview.net
7
24
On the Adversarial Robustness of Vision Transformers Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, Cho-Jui Hsieh openreview.net/forum?id=lE7K…

On the Adversarial Robustness of Vision Transformers

Following the success in advancing natural language processing and understanding, transformers are expected to bring revolutionary changes to computer vision. This work provides a comprehensive...

openreview.net
4
23
Planning with Consistency Models for Model-Based Offline Reinforcement Learning Guanquan Wang, Takuya Hiraoka, Yoshimasa Tsuruoka. Action editor: Manuel Haussmann. openreview.net/forum?id=TuAC… #reinforcement #planning #iterative

Planning with Consistency Models for Model-Based Offline...

This paper introduces consistency models to the problem of sequential decision-making. Previous work applying diffusion models to planning within a model-based reinforcement learning framework...

openreview.net
1
4
23
2,620
UnIVAL: Unified Model for Image, Video, Audio and Language Tasks Mustafa Shukor, Corentin Dancette, Alexandre Rame, Matthieu Cord. Action editor: Stefan Lee. openreview.net/forum?id=4ufl… #multimodal #models #unified

UnIVAL: Unified Model for Image, Video, Audio and Language Tasks

Large Language Models (LLMs) have made the ambitious quest for generalist agents significantly far from being a fantasy. A key hurdle for building such general models is the diversity and...

openreview.net
7
21
2,118
On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning Philipp Becker, Gerhard Neumann openreview.net/forum?id=UQXd…

On Uncertainty in Deep State Space Models for Model-Based...

Improved state space models, such as Recurrent State Space Models (RSSMs), are a key factor behind recent advances in model-based reinforcement learning (RL). Yet, despite their empirical success,...

openreview.net
3
21
Empirical Study on Optimizer Selection for Out-of-Distribution Generalization Hiroki Naganuma, Kartik Ahuja, Shiro Takagi et al.. Action editor: Robert Gower. openreview.net/forum?id=ipe0… #distributional #classification #hyperparameters

Empirical Study on Optimizer Selection for Out-of-Distribution...

Modern deep learning systems do not generalize well when the test data distribution is slightly different to the training data distribution. While much promising work has been accomplished to...

openreview.net
5
19
11,790
DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents Kushagra Pandey, Avideep Mukherjee, Piyush Rai, Abhishek Kumar openreview.net/forum?id=ygoN… #NewPaper #PaperPost

DiffuseVAE: Efficient, Controllable and High-Fidelity Generation...

Diffusion probabilistic models have been shown to generate state-of-the-art results on several competitive image synthesis benchmarks but lack a low-dimensional, interpretable latent space, and are...

openreview.net
3
22
The Stack: 3 TB of permissively licensed source code Denis Kocetkov, Raymond Li, Loubna Ben allal et al.. Action editor: Swarat Chaudhuri. openreview.net/forum?id=pxpb… #text2code #bigcode #dataset

The Stack: 3 TB of permissively licensed source code

Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation. To...

openreview.net
1
21
5,390
Comparative Generalization Bounds for Deep Neural Networks Tomer Galanti, Liane Galanti, Ido Ben-Shaul. Action editor: Yunhe Wang. openreview.net/forum?id=162T… #deep #depth #depths

Comparative Generalization Bounds for Deep Neural Networks

In this work, we investigate the generalization capabilities of deep neural networks. We introduce a novel measure of the effective depth of neural networks, defined as the first layer at which...

openreview.net
5
20
6,828
Pre-trained Hypergraph Convolutional Neural Networks with Self-supervised Learning Yihe Deng, Ruochi Zhang, Pan Xu, Jian Ma, Quanquan Gu. Action editor: Ying Wei. openreview.net/forum?id=0VWX… #hypergraph #hypergraphs #chromatin

Pre-trained Hypergraph Convolutional Neural Networks with...

Hypergraphs are powerful tools for modeling complex interactions across various domains, including biomedicine. However, learning meaningful node representations from hypergraphs remains a...

openreview.net
4
21
1,922
Improved baselines for vision-language pre-training Enrico Fini, Pietro Astolfi, Adriana Romero-Soriano, Jakob Verbeek, Michal Drozdzal. Action editor: Hanwang Zhang. openreview.net/forum?id=a7nv… #multimodal #regularization #augmentation

Improved baselines for vision-language pre-training

Contrastive learning has emerged as an efficient framework to learn multimodal representations. CLIP, a seminal work in this area, achieved impressive results by training on paired image-text data ...

openreview.net
6
21
3,515
Memorisation in Machine Learning: A Survey of Results Dmitrii Usynin, Moritz Knolle, Georgios Kaissis. Action editor: Kamalika Chaudhuri. openreview.net/forum?id=HVWO… #privacy #adversarial #memorisation

Memorisation in Machine Learning: A Survey of Results

Quantifying the impact of individual data samples on machine learning models is an open research problem. This is particularly relevant when complex and high-dimensional relationships have to be...

openreview.net
5
21
3,628
Semantic Representations of Mathematical Expressions in a Continuous Vector Space Neeraj Gangwar, Nickvash Kani. Action editor: Yonatan Bisk. openreview.net/forum?id=EWPA… #embeddings #expressions #representing

Semantic Representations of Mathematical Expressions in a...

Mathematical notation makes up a large portion of STEM literature, yet finding semantic representations for formulae remains a challenging problem. Because mathematical notation is precise, and its...

openreview.net
5
22
2,074
UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed... Francisca Vasconcelos, Bobby He, Nalini M Singh, Yee Whye Teh. Action editor: Matthew Blaschko. openreview.net/forum?id=jdGM… #accuracy #deep #cnn

UncertaINR: Uncertainty Quantification of End-to-End Implicit...

Implicit neural representations (INRs) have achieved impressive results for scene reconstruction and computer graphics, where their performance has primarily been assessed on reconstruction...

openreview.net
3
20
10,398
A Measure of the Complexity of Neural Representations based on Partial Information Decomposition David Alexander Ehrlich, Andreas Christian Schneider, Viola Priesemann et al.. Action editor: Jean Barbier. openreview.net/forum?id=R8TU… #complexity #neurons #repres

A Measure of the Complexity of Neural Representations based on...

In neural networks, task-relevant information is represented jointly by groups of neurons. However, the specific way in which this mutual information about the classification label is distributed...

openreview.net
13
20
3,447
Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error Ondrej Bohdal, Yongxin Yang, Timothy Hospedales. Action editor: Yingzhen Li. openreview.net/forum?id=R2hU… #calibration #prediction #optimise

Meta-Calibration: Learning of Model Calibration Using...

Calibration of neural networks is a topical problem that is becoming more and more important as neural networks increasingly underpin real-world applications. The problem is especially noticeable...

openreview.net
3
20
5,187
Generalizability of Adversarial Robustness Under Distribution Shifts Kumail Alhamoud, Hasan Abed Al Kader Hammoud, Motasem Alfarra, Bernard Ghanem. Action editor: Gang Niu. openreview.net/forum?id=XNFo… #adversarial #robustness #robust

Generalizability of Adversarial Robustness Under Distribution Shifts

Recent progress in empirical and certified robustness promises to deliver reliable and deployable Deep Neural Networks (DNNs). Despite that success, most existing evaluations of DNN robustness have...

openreview.net
9
19
5,897
Out-of-Distribution Optimality of Invariant Risk Minimization Shoji Toyota, Kenji Fukumizu. Action editor: Jeffrey Pennington. openreview.net/forum?id=pWsf… #minimization #minimizes #minimize

Out-of-Distribution Optimality of Invariant Risk Minimization

Deep Neural Networks often inherit spurious correlations embedded in training data and hence may fail to generalize to unseen domains, which have different distributions from the domain to provide...

openreview.net
4
21
2,871
Dr-Fairness: Dynamic Data Ratio Adjustment for Fair Training on Real and Generated Data Yuji Roh, Weili Nie, De-An Huang, Steven Euijong Whang, Arash Vahdat, Anima Anandkumar. Action editor: Yingzhen Li. openreview.net/forum?id=TyBd… #sampling #fairness #unfairne

Dr-Fairness: Dynamic Data Ratio Adjustment for Fair Training on...

Fair visual recognition has become critical for preventing demographic disparity. A major cause of model unfairness is the imbalanced representation of different groups in training data. Recently...

openreview.net
5
20
11,980
Contrastive Search Is What You Need For Neural Text Generation Yixuan Su, Nigel Collier. Action editor: Matthew Blaschko. openreview.net/forum?id=GbkW… #decoding #nlp #contrastive

Contrastive Search Is What You Need For Neural Text Generation

Generating text with autoregressive language models (LMs) is of great importance to many natural language processing (NLP) applications. Previous solutions for this task often produce text that...

openreview.net
6
21
3,120
Catastrophic overfitting can be induced with discriminative non-robust features Guillermo Ortiz-Jimenez, Pau de Jorge, Amartya Sanyal et al.. Action editor: Jakub Tomczak. openreview.net/forum?id=10hC… #adversarial #overfitting #robust

Catastrophic overfitting can be induced with discriminative...

Adversarial training (AT) is the de facto method for building robust neural networks, but it can be computationally expensive. To mitigate this, fast single-step attacks can be used, but this may...

openreview.net
3
21
6,621
A Generalist Agent Scott Reed, Konrad Zolna, Emilio Parisotto et al. openreview.net/forum?id=1ikK…
1
21
On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization Dongruo Zhou, Jinghui Chen, Yuan Cao, Ziyan Yang, Quanquan Gu. Action editor: Peter Richtárik. openreview.net/forum?id=Gh0c… #gradient #adaptive #nonconvex

On the Convergence of Adaptive Gradient Methods for Nonconvex...

Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly studied. In this...

openreview.net
3
20
3,293
Integrating Bayesian Network Structure into Residual Flows and Variational Autoencoders Jacobie Mouton, Rodney Stephen Kroon. Action editor: George Papamakarios. openreview.net/forum?id=OsKX… #autoencoders #generative #flow

Integrating Bayesian Network Structure into Residual Flows and...

Deep generative models have become more popular in recent years due to their scalability and representation capacity. Unlike probabilistic graphical models, they typically do not incorporate...

openreview.net
3
21
4,526
Learned feature representations are biased by complexity, learning order, position, and more Andrew Kyle Lampinen, Stephanie C.Y. Chan, Katherine Hermann. Action editor: Quanshi Zhang. openreview.net/forum?id=aY2n… #representations #representation #features

Learned feature representations are biased by complexity, learning...

Representation learning, and interpreting learned representations, are key areas of focus in machine learning and neuroscience. Both fields generally use representations as a means to understand or...

openreview.net
1
3
22
1,942
Unsupervised Dense Information Retrieval with Contrastive Learning Gautier Izacard, Mathilde Caron, Lucas Hosseini et al. openreview.net/forum?id=jKN1…

Unsupervised Dense Information Retrieval with Contrastive Learning

Recently, information retrieval has seen the emergence of dense retrievers, using neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained...

openreview.net
6
21
Sparse MoEs meet Efficient Ensembles James Urquhart Allingham, Florian Wenzel, Zelda E Mariet et al. openreview.net/forum?id=i0ZM…
3
21
Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits Daniel Morales-Brotons, Thijs Vogels, Hadrien Hendrikx. Action editor: Atsushi Nitanda. openreview.net/forum?id=2M9C… #weights #sgd #regularization

Exponential Moving Average of Weights in Deep Learning: Dynamics...

Weight averaging of Stochastic Gradient Descent (SGD) iterates is a popular method for training deep learning models. While it is often used as part of complex training pipelines to improve...

openreview.net
3
20
1,972