Enhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learning
AuthorsEtai Littwin, Vimal Thilak, Anand Gopalakrishnan
AuthorsEtai Littwin, Vimal Thilak, Anand Gopalakrishnan
This paper was accepted at the Self-Supervised Learning - Theory and Practice (SSLTP) Workshop at NeurIPS 2024.
Image-based Joint-Embedding Predictive Architecture (IJEPA) offers an attractive alternative to Masked Autoencoder (MAE) for representation learning using the Masked Image Modeling framework. IJEPA drives representations to capture useful semantic information by predicting in latent rather than input space. However, IJEPA relies on carefully designed context and target windows to avoid representational collapse. The encoder modules in IJEPA cannot adaptively modulate the type of predicted and/or target features based on the feasibility of the masked prediction task, as they are not given sufficient information of both context and targets. Based on the intuition that in natural images, information has a strong spatial bias, with spatially local regions being highly predictive of one another compared to distant ones, we condition the target encoder and context encoder modules in IJEPA with positions of context and target windows respectively. Our "conditional" encoders show performance gains on several image classification benchmark datasets, improved robustness to context window size, and sample-efficiency during pretraining.
April 14, 2025research area Computer Vision, research area Methods and AlgorithmsWorkshop at ICLR
This paper was accepted at the Workshop on Foundation Models in the Wild at ICLR 2025.
Visual understanding is inherently contextual - what we focus on in an image depends on the task at hand. For instance, given an image of a person holding a bouquet of flowers, we may focus on either the person such as their clothing, or the type of flowers, depending on the context of interest. Yet, most existing image encoding paradigms represent an image as...
March 14, 2025research area Speech and Natural Language Processingconference ICASSP
Speech foundation models, such as HuBERT and its variants, are pre-trained on large amounts of unlabeled speech data and then used for a range of downstream tasks. These models use a masked prediction objective, where the model learns to predict information about masked input segments from the unmasked context. The choice of prediction targets in this framework impacts their performance on downstream tasks. For instance, models pre-trained with...