Latent Space Energy-based Model for Fine-grained Open Set Recognition
Fine-grained open-set recognition (FineOSR) aims to recognize images belonging to classes with subtle appearance differences while rejecting images of unknown classes. A recent trend in OSR shows the benefit of generative models to discriminative unknown detection. As a type of generative model, energy-based models (EBM) are the potential for hybrid modeling of generative and discriminative tasks. However, most existing EBMs suffer from density estimation in high-dimensional space, which is critical to recognizing images from fine-grained classes. In this paper, we explore the low-dimensional latent space with energy-based prior distribution for OSR in a fine-grained visual world. Specifically, based on the latent space EBM, we propose an attribute-aware information bottleneck (AIB), a residual attribute feature aggregation (RAFA) module, and an uncertainty-based virtual outlier synthesis (UVOS) module to improve the expressivity, granularity, and density of the samples in fine-grained classes, respectively. Our method is flexible to take advantage of recent vision transformers for powerful visual classification and generation. The method is validated on both fine-grained and general visual classification datasets while preserving the capability of generating photo-realistic fake images with high resolution.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeDensity EstimationOpen Set LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Is CLIP the main roadblock for fine-grained open-world perception?
Modern applications increasingly demand flexible computer vision models that adapt to novel concepts not encountered during training. This necessity is pivotal in emerging domains like extended reality, robotics, and aut…
Autonomous DrivingNovel ConceptsObjectobject-detection+4Prototype Fission: Closing Set for Robust Open-set Semi-supervised Learning
Semi-supervised Learning (SSL) has been proven vulnerable to out-of-distribution (OOD) samples in realistic large-scale unsupervised datasets due to over-confident pseudo-labeling OODs as in-distribution (ID). A key unde…
DiversitySplatent: Splatting Diffusion Latents for Novel View Synthesis
Radiance field representations have recently been explored in the latent space of VAEs that are commonly used by diffusion models. This direction offers efficient rendering and seamless integration with diffusion-based p…
Novel View Synthesis3D ReconstructionEnhancing Energy-Awareness in Deep Learning through Fine-Grained Energy Measurement
With the increasing usage, scale, and complexity of Deep Learning (DL) models, their rapidly growing energy consumption has become a critical concern. Promoting green development and energy awareness at different granula…
Paired Cross-Modal Data Augmentation for Fine-Grained Image-to-Text Retrieval
This paper investigates an open research problem of generating text-image pairs to improve the training of fine-grained image-to-text cross-modal retrieval task, and proposes a novel framework for paired data augmentatio…
Cross-Modal RetrievalData AugmentationImage to textImage-to-Text Retrieval+2