VTAMIQ: Transformers for Attention Modulated Image Quality Assessment
Following the major successes of self-attention and Transformers for image analysis, we investigate the use of such attention mechanisms in the context of Image Quality Assessment (IQA) and propose a novel full-reference IQA method, Vision Transformer for Attention Modulated Image Quality (VTAMIQ). Our method achieves competitive or state-of-the-art performance on the existing IQA datasets and significantly outperforms previous metrics in cross-database evaluations. Most patch-wise IQA methods treat each patch independently; this partially discards global information and limits the ability to model long-distance interactions. We avoid this problem altogether by employing a transformer to encode a sequence of patches as a single global representation, which by design considers interdependencies between patches. We rely on various attention mechanisms -- first with self-attention within the Transformer, and second with channel attention within our difference modulation network -- specifically to reveal and enhance the more salient features throughout our architecture. With large-scale pre-training for both classification and IQA tasks, VTAMIQ generalizes well to unseen sets of images and distortions, further demonstrating the strength of transformer-based networks for vision modelling.
Code (1)
Tasks
Image Quality AssessmentMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
Studies on self-supervised visual representation learning (SSL) improve encoder backbones to discriminate training samples without labels. While CNN encoders via SSL achieve comparable recognition performance to those vi…
image-classificationImage Classificationobject-detectionObject Detection+2Revitalizing CNN Attention via Transformers in Self-Supervised Visual Representation Learning
Studies on self-supervised visual representation learning (SSL) improve encoder backbones to discriminate training samples without labels. While CNN encoders via SSL achieve comparable recognition performance to those vi…
image-classificationImage Classificationobject-detectionObject Detection+2A Unified Geometric Field Theory Framework for Transformers: From Manifold Embeddings to Kernel Modulation
The Transformer architecture has achieved tremendous success in natural language processing, computer vision, and scientific computing through its self-attention mechanism. However, its core components-positional encodin…
Band-Attention Modulated RetNet for Face Forgery Detection
The transformer networks are extensively utilized in face forgery detection due to their scalability across large datasets.Despite their success, transformers face challenges in balancing the capture of global context, w…
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
Co-modulated GANs link image-conditional GANs and unconditional modulated models to address large-scale image completion tasks. Co-modulation brings stochastic and conditional style representations together. To improve e…
Image-to-Image TranslationTranslation