Learning Spectral-Decomposed Tokens for Domain Generalized Semantic Segmentation
The rapid development of Vision Foundation Model (VFM) brings inherent out-domain generalization for a variety of down-stream tasks. Among them, domain generalized semantic segmentation (DGSS) holds unique challenges as the cross-domain images share common pixel-wise content information but vary greatly in terms of the style. In this paper, we present a novel Spectral-dEcomposed Token (SET) learning framework to advance the frontier. Delving into further than existing fine-tuning token & frozen backbone paradigm, the proposed SET especially focuses on the way learning style-invariant features from these learnable tokens. Particularly, the frozen VFM features are first decomposed into the phase and amplitude components in the frequency space, which mainly contain the information of content and style, respectively, and then separately processed by learnable tokens for task-specific information extraction. After the decomposition, style variation primarily impacts the token-based feature enhancement within the amplitude branch. To address this issue, we further develop an attention optimization method to bridge the gap between style-affected representation and static tokens during inference. Extensive cross-domain experiments show its state-of-the-art performance.
Code (1)
Tasks
Domain GeneralizationSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SIEFormer: Spectral-Interpretable and -Enhanced Transformer for Generalized Category Discovery
This paper presents a novel approach, Spectral-Interpretable and -Enhanced Transformer (SIEFormer), which leverages spectral analysis to reinterpret the attention mechanism within Vision Transformer (ViT) and enhance fea…
SoRA: Singular Value Decomposed Low-Rank Adaptation for Domain Generalizable Representation Learning
Domain generalization (DG) aims to adapt a model using one or multiple source domains to ensure robust performance in unseen target domains. Recently, Parameter-Efficient Fine-Tuning (PEFT) of foundation models has shown…
Domain AdaptationDomain Generalizationobject-detectionObject Detection+3SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learning
Domain generalization (DG) aims to adapt a model using one or multiple source domains to ensure robust performance in unseen target domains. Recently, Parameter-Efficient Fine-Tuning (PEFT) of foundation models has s…
Domain Generalizationparameter-efficient fine-tuningRepresentation LearningSemantic SegmentationDegradation Frequency Curve: An Explicit Frequency-Quantified Representation for All-in-One Image Restoration
A fundamental difficulty in all-in-one blind image restoration is that degradation is usually treated as an implicit factor hidden in degraded-to-clean mapping, rather than as an explicit object that can be measured and …
Image RestorationA spectral mixture representation of isotropic kernels to generalize random Fourier features
Rahimi and Recht (2007) introduced the idea of decomposing positive definite shift-invariant kernels by randomly sampling from their spectral distribution. This famous technique, known as Random Fourier Features (RFF), i…
Gaussian Processes