Towards Modality Transferable Visual Information Representation with Optimal Model Compression
Compactly representing the visual signals is of fundamental importance in various image/video-centered applications. Although numerous approaches were developed for improving the image and video coding performance by removing the redundancies within visual signals, much less work has been dedicated to the transformation of the visual signals to another well-established modality for better representation capability. In this paper, we propose a new scheme for visual signal representation that leverages the philosophy of transferable modality. In particular, the deep learning model, which characterizes and absorbs the statistics of the input scene with online training, could be efficiently represented in the sense of rate-utility optimization to serve as the enhancement layer in the bitstream. As such, the overall performance can be further guaranteed by optimizing the new modality incorporated. The proposed framework is implemented on the state-of-the-art video coding standard (i.e., versatile video coding), and significantly better representation capability has been observed based on extensive evaluations.
Code (0)
등록된 구현이 없습니다.
Tasks
Model CompressionPhilosophySimilar Papers 제목 키워드 기반
Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition
Despite the recent achievements made in the multi-modal emotion recognition task, two problems still exist and have not been well investigated: 1) the relationship between different emotion categories are not utilized, w…
Emotion RecognitionMultimodal Emotion RecognitionWord EmbeddingsDual Modality Prompt Tuning for Vision-Language Pre-Trained Model
With the emergence of large pre-trained vison-language model like CLIP, transferable representations can be adapted to a wide range of downstream tasks via prompt tuning. Prompt tuning tries to probe the beneficial infor…
General KnowledgeLanguage ModellingVisual Prompt TuningVT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts
Contrastive Language-Image Pre-training (CLIP) has drawn increasing attention recently for its transferable visual representation learning. However, due to the semantic gap within datasets, CLIP's pre-trained image-text …
Language ModellingRepresentation LearningZero-Shot LearningUnsupervised Modality-Transferable Video Highlight Detection with Representation Activation Sequence Learning
Identifying highlight moments of raw video materials is crucial for improving the efficiency of editing videos that are pervasive on internet platforms. However, the extensive work of manually labeling footage has create…
Contrastive LearningHighlight DetectionLeveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning
Audio-visual speech recognition (AVSR) has gained remarkable success for ameliorating the noise-robustness of speech recognition. Mainstream methods focus on fusing audio and visual inputs to obtain modality-invariant re…
Audio-Visual Speech Recognitionreinforcement-learningReinforcement Learning (RL)speech-recognition+2