Modulating and attending the source image during encoding improves Multimodal Translation
We propose a new and fully end-to-end approach for multimodal translation where the source text encoder modulates the entire visual input processing using conditional batch normalization, in order to compute the most informative image features for our task. Additionally, we propose a new attention mechanism derived from this original idea, where the attention model for the visual input is conditioned on the source text encoder representations. In the paper, we detail our models as well as the image analysis pipeline. Finally, we report experimental results. They are, as far as we know, the new state of the art on three different test sets.
Code (1)
Tasks
TranslationSimilar Papers 제목 키워드 기반
Block Modulating Video Compression: An Ultra Low Complexity Image Compression Encoder for Resource Limited Platforms
We consider the image and video compression on resource limited platforms. An ultra low-cost image encoder, named Block Modulating Video Compression (BMVC) with an encoding complexity ${\cal O}(1)$ is proposed to be impl…
DecoderImage CompressionQuantizationVideo CompressionUntwisting RoPE: Frequency Control for Shared Attention in DiTs
Positional encodings are essential to transformer-based generative models, yet their behavior in multimodal and attention-sharing settings is not fully understood. In this work, we present a principled analysis of Rotary…
Semantic SimilarityStyle TransferContextual Position Encoding: Learning to Count What's Important
The attention mechanism is a critical component of Large Language Models (LLMs) that allows tokens in a sequence to interact with each other, but is order-invariant. Incorporating position encoding (PE) makes it possible…
Language ModelingLanguage ModellingPositionSentenceMulti-Modulation Network for Audio-Visual Event Localization
We study the problem of localizing audio-visual events that are both audible and visible in a video. Existing works focus on encoding and aligning audio and visual features at the segment level while neglecting informati…
audio-visual event localizationAction-Attending Graphic Neural Network
The motion analysis of human skeletons is crucial for human action recognition, which is one of the most active topics in computer vision. In this paper, we propose a fully end-to-end action-attending graphic neural netw…
Action AnalysisAction RecognitionAttributeSkeleton Based Action Recognition+1