Accurate Word Representations with Universal Visual Guidance
Word representation is a fundamental component in neural language understanding models. Recently, pre-trained language models (PrLMs) offer a new performant method of contextualized word representations by leveraging the sequence-level context for modeling. Although the PrLMs generally give more accurate contextualized word representations than non-contextualized models do, they are still subject to a sequence of text contexts without diverse hints for word representation from multimodality. This paper thus proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. In detail, we build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. The texts and paired images are encoded in parallel, followed by an attention layer to integrate the multimodal representations. We show that the method substantially improves the accuracy of disambiguation. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNatural Language UnderstandingTranslationSimilar Papers 제목 키워드 기반
Is ``Universal Syntax'' Universally Useful for Learning Distributed Word Representations?
WIPES: Wavelet-based Visual Primitives
Pursuing a continuous visual representation that offers flexible frequency modulation and fast rendering speed has recently garnered increasing attention in the fields of 3D vision and graphics. However, existing represe…
Novel View SynthesisMultilingual Transformer Encoders: a Word-Level Task-Agnostic Evaluation
Some Transformer-based models can perform cross-lingual transfer learning: those models can be trained on a specific task in one language and give relatively good results on the same task in another language, despite hav…
Cross-Lingual TransferTransfer LearningMetaUAS: Universal Anomaly Segmentation with One-Prompt Meta-Learning
Zero- and few-shot visual anomaly segmentation relies on powerful vision-language models that detect unseen anomalies using manually designed textual prompts. However, visual representations are inherently independent of…
Anomaly DetectionAnomaly SegmentationMeta-LearningSegmentation+1Unsupervised Learning of Word-Sequence Representations from Scratch via Convolutional Tensor Decomposition
Unsupervised text embeddings extraction is crucial for text understanding in machine learning. Word2Vec and its variants have received substantial success in mapping words with similar syntactic or semantic meaning to ve…
Dictionary LearningSentenceTensor Decomposition