Language Controls More Than Top-Down Attention: Modulating Bottom-Up Visual Processing with Referring Expressions
How to best integrate linguistic and perceptual processing in multimodal tasks is an important open problem. In this work we argue that the common technique of using language to direct visual attention over high-level visual features may not be optimal. Using language throughout the bottom-up visual pathway, going from pixels to high-level features, may be necessary. Our experiments on several English referring expression datasets show significant improvements when language is used to control the filters for bottom-up visual processing in addition to top-down attention.
Code (0)
등록된 구현이 없습니다.
Tasks
Referring ExpressionSimilar Papers 제목 키워드 기반
Component Ablation for Efficient Hybrid Language Model Architectures: Performance, Resilience, and Compression Implications
Hybrid language models combine softmax attention with linear-time sequence mechanisms such as state-space or linear-attention layers, but the functional contribution of each component type remains insufficiently characte…
Representation as a Bottleneck for Mechanistic Interpretability: The Manifestation Unit Protocol
Mechanistic interpretability has produced a rich inventory of component-level analyses that characterise what neural-network components encode and how they interact. Their outputs, however, are not easily reusable: selec…
The Attentional White Bear Effect in Transformer Language Models
Instruction-based suppression is widely used to prevent language models from generating prohibited content, yet it remains unclear whether suppression reduces internal representation or merely suppresses expression. We i…
LILA: Language-Informed Latent Actions
We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration. LILA falls under the shared autonomy paradigm: in addition to provid…
Imitation LearningSemantic Characteristics of Schizophrenic Speech
Natural language processing tools are used to automatically detect disturbances in transcribed speech of schizophrenia inpatients who speak Hebrew. We measure topic mutation over time and show that controls maintain more…