Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations

2018-06-08 · ICML 2018 7 · Ashwin Kalyan, Stefan Lee, Anitha Kannan, Dhruv Batra

Many structured prediction problems (particularly in vision and language domains) are ambiguous, with multiple outputs being correct for an input - e.g. there are many ways of describing an image, multiple ways of translating a sentence; however, exhaustively annotating the applicability of all possible outputs is intractable due to exponentially large output spaces (e.g. all English sentences). In practice, these problems are cast as multi-class prediction, with the likelihood of only a sparse set of annotations being maximized - unfortunately penalizing for placing beliefs on plausible but unannotated outputs. We make and test the following hypothesis - for a given input, the annotations of its neighbors may serve as an additional supervisory signal. Specifically, we propose an objective that transfers supervision from neighboring examples. We first study the properties of our developed method in a controlled toy setup before reporting results on multi-label classification and two image-grounded sequence modeling tasks - captioning and question generation. We evaluate using standard task-specific metrics and measures of output diversity, finding consistent improvements over standard maximum likelihood training and other baselines.

📄 PDF Abstract BibTeX arXiv:1806.02934

Code (0)

등록된 구현이 없습니다.

Tasks

DiversityMulti-Label Classification MUlTI-LABEL-ClASSIFICATION Question Generation Question-GenerationSentenceStructured Prediction

Do Neural Network Cross-Modal Mappings Really Bridge Modalities?

2018-05-19 · ACL 2018 7 · Guillem Collell, Marie-Francine Moens

Feed-forward networks are widely used in cross-modal applications to bridge modalities by mapping distributed vectors of one modality to the other, or to a shared space. The predicted vectors are then used to perform e.g…

Diversity Retrieval

Multimodal Graph Learning for Generative Tasks

2023-10-11 · Minji Yoon, Jing Yu Koh, Bryan Hooi, Ruslan Salakhutdinov

Multimodal learning combines multiple data modalities, broadening the types and complexity of data our models can utilize: for example, from plain text to image-caption pairs. Most multimodal learning algorithms focus on…

Graph Learning Text Generation

Don't Lose Yourself: Boosting Multimodal Recommendation via Reducing Node-neighbor Discrepancy in Graph Convolutional Network

2024-12-25 · Zheyu Chen, Jinfeng Xu, Haibo Hu

The rapid expansion of multimedia contents has led to the emergence of multimodal recommendation systems. It has attracted increasing attention in recommendation systems because its full utilization of data from differen…

Multimodal Recommendation Recommendation Systems

Graph4MM: Weaving Multimodal Learning with Structural Information

2025-10-19 · Xuying Ning, Dongqi Fu, Tianxin Wei, Wujiang Xu 외 arxiv

Real-world multimodal data usually exhibit complex structural relationships beyond traditional one-to-one mappings like image-caption pairs. Entities across modalities interact in intricate ways, with images and text for…

Progressive Pixel-Neighborhood Deformable Cross-Attention for Multispectral Object Detection

2026-06-23 · Tian Qiu, Jifeng Shen, Xin Zuo arxiv

Effective cross-modal feature alignment and interaction are central challenges in multispectral object detection. Although global cross-attention provides strong long-range modeling ability, its quadratic complexity with…

Multispectral Object Detection Semantic correspondence Long-range modeling