Multi-modal Representation Learning for Social Post Location Inference
Inferring geographic locations via social posts is essential for many practical location-based applications such as product marketing, point-of-interest recommendation, and infector tracking for COVID-19. Unlike image-based location retrieval or social-post text embedding-based location inference, the combined effect of multi-modal information (i.e., post images, text, and hashtags) for social post positioning receives less attention. In this work, we collect real datasets of social posts with images, texts, and hashtags from Instagram and propose a novel Multi-modal Representation Learning Framework (MRLF) capable of fusing different modalities of social posts for location inference. MRLF integrates a multi-head attention mechanism to enhance location-salient information extraction while significantly improving location inference compared with single domain-based methods. To overcome the noisy user-generated textual content, we introduce a novel attention-based character-aware module that considers the relative dependencies between characters of social post texts and hashtags for flexible multi-model information fusion. The experimental results show that MRLF can make accurate location predictions and open a new door to understanding the multi-modal data of social posts for online inference tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
MarketingRepresentation LearningRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Transformer-based Framework for POI-level Social Post Geolocation
POI-level geo-information of social posts is critical to many location-based applications and services. However, the multi-modality, complexity and diverse nature of social media data and their platforms limit the perfor…
Similarity Guided Multimodal Fusion Transformer for Semantic Location Prediction in Social Media
Semantic location prediction aims to derive meaningful location insights from multimodal social media posts, offering a more contextual understanding of daily activities than using GPS coordinates. This task faces signif…
Language ModellingLeveraging Contrastive Learning for Few-shot Geolocation of Social Posts
Social geolocation is an important problem of predicting the originating locations of social media posts. However, this task is challenging due to the need for a substantial volume of training data, alongside well-annota…
Contrastive LearningXBully: Cyberbullying Detection within a Multi-Modal Context
Over the last decade, research has revealed the high prevalence of cyberbullying among youth and raised serious concerns in society. Information on the social media platforms where cyberbullying is most prevalent (e.g…
AttributeImproving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks
Effectively leveraging multimodal information from social media posts is essential to various downstream tasks such as sentiment analysis, sarcasm detection or hate speech classification. Jointly modeling text and images…
Image-text matchingSarcasm DetectionSemantic correspondenceSentiment Analysis+1