Building a Bridge: A Method for Image-Text Sarcasm Detection Without Pretraining on Image-Text Data
Sarcasm detection in social media with text and image is becoming more challenging. Previous works of image-text sarcasm detection were mainly to fuse the summaries of text and image: different sub-models read the text and image respectively to get the summaries, and fuses the summaries. Recently, some multi-modal models based on the architecture of BERT are proposed such as ViLBERT. However, they can only be pretrained on the image-text data. In this paper, we propose an image-text model for sarcasm detection using the pretrained BERT and ResNet without any further pretraining. BERT and ResNet have been pretrained on much larger text or image data than image-text data. We connect the vector spaces of BERT and ResNet to utilize more data. We use the pretrained Multi-Head Attention of BERT to model the text and image. Besides, we propose a 2D-Intra-Attention to extract the relationships between words and images. In experiments, our model outperforms the state-of-the-art model.
Code (0)
등록된 구현이 없습니다.
Tasks
Sarcasm DetectionSimilar Papers 제목 키워드 기반
Multi-Modal Sarcasm Detection via Cross-Modal Graph Convolutional Network
With the increasing popularity of posting multimodal messages online, many recent studies have been carried out utilizing both textual and visual information for multi-modal sarcasm detection. In this paper, we investiga…
Sarcasm DetectionMMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System
Multi-modal sarcasm detection has attracted much recent attention. Nevertheless, the existing benchmark (MMSD) has some shortcomings that hinder the development of reliable multi-modal sarcasm detection system: (1) There…
Sarcasm DetectionCFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark
Multimodal sarcasm detection has recently garnered significant attention. However, existing benchmarks suffer from coarse-grained annotations and limited cultural coverage, which hinder research into fine-grained semanti…
Explanation GenerationReinforcement LearningSarcasm DetectionMMSD3.0: A Multi-Image Benchmark for Real-World Multimodal Sarcasm Detection
Despite progress in multimodal sarcasm detection, existing datasets and methods predominantly focus on single-image scenarios, overlooking potential semantic and affective relations across multiple images. This leaves a …
Sarcasm DetectionMulti-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model
Sarcasm is a subtle form of language in which people express the opposite of what is implied. Previous works of sarcasm detection focused on texts. However, more and more social media platforms like Twitter allow users t…
AttributeSarcasm Detection