TT-BLIP: Enhancing Fake News Detection Using BLIP and Tri-Transformer
Detecting fake news has received a lot of attention. Many previous methods concatenate independently encoded unimodal data, ignoring the benefits of integrated multimodal information. Also, the absence of specialized feature extraction for text and images further limits these methods. This paper introduces an end-to-end model called TT-BLIP that applies the bootstrapping language-image pretraining for unified vision-language understanding and generation (BLIP) for three types of information: BERT and BLIPTxt for text, ResNet and BLIPImg for images, and bidirectional BLIP encoders for multimodal information. The Multimodal Tri-Transformer fuses tri-modal features using three types of multi-head attention mechanisms, ensuring integrated modalities for enhanced representations and improved multimodal data analysis. The experiments are performed using two fake news datasets, Weibo and Gossipcop. The results indicate TT-BLIP outperforms the state-of-the-art models.
Code (0)
등록된 구현이 없습니다.
Tasks
Fake News DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CroMe: Multimodal Fake News Detection using Cross-Modal Tri-Transformer and Metric Learning
Multimodal Fake News Detection has received increasing attention recently. Existing methods rely on independently encoded unimodal data and overlook the advantages of capturing intra-modality relationships and integratin…
Fake News DetectionMetric LearningAntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors
Deep generative models can create remarkably photorealistic fake images while raising concerns about misinformation and copyright infringement, known as deepfake threats. Deepfake detection technique is developed to dist…
DeepFake DetectionFace SwappingImage GenerationMisinformation+4Enhancing Journalism with AI: A Study of Contextualized Image Captioning for News Articles using LLMs and LMMs
Large language models (LLMs) and large multimodal models (LMMs) have significantly impacted the AI community, industry, and various economic sectors. In journalism, integrating AI poses unique challenges and opportunitie…
ArticlesImage CaptioningVLRM: Vision-Language Models act as Reward Models for Image Captioning
In this work, we present an unsupervised method for enhancing an image captioning model (in our case, BLIP2) using reinforcement learning and vision-language models like CLIP and BLIP2-ITM as reward models. The RL-tuned …
Image Captioningreinforcement-learningReinforcement LearningMemeBLIP2: A novel lightweight multimodal system to detect harmful memes
Memes often merge visuals with brief text to share humor or opinions, yet some memes contain harmful messages such as hate speech. In this paper, we introduces MemeBLIP2, a light weight multimodal system that detects har…
Language ModelingLanguage Modelling