Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model

2023-11-12 · Minh-Hao Van, Xintao Wu

Recently, large language models (LLMs) have taken the spotlight in natural language processing. Further, integrating LLMs with vision enables the users to explore more emergent abilities in multimodality. Visual language models (VLMs), such as LLaVA, Flamingo, or GPT-4, have demonstrated impressive performance on various visio-linguistic tasks. Consequently, there are enormous applications of large models that could be potentially used on social media platforms. Despite that, there is a lack of related work on detecting or correcting hateful memes with VLMs. In this work, we study the ability of VLMs on hateful meme detection and hateful meme correction tasks with zero-shot prompting. From our empirical experiments, we show the effectiveness of the pretrained LLaVA model and discuss its strengths and weaknesses in these tasks.

📄 PDF Abstract BibTeX arXiv:2311.06737

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Adam 설명 없음

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

2020-12-23 · Riza Velioglu, Jewgeni Rose

Memes on the Internet are often harmless and sometimes amusing. However, by using certain types of images, text, or combinations of both, the seemingly harmless meme becomes a multimodal type of hate speech -- a hateful …

Ensemble Learning Meme Classification Multimodal Deep Learning Multimodal Text and Image Classification

Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge

2020-12-02 · Xiayu Zhong

Hateful Memes is a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. Difficult examples are added to the dataset to make it hard to rely on unimodal signals, which me…

Classification General Classification

Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models

2025-04-30 · Minh-Hao Van, Xintao Wu

The rapid evolution of social media has provided enhanced communication channels for individuals to create online content, enabling them to express their thoughts and opinions. Multimodal memes, often utilized for playfu…

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

2020-05-10 · NeurIPS 2020 12 · Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami 외

This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: di…

Binary Classification Classification General Classification Meme Classification

HateSieve: A Contrastive Learning Framework for Detecting and Segmenting Hateful Content in Multimodal Memes

2024-08-11 · Xuanyu Su, Yansong Li, Diana Inkpen, Nathalie Japkowicz

Amidst the rise of Large Multimodal Models (LMMs) and their widespread application in generating and interpreting complex content, the risk of propagating biased and harmful memes remains significant. Current safety meas…

Contrastive Learning Triplet