Detecting Offensive Memes with Social Biases in Singapore Context Using Multimodal Large Language Models
Traditional online content moderation systems struggle to classify modern multimodal means of communication, such as memes, a highly nuanced and information-dense medium. This task is especially hard in a culturally diverse society like Singapore, where low-resource languages are used and extensive knowledge on local context is needed to interpret online content. We curate a large collection of 112K memes labeled by GPT-4V for fine-tuning a VLM to classify offensive memes in Singapore context. We show the effectiveness of fine-tuned VLMs on our dataset, and propose a pipeline containing OCR, translation and a 7-billion parameter-class VLM. Our solutions reach 80.62% accuracy and 0.8192 AUROC on a held-out test set, and can greatly aid human in moderating online contents. The dataset, code, and model weights will be open-sourced at https://github.com/aliencaocao/vlm-for-memes-aisg.
Code (1)
Tasks
Optical Character Recognition (OCR)Similar Papers 제목 키워드 기반
AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Media
This paper focuses on an important problem of detecting offensive analogy meme on online social media where the visual content and the texts/captions of the meme together make an analogy to convey the offensive informati…
OSPC: Detecting Harmful Memes with Large Language Model as a Catalyst
Memes, which rapidly disseminate personal opinions and positions across the internet, also pose significant challenges in propagating social bias and prejudice. This study presents a novel approach to detecting harmful m…
Image CaptioningLanguage ModelingLanguage ModellingLarge Language Model+2TrollMeta@DravidianLangTech-EACL2021: Meme classification using deep learning
Memes act as a medium to carry one’s feelings, cultural ideas, or practices by means of symbols, imitations, or simply images. Whenever social media is involved, hurting the feelings of others and abusing others are alwa…
ClassificationDeep LearningMeme ClassificationDetecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models
The rise of social media and online communication platforms has led to the spread of Arabic textual posts and memes as a key form of digital expression. While these contents can be humorous and informative, they are also…
SemEval-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor!
Information on social media comprises of various modalities such as textual, visual and audio. NLP and Computer Vision communities often leverage only one prominent modality in isolation to study social media. However, c…
Emotion Recognition