Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation
With the growth of social media and large language models, content moderation has become crucial. Many existing datasets lack adequate representation of different groups, resulting in unreliable assessments. To tackle this, we propose a socio-culturally aware evaluation framework for LLM-driven content moderation and introduce a scalable method for creating diverse datasets using persona-based generation. Our analysis reveals that these datasets provide broader perspectives and pose greater challenges for LLMs than diversity-focused generation methods without personas. This challenge is especially pronounced in smaller LLMs, emphasizing the difficulties they encounter in moderating such diverse content.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Building Socio-culturally Inclusive Stereotype Resources with Community Engagement
With rapid development and deployment of generative language models in global settings, there is an urgent need to also scale our measurements of harm, not just in the number and types of harms covered, but also how well…
CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generation
Existing red-teaming benchmarks, when adapted to new languages via direct translation, fail to capture socio-technical vulnerabilities rooted in local culture and law, creating a critical blind spot in LLM safety evaluat…
Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation
As online communication increasingly incorporates under-represented languages and colloquial dialects, standard translation systems often fail to preserve local slang, code-mixing, and culturally embedded markers of harm…
Semantic SimilarityPrompt EngineeringTowards Global, Socio-Economic, and Culturally Aware Recommender Systems
Recommender systems have gained increasing attention to personalise consumer preferences. While these systems have primarily focused on applications such as advertisement recommendations (e.g., Google), personalized sugg…
Recommendation SystemsSESGO: Spanish Evaluation of Stereotypical Generative Outputs
This paper addresses the critical gap in evaluating bias in multilingual Large Language Models (LLMs), with a specific focus on Spanish language within culturally-aware Latin American contexts. Despite widespread global …