Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communities
Large language models (LLMs) have shown promise in representing individuals and communities, offering new ways to study complex social dynamics. However, effectively aligning LLMs with specific human groups and systematically assessing the fidelity of the alignment remains a challenge. This paper presents a robust framework for aligning LLMs with online communities via instruction-tuning and comprehensively evaluating alignment across various aspects of language, including authenticity, emotional tone, toxicity, and harm. We demonstrate the utility of our approach by applying it to online communities centered on dieting and body image. We administer an eating disorder psychometric test to the aligned LLMs to reveal unhealthy beliefs and successfully differentiate communities with varying levels of eating disorder risk. Our results highlight the potential of LLMs in automated moderation and broader applications in public health and social science research.
Code (0)
등록된 구현이 없습니다.
Tasks
Models AlignmentSimilar Papers 제목 키워드 기반
Bottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs
Large language models (LLMs) generate diverse, situated, persuasive texts from a plurality of potential perspectives, influenced heavily by their prompts and training data. As part of LLM adoption, we seek to characteriz…
Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making
Large language models (LLMs) are increasingly used in social science simulations. While their performance on reasoning and optimization tasks has been extensively evaluated, less attention has been paid to their ability …
A Metric for MLLM Alignment in Large-scale Recommendation
Multimodal recommendation has emerged as a critical technique in modern recommender systems, leveraging content representations from advanced multimodal large language models (MLLMs). To ensure these representations are …
Multimodal RecommendationA Benchmarking Methodology to Assess Open-Source Video Large Language Models in Automatic Captioning of News Videos
News videos are among the most prevalent content types produced by television stations and online streaming platforms, yet generating textual descriptions to facilitate indexing and retrieval largely remains a manual pro…
Video CaptioningEarly Estimation of Language to Latent Alignment in Diffusion Models
Conditional diffusion models frequently suffer from language-image misalignments. Due to the ambiguity of intermediate noise corrupted latents, assessing prompt adherence currently requires completing the entire sampling…
Data Augmentation