Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions
Large language models are increasingly used to simulate social media users and infer how individuals may respond to online discussions. However, it remains unclear whether these simulations reflect precise user-specific beliefs or whether they are highly sensitive to semantically independent changes in conversational contexts. In this work, we study counterfactual context revision as a framework for auditing LLM-based stance simulation. Given an original online conversation, we first infer a target user's stance toward a specific topic. We then apply controlled revision strategies to the conversational context and simulate the user's stance again under the revised context. We compare text-only revision strategies with a multimodal one that incorporates meme-based context and evaluate two main effectiveness metrics, i.e., average directional stance shift and stance transition rate. The results reveal effective and robust stance transitions in both text-only and multimodal strategies across different polarization-preference mechanisms. Our study contributes an evaluation framework for understanding the context sensitivity of LLM-based stance simulation (https://github.com/STARResearchLab/StanceShift). More broadly, it highlights both the promise and risk of using LLMs as proxies for social media users in social simulation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics
Chinese web pollution has surfaced in LLMs, motivating audits of upstream Chinese corpora. However, auditing such corpora faces three challenges: (1) their web-scale size makes full scan costly; (2) prior analyses are of…
DECOR: Auditing LLM Deception via Information Manipulation Theory
Large language models can deceive by subtly manipulating truthful information -- omitting key facts, shifting focus, or obscuring meaning -- making such behavior difficult to detect. Existing black-box methods rely on co…
Closing the Loop: Learning to Generate Writing Feedback via Language Model Simulated Student Revisions
Providing feedback is widely recognized as crucial for refining students' writing skills. Recent advances in language models (LMs) have made it possible to automatically generate feedback that is actionable and well-alig…
Language ModelingLanguage ModellingA Generative Approach for Semantic Auditing of Electronic Health Records
The reliability of clinical artificial intelligence (AI) depends on high-quality data, yet Electronic Health Records are often inconsistent with existing scientific knowledge. Current quality assessments are limited: the…
Gram: Assessing sabotage propensities via automated alignment auditing
We introduce Gram, an automated alignment auditing framework to assess the propensity of AI agents to engage in sabotage. We evaluate Gemini models across 17 simulated agentic deployment scenarios that incentivize sabota…