On the Reliability and Validity of Detecting Approval of Political Actors in Tweets
Social media sites like Twitter possess the potential to complement surveys that measure political opinions and, more specifically, political actors{'} approval. However, new challenges related to the reliability and validity of social-media-based estimates arise. Various sentiment analysis and stance detection methods have been developed and used in previous research to measure users{'} political opinions based on their content on social media. In this work, we attempt to gauge the efficacy of untargeted sentiment, targeted sentiment, and stance detection methods in labeling various political actors{'} approval by benchmarking them across several datasets. We also contrast the performance of these pretrained methods that can be used in an off-the-shelf (OTS) manner against a set of models trained on minimal custom data. We find that OTS methods have low generalizability on unseen and familiar targets, while low-resource custom models are more robust. Our work sheds light on the strengths and limitations of existing methods proposed for understanding politicians{'} approval from tweets.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingSentiment AnalysisStance DetectionSimilar Papers 제목 키워드 기반
Prediction of Political Leanings of Chinese Speaking Twitter Users
This work presents a supervised method for generating a classifier model of the stances held by Chinese-speaking politicians and other Twitter users. Many previous works of political tweets prediction exist on English tw…
PredictionSegmentationA Hierarchical Error Framework for Reliable Automated Coding in Communication Research: Applications to Health and Political Communication
Automated content analysis increasingly supports communication research, yet scaling manual coding into computational pipelines raises concerns about measurement reliability and validity. We introduce a Hierarchical Erro…
Bias DetectionFact or Fiction? Can LLMs be Reliable Annotators for Political Truths?
Political misinformation poses significant challenges to democratic processes, shaping public opinion and trust in media. Manual fact-checking methods face issues of scalability and annotator bias, while machine learning…
ArticlesFact CheckingMisinformationDetecting Multidimensional Political Incivility on Social Media
The rise of social media has been argued to intensify uncivil and hostile online political discourse. Yet, to date, there is a lack of clarity on what incivility means in the political sphere. In this work, we utilize a …
Stance DetectionHigh Risk of Political Bias in Black Box Emotion Inference Models
This paper investigates the presence of political bias in emotion inference models used for sentiment analysis (SA) in social science research. Machine learning models often reflect biases in their training data, impacti…
Sentiment Analysis