It's All Relative: Interpretable Models for Scoring Bias in Documents
We propose an interpretable model to score the bias present in web documents, based only on their textual content. Our model incorporates assumptions reminiscent of the Bradley-Terry axioms and is trained on pairs of revisions of the same Wikipedia article, where one version is more biased than the other. While prior approaches based on absolute bias classification have struggled to obtain a high accuracy for the task, we are able to develop a useful model for scoring bias by learning to perform pairwise comparisons of bias accurately. We show that we can interpret the parameters of the trained model to discover the words most indicative of bias. We also apply our model in three different settings - studying the temporal evolution of bias in Wikipedia articles, comparing news sources based on bias, and scoring bias in law amendments. In each case, we demonstrate that the outputs of the model can be explained and validated, even for the two domains that are outside the training-data domain. We also use the model to compare the general level of bias between domains, where we see that legal texts are the least biased and news media are the most biased, with Wikipedia articles in between. Given its high performance, simplicity, interpretability, and wide applicability, we hope the model will be useful for a large community, including Wikipedia and news editors, political and social scientists, and the general public.
Code (0)
등록된 구현이 없습니다.
Tasks
AllArticlesSimilar Papers 제목 키워드 기반
Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correction
Recent advances in unbiased learning to rank (LTR) count on Inverse Propensity Scoring (IPS) to eliminate bias in implicit feedback. Though theoretically sound in correcting the bias introduced by treating clicked docume…
Learning-To-RankWord-level Human Interpretable Scoring Mechanism for Novel Text Detection Using Tsetlin Machines
Recent research in novelty detection focuses mainly on document-level classification, employing deep neural networks (DNN). However, the black-box nature of DNNs makes it difficult to extract an exact explanation of why …
Novelty DetectionText DetectionReinforcement Online Learning to Rank with Unbiased Reward Shaping
Online learning to rank (OLTR) aims to learn a ranker directly from implicit feedback derived from users' interactions, such as clicks. Clicks however are a biased signal: specifically, top-ranked documents are likely to…
Learning-To-RankPositionLearning Groupwise Multivariate Scoring Functions Using Deep Neural Networks
While in a classification or a regression setting a label or a value is assigned to each individual document, in a ranking setting we determine the relevance ordering of the entire input document list. This difference le…
Learning-To-RankAnalysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank
Leveraging biased click data for optimizing learning to rank systems has been a popular approach in information retrieval. Because click data is often noisy and biased, a variety of methods have been proposed to construc…
Information RetrievalLearning-To-RankRetrieval