Pessimistic Evaluation
Traditional evaluation of information access systems has focused primarily on average utility across a set of information needs (information retrieval) or users (recommender systems). In this work, we argue that evaluating only with average metric measurements assumes utilitarian values not aligned with traditions of information access based on equal access. We advocate for pessimistic evaluation of information access systems focusing on worst case utility. These methods are (a) grounded in ethical and pragmatic concepts, (b) theoretically complementary to existing robustness and fairness methods, and (c) empirically validated across a set of retrieval and recommendation tasks. These results suggest that pessimistic evaluation should be included in existing experimentation processes to better understand the behavior of systems, especially when concerned with principles of social good.
Code (0)
등록된 구현이 없습니다.
Tasks
FairnessInformation RetrievalRecommendation SystemsRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Pessimistic Iterative Planning for Robust POMDPs
Robust POMDPs extend classical POMDPs to handle model uncertainty. Specifically, robust POMDPs exhibit so-called uncertainty sets on the transition and observation models, effectively defining ranges of probabilities. Po…
MOReL: Model-Based Offline Reinforcement Learning
In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. This serves as an extreme test for an agent's ability to e…
modelOffline RLreinforcement-learningReinforcement Learning+1Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions
The privacy loss distribution (PLD) provides a tight characterization of the privacy loss of a mechanism in the context of differential privacy (DP). Recent work has shown that PLD-based accounting allows for tighter $(\…
Pessimistic Auxiliary Policy for Offline Reinforcement Learning
Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning proce…
Reinforcement LearningOffline RLOptimistic and Pessimistic Neural Networks for Scene and Object Recognition
In this paper the application of uncertainty modeling to convolutional neural networks is evaluated. A novel method for adjusting the network's predictions based on uncertainty information is introduced. This allows the …
AttributeObject Recognition