Assessing AI Utility: The Random Guesser Test for Sequential Decision-Making Systems
We propose a general approach to quantitatively assessing the risk and vulnerability of artificial intelligence (AI) systems to biased decisions. The guiding principle of the proposed approach is that any AI algorithm must outperform a random guesser. This may appear trivial, but empirical results from a simplistic sequential decision-making scenario involving roulette games show that sophisticated AI-based approaches often underperform the random guesser by a significant margin. We highlight that modern recommender systems may exhibit a similar tendency to favor overly low-risk options. We argue that this "random guesser test" can serve as a useful tool for evaluating the utility of AI actions, and also points towards increasing exploration as a potential improvement to such systems.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingRecommendation SystemsSequential Decision MakingSimilar Papers 제목 키워드 기반
Learning Better Visual Dialog Agents with Pretrained Visual-Linguistic Representation
GuessWhat?! is a two-player visual dialog guessing game where player A asks a sequence of yes/no questions (Questioner) and makes a final guess (Guesser) about a target object in an image, based on answers from player B …
Referring ExpressionReferring Expression ComprehensionVisual DialogVisual GroundingAssessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents
Using persona-conditioned LLMs as synthetic survey respondents has become a common practice in computational social science and agent-based simulations. Yet, it remains unclear whether multi-attribute persona prompting i…
Guessing State Tracking for Visual Dialogue
The Guesser is a task of visual grounding in GuessWhat?! like visual dialogue. It locates the target object in an image supposed by an Oracle oneself over a question-answer based dialogue between a Questioner and the Ora…
Visual GroundingA Zero-Shot Classification Approach for a Word-Guessing Challenge
The Taboo Challenge competition, a task based on the well-known Taboo game, has been proposed to stimulate research in the AI field. The challenge requires building systems able to comprehend the implied inferences betwe…
ClassificationLanguage ModelingLanguage Modellingzero-shot-classification+1Category-Based Strategy-Driven Question Generator for Visual Dialogue
“GuessWhat?! is a task-oriented visual dialogue task which has two players a guesser and anoracle. Guesser aims to locate the object supposed by oracle by asking several Yes/No questions which are answered by oracle. How…
Sentence