What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
Machine learning is an important tool for decision making, but its ethical and responsible application requires rigorous vetting of its interpretability and utility: an understudied problem, particularly for natural language processing models. We propose an evaluation of interpretation on a real task with real human users, where the effectiveness of interpretation is measured by how much it improves human performance. We design a grounded, realistic human-computer cooperative setting using a question answering task, Quizbowl. We recruit both trivia experts and novices to play this game with computer as their teammate, who communicates its prediction via three different interpretations. We also provide design guidance for natural language processing human-in-the-loop settings.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningDecision MakingQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A mathematical theory of cooperative communication
Cooperative communication plays a central role in theories of human cognition, language, development, culture, and human-robot interaction. Prior models of cooperative communication are algorithmic in nature and do not s…
Cultural Vocal Bursts Intensity PredictionQuantum Frog: Emergent Cooperation and Difficulty Scaling in a Quantized-Time Cooperative Game
We introduce \emph{Quantum Frog}, a two-player cooperative game built on a novel \emph{quantized-time} mechanic in which the environment advances only when a player acts. Inspired by the classic arcade game Frogger, Quan…
Reinforcement LearningComputational Hermeneutics: Evaluating generative AI as a cultural technology
Generative AI systems are increasingly recognized as cultural technologies, yet current evaluation frameworks often treat culture as a variable to be measured rather than fundamental to the system's operation. Drawing on…
Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination
Cooperative artificial intelligence with human or superhuman proficiency in collaborative tasks stands at the frontier of machine learning research. Prior work has tended to evaluate cooperative AI performance under the …
DecoderDiversityGeneralizing the theory of cooperative inference
Cooperation information sharing is important to theories of human learning and has potential implications for machine learning. Prior work derived conditions for achieving optimal Cooperative Inference given strong, rela…
BIG-bench Machine Learning