The Difficulty of Novelty Detection in Open-World Physical Domains: An Application to Angry Birds
Detecting and responding to novel situations in open-world environments is a key capability of human cognition and is a persistent problem for AI systems. In an open-world, novelties can appear in many different forms and may be easy or hard to detect. Therefore, to accurately evaluate the novelty detection capability of AI systems, it is necessary to investigate how difficult it may be to detect different types of novelty. In this paper, we propose a qualitative physics-based method to quantify the difficulty of novelty detection focusing on open-world physical domains. We apply our method in the popular physics simulation game Angry Birds, and conduct a user study across different novelties to validate our method. Results indicate that our calculated detection difficulties are in line with those of human users.
Code (0)
등록된 구현이 없습니다.
Tasks
Novelty DetectionSimilar Papers 제목 키워드 기반
Measuring Difficulty of Novelty Reaction
Current AI systems are designed to solve close-world problems with the assumption that the underlying world is remaining more or less the same. However, when dealing with real-world problems such assumptions can be inval…
NovPhy: A Testbed for Physical Reasoning in Open-world Environments
Due to the emergence of AI systems that interact with the physical environment, there is an increased interest in incorporating physical reasoning capabilities into those AI systems. But is it enough to only have physica…
NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds
In order for artificial agents to successfully perform tasks in changing environments, they must be able to both detect and adapt to novelty. However, visual novelty detection research often only evaluates on repurposed …
MinecraftNovelty DetectionOpen-World Continual Learning: Unifying Novelty Detection and Continual Learning
As AI agents are increasingly used in the real open world with unknowns or novelties, they need the ability to (1) recognize objects that (a) they have learned before and (b) detect items that they have never seen or lea…
class-incremental learningClass Incremental LearningContinual LearningIncremental Learning+2Why does in-context learning fail sometimes? Evaluating in-context learning on open and closed questions
We measure the performance of in-context learning as a function of task novelty and difficulty for open and closed questions. For that purpose, we created a novel benchmark consisting of hard scientific questions, each p…
In-Context LearningRAGRetrieval-augmented Generation