How Hard is it to Confuse a World Model?
In reinforcement learning (RL) theory, the concept of most confusing instances is central to establishing regret lower bounds, that is, the minimal exploration needed to solve a problem. Given a reference model and its optimal policy, a most confusing instance is the statistically closest alternative model that makes a suboptimal policy optimal. While this concept is well-studied in multi-armed bandits and ergodic tabular Markov decision processes, constructing such instances remains an open question in the general case. In this paper, we formalize this problem for neural network world models as a constrained optimization: finding a modified model that is statistically close to the reference one, while producing divergent performance between optimal and suboptimal policies. We propose an adversarial training procedure to solve this problem and conduct an empirical study across world models of varying quality. Our results suggest that the degree of achievable confusion correlates with uncertainty in the approximate model, which may inform theoretically-grounded exploration strategies for deep model-based RL.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningMulti-Armed BanditsSimilar Papers 제목 키워드 기반
Towards Generalizable Medical Image Segmentation with Pixel-wise Uncertainty Estimation
Deep neural networks (DNNs) achieve promising performance in visual recognition under the independent and identically distributed (IID) hypothesis. In contrast, the IID hypothesis is not universally guaranteed in numerou…
Image SegmentationMedical Image AnalysisMedical Image SegmentationSemantic SegmentationEnergy Confused Adversarial Metric Learning for Zero-Shot Image Retrieval and Clustering
Deep metric learning has been widely applied in many computer vision tasks, and recently, it is more attractive in \emph{zero-shot image retrieval and clustering}(ZSRC) where a good embedding is requested such that the u…
ClusteringImage RetrievalMetric LearningRetrieval+1U-Define: Designing User Workflows for Hard and Soft Constraints in LLM-Based Planning
LLMs are increasingly used for end-user task planning, yet their black-box nature limits users' ability to ensure reliability and control. While recent systems incorporate verification techniques, it remains unclear how …
Physical Adversarial Textures that Fool Visual Object Tracking
We present a system for generating inconspicuous-looking textures that, when displayed in the physical world as digital or printed posters, cause visual object tracking systems to become confused. For instance, as a targ…
ObjectObject TrackingVisual Object TrackingBatch Group Normalization
Deep Convolutional Neural Networks (DCNNs) are hard and time-consuming to train. Normalization is one of the effective solutions. Among previous normalization methods, Batch Normalization (BN) performs well at medium and…
Domain AdaptationFew-Shot LearningGPUimage-classification+3