Credit Assignment: Challenges and Opportunities in Developing Human-like AI Agents
Temporal credit assignment is crucial for learning and skill development in natural and artificial intelligence. While computational methods like the TD approach in reinforcement learning have been proposed, it's unclear if they accurately represent how humans handle feedback delays. Cognitive models intend to represent the mental steps by which humans solve problems and perform a number of tasks, but limited research in cognitive science has addressed the credit assignment problem in humans and cognitive models. Our research uses a cognitive model based on a theory of decisions from experience, Instance-Based Learning Theory (IBLT), to test different credit assignment mechanisms in a goal-seeking navigation task with varying levels of decision complexity. Instance-Based Learning (IBL) models simulate the process of making sequential choices with different credit assignment mechanisms, including a new IBL-TD model that combines the IBL decision mechanism with the TD approach. We found that (1) An IBL model that gives equal credit assignment to all decisions is able to match human performance better than other models, including IBL-TD and Q-learning; (2) IBL-TD and Q-learning models underperform compared to humans initially, but eventually, they outperform humans; (3) humans are influenced by decision complexity, while models are not. Our study provides insights into the challenges of capturing human behavior and the potential opportunities to use these models in future AI systems to support human activities.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning TheoryQ-LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hindsight Credit Assignment
We consider the problem of efficient credit assignment in reinforcement learning. In order to efficiently and meaningfully utilize new data, we propose to explicitly assign credit to past decisions based on the likelihoo…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs
Developing Large Language Models (LLMs) to cooperate and compete effectively within multi-agent systems (MASs) is a critical step towards more advanced intelligence. While reinforcement learning (RL) has proven effective…
Reinforcement LearningWhen Do Transformers Shine in RL? Decoupling Memory from Credit Assignment
Reinforcement learning (RL) algorithms face two distinct challenges: learning effective representations of past and present observations, and determining how actions influence future returns. Both challenges involve mode…
Reinforcement Learning (RL)Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning
Training cooperative agents in sparse-reward scenarios poses significant challenges for multi-agent reinforcement learning (MARL). Without clear feedback on actions at each step in sparse-reward setting, previous methods…
Efficient ExplorationMulti-agent Reinforcement LearningShapley Counterfactual Credits for Multi-Agent Reinforcement Learning
Centralized Training with Decentralized Execution (CTDE) has been a popular paradigm in cooperative Multi-Agent Reinforcement Learning (MARL) settings and is widely used in many real applications. One of the major challe…
counterfactualMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+3