Multi-Hop Knowledge Graph Reasoning with Reward Shaping
Multi-hop reasoning is an effective approach for query answering (QA) over incomplete knowledge graphs (KGs). The problem can be formulated in a reinforcement learning (RL) setup, where a policy-based agent sequentially extends its inference path until it reaches a target. However, in an incomplete KG environment, the agent receives low-quality rewards corrupted by false negatives in the training data, which harms generalization at test time. Furthermore, since no golden action sequence is used for training, the agent can be misled by spurious search trajectories that incidentally lead to the correct answer. We propose two modeling advances to address both issues: (1) we reduce the impact of false negative supervision by adopting a pretrained one-hop embedding model to estimate the reward of unobserved facts; (2) we counter the sensitivity to spurious paths of on-policy RL by forcing the agent to explore a diverse set of paths using randomly generated edge masks. Our approach significantly improves over existing path-based KGQA models on several benchmark datasets and is comparable or better than embedding-based models.
Code (3)
Tasks
Knowledge Graph EmbeddingsKnowledge GraphsReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Story Shaping: Teaching Agents Human-like Behavior with Stories
Reward design for reinforcement learning agents can be difficult in situations where one not only wants the agent to achieve some effect in the world but where one also cares about how that effect is achieved. For exampl…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)text-based gamesEnhancing Multi-Hop Knowledge Graph Reasoning through Reward Shaping Techniques
In the realm of computational knowledge representation, Knowledge Graph Reasoning (KG-R) stands at the forefront of facilitating sophisticated inferential capabilities across multifarious domains. The quintessence of thi…
Knowledge GraphsNavigatePrompt LearningReinforcement Learning (RL)ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning
Sparse rewards are a major bottleneck in multi-agent reinforcement learning (MARL), where simultaneous learning induces non-stationarity and makes reward design especially delicate. Reward shaping can accelerate learning…
Multi-agent Reinforcement LearningLearning to Utilize Shaping Rewards: A New Approach of Reward Shaping
Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL). Existing approaches such as potential-based reward shaping normally make full use of a given shaping reward fu…
MuJoCoReinforcement Learning (RL)TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs
Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remains a significant challenge. The optimiza…
Open-Domain Question AnsweringReinforcement Learning