Augmenting Unsupervised Reinforcement Learning with Self-Reference
Humans possess the ability to draw on past experiences explicitly when learning new tasks and applying them accordingly. We believe this capacity for self-referencing is especially advantageous for reinforcement learning agents in the unsupervised pretrain-then-finetune setting. During pretraining, an agent's past experiences can be explicitly utilized to mitigate the nonstationarity of intrinsic rewards. In the finetuning phase, referencing historical trajectories prevents the unlearning of valuable exploratory behaviors. Motivated by these benefits, we propose the Self-Reference (SR) approach, an add-on module explicitly designed to leverage historical information and enhance agent performance within the pretrain-finetune paradigm. Our approach achieves state-of-the-art results in terms of Interquartile Mean (IQM) performance and Optimality Gap reduction on the Unsupervised Reinforcement Learning Benchmark for model-free methods, recording an 86% IQM and a 16% Optimality Gap. Additionally, it improves current algorithms by up to 17% IQM and reduces the Optimality Gap by 31%. Beyond performance enhancement, the Self-Reference add-on also increases sample efficiency, a crucial attribute for real-world applications.
Code (0)
등록된 구현이 없습니다.
Tasks
Attributereinforcement-learningReinforcement LearningUnsupervised Reinforcement LearningSimilar Papers 제목 키워드 기반
West-of-N: Synthetic Preferences for Self-Improving Reward Models
The success of reinforcement learning from human feedback (RLHF) in language model alignment is strongly dependent on the quality of the underlying reward model. In this paper, we present a novel approach to improve rewa…
Language ModelingLanguage ModellingSSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation
Large language models (LLMs) have recently demonstrated remarkable capabilities in machine translation (MT). However, most advanced MT-specific LLMs heavily rely on external supervision signals during training, such as h…
Machine TranslationReinforcement Learning (RL)TranslationUnsupervised Domain Adaptation by Optical Flow Augmentation in Semantic Segmentation
It is expensive to generate real-life image labels and there is a domain gap between real-life and simulated images, hence a model trained on the latter cannot adapt to the former. Solving this can totally eliminate the …
Domain AdaptationOptical Flow EstimationSegmentationSemantic Segmentation+1PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success …
Contrastive LearningOpen-Ended Question AnsweringSentenceSentence Embeddings+1Unsupervised Bitext Mining and Translation via Self-trained Contextual Embeddings
We describe an unsupervised method to create pseudo-parallel corpora for machine translation (MT) from unaligned text. We use multilingual BERT to create source and target sentence embeddings for nearest-neighbor search …
Machine TranslationSentenceSentence EmbeddingsTranslation