RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment
We propose Reinforcement Learning from Contrastive Distillation (RLCD), a method for aligning language models to follow principles expressed in natural language (e.g., to be more harmless) without using human feedback. RLCD creates preference pairs from two contrasting model outputs, one using a positive prompt designed to encourage following the given principles, and one using a negative prompt designed to encourage violating them. Using two different prompts causes model outputs to be more differentiated on average, resulting in cleaner preference labels in the absence of human annotations. We then use the preference pairs to train a preference model, which is in turn used to improve a base unaligned language model via reinforcement learning. Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context distillation (Huang et al., 2022) baselines across three diverse alignment tasks--harmlessness, helpfulness, and story outline generation--and when using both 7B and 30B model scales for simulating preference data.
Code (2)
Tasks
Language ModelingLanguage Modellingreinforcement-learningReinforcement LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures
Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judges that spend a decoding pass on every criterion, and classifiers that read token probabilities, suc…
Reinforcement LearningEnhancing Medical Large Vision-Language Models via Alignment Distillation
Medical Large Vision-Language Models (Med-LVLMs) have shown promising results in clinical applications, but often suffer from hallucinated outputs due to misaligned visual understanding. In this work, we identify two fun…
Visual Question AnsweringMedical Report GenerationRepresentation LearningRadarLCD: Learnable Radar-based Loop Closure Detection Pipeline
Loop Closure Detection (LCD) is an essential task in robotics and computer vision, serving as a fundamental component for various applications across diverse domains. These applications encompass object recognition, imag…
Image RetrievalLoop Closure DetectionObject RecognitionRadar odometryContrastive Knowledge Transfer and Robust Optimization for Secure Alignment of Large Language Models
This paper addresses the limitations of large-scale language models in safety alignment and robustness by proposing a fine-tuning method that combines contrastive distillation with noise-robust training. The method freez…
parameter-efficient fine-tuningCL-XABSA: Contrastive Learning for Cross-lingual Aspect-based Sentiment Analysis
As an extensive research in the field of natural language processing (NLP), aspect-based sentiment analysis (ABSA) is the task of predicting the sentiment expressed in a text relative to the corresponding aspect. Unfortu…
Aspect-Based Sentiment AnalysisAspect-Based Sentiment Analysis (ABSA)Contrastive LearningKnowledge Distillation+1