Value Alignment from Unstructured Text
Aligning large language models (LLMs) to value systems has emerged as a significant area of research within the fields of AI and NLP. Currently, this alignment process relies on the availability of high-quality supervised and preference data, which can be both time-consuming and expensive to curate or annotate. In this paper, we introduce a systematic end-to-end methodology for aligning LLMs to the implicit and explicit values represented in unstructured text data. Our proposed approach leverages the use of scalable synthetic data generation techniques to effectively align the model to the values present in the unstructured data. Through two distinct use-cases, we demonstrate the efficiency of our methodology on the Mistral-7B-Instruct model. Our approach credibly aligns LLMs to the values embedded within documents, and shows improved performance against other approaches, as quantified through the use of automatic metrics and win rates.
Code (0)
등록된 구현이 없습니다.
Tasks
Synthetic Data GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value
Beneficial societal outcomes cannot be guaranteed by aligning individual AI systems with the intentions of their operators or users. Even an AI system that is perfectly aligned to the intentions of its operating organiza…
Towards Sample Efficient Agents through Algorithmic Alignment
In this work, we propose and explore Deep Graph Value Network (DeepGV) as a promising method to work around sample complexity in deep reinforcement-learning agents using a message-passing mechanism. The main idea is that…
Deep Reinforcement LearningGraph Neural NetworkReinforcement Learning (RL)AIGP: An LLM-Based Framework for Long-Term Value Alignment in E-Commerce Pricing
Traditional dynamic pricing models in large-scale e-commerce suffer from limited interpretability, poor utilization of unstructured information, and misalignment with long-term business objectives such as cumulative Gros…
Knowledge DistillationReinforcement LearningFusing Narrative Semantics for Financial Volatility Forecasting
We introduce M2VN: Multi-Modal Volatility Network, a novel deep learning-based framework for financial volatility forecasting that unifies time series features with unstructured news data. M2VN leverages the representati…
Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models
Recent knowledge editing methods have primarily focused on modifying structured knowledge in large language models. However, this task setting overlooks the fact that a significant portion of real-world knowledge is stor…
knowledge editingWorld Knowledge