홈 › Papers

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

2024-01-21 · Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, Qi Zhang, Dahua Lin

The success of AI assistants based on Language Models (LLMs) hinges on Reinforcement Learning from Human Feedback (RLHF) to comprehend and align with user intentions. However, traditional alignment algorithms, such as PPO, are hampered by complex annotation and training requirements. This reliance limits the applicability of RLHF and hinders the development of professional assistants tailored to diverse human preferences. In this work, we introduce \textit{Linear Alignment}, a novel algorithm that aligns language models with human preferences in one single inference step, eliminating the reliance on data annotation and model training. Linear alignment incorporates a new parameterization for policy optimization under divergence constraints, which enables the extraction of optimal policy in a closed-form manner and facilitates the direct estimation of the aligned response. Extensive experiments on both general and personalized preference datasets demonstrate that linear alignment significantly enhances the performance and efficiency of LLM alignment across diverse scenarios. Our code and dataset is published on \url{https://github.com/Wizardcoast/Linear_Alignment.git}.

📄 PDF Abstract BibTeX arXiv:2401.11458

Code (1)

wizardcoast/linear_alignment 공식 구현 pytorch

Tasks

Form

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

SALT: Subspace Alignment as an Auxiliary Learning Task for Domain Adaptation

2019-06-11 · Kowshik Thopalli, Jayaraman J. Thiagarajan, Rushil Anirudh, Pavan Turaga

Unsupervised domain adaptation aims to transfer and adapt knowledge learned from a labeled source domain to an unlabeled target domain. Key components of unsupervised domain adaptation include: (a) maximizing performance…

Auxiliary Learning Domain Adaptation Form Unsupervised Domain Adaptation

Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO

2026-02-02 · Shokichi Takakura, Akifumi Wachi, Rei Higuchi, Kohei Miyaguchi 외 arxiv

Aligning large language models (LLMs) to diverse human preferences is fundamentally challenging since criteria can often conflict with each other. Inference-time alignment methods have recently gained popularity as they …

Agent Alignment in Evolving Social Norms

2024-01-09 · ShiMin Li, Tianxiang Sun, Qinyuan Cheng, Xipeng Qiu

Agents based on Large Language Models (LLMs) are increasingly permeating various domains of human production and life, highlighting the importance of aligning them with human values. The current alignment of AI systems p…

Diffeomorphic Temporal Alignment Nets

2019-12-10 · NeurIPS 2019 12 · Ron Shapira Weber, Matan Eyal, Nicki Skafte, Oren Shriki 외

Time-series analysis is confounded by nonlinear time warping of the data. Traditional methods for joint alignment do not generalize: after aligning a given signal ensemble, they lack a mechanism, that does not require so…

ECG Classification Electrocardiography (ECG)Time Series Time Series Alignment+2

Closed Form Word Embedding Alignment

2018-06-04 · Sunipa Dev, Safia Hassan, Jeff M. Phillips

We develop a family of techniques to align word embeddings which are derived from different source datasets or created using different mechanisms (e.g., GloVe or word2vec). Our methods are simple and have a closed form t…

Form Word Embeddings