홈 › Papers

RLTHF: Targeted Human Feedback for LLM Alignment

2025-02-19 · Yifei Xu, Tusher Chakraborty, Emre Kiciman, Bibek Aryal, Eduardo Rodrigues, Srinagesh Sharma, Roberto Estevao, Maria Angels de Luis Balaguer, Jessica Wolk, Rafael Padilha, Leonardo Nunes, Shobana Balakrishnan, Songwu Lu, Ranveer Chandra

Fine-tuning large language models (LLMs) to align with user preferences is challenging due to the high cost of quality human annotations in Reinforcement Learning from Human Feedback (RLHF) and the generalizability limitations of AI Feedback. To address these challenges, we propose RLTHF, a human-AI hybrid framework that combines LLM-based initial alignment with selective human annotations to achieve full-human annotation alignment with minimal effort. RLTHF identifies hard-to-annotate samples mislabeled by LLMs using a reward model's reward distribution and iteratively enhances alignment by integrating strategic human corrections while leveraging LLM's correctly labeled samples. Evaluations on HH-RLHF and TL;DR datasets show that RLTHF reaches full-human annotation-level alignment with only 6-7% of the human annotation effort. Furthermore, models trained on RLTHF's curated datasets for downstream tasks outperform those trained on fully human-annotated datasets, underscoring the effectiveness of RLTHF's strategic data curation.

📄 PDF Abstract BibTeX arXiv:2502.13417

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans

2025-12-29 · Sky CH-Wang, Justin Svegliato, Helen Appel, Jason Eisner arxiv

We present a method and dataset for fine-tuning language models with preference supervision using feedback-driven improvement chains. Given a model response, an annotator provides fine-grained feedback by marking ``liked…

Feedback Indicators: The Alignment between Llama and a Teacher in Language Learning

2025-08-15 · Sylvio Rüdian, Yassin Elsir, Marvin Kretschmer, Sabine Cayrou 외 arxiv

Automated feedback generation has the potential to enhance students' learning progress by providing timely and targeted feedback. Moreover, it can assist teachers in optimizing their time, allowing them to focus on more …

Data-efficient Targeted Token-level Preference Optimization for LLM-based Text-to-Speech

2025-10-07 · Rikuto Kotoge, Yuichi Sasaki arxiv

Aligning text-to-speech (TTS) system outputs with human feedback through preference optimization has been shown to effectively improve the robustness and naturalness of language model-based TTS models. Current approaches…

Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning

2025-02-03 · Udita Ghosh, Dripta S. Raychaudhuri, Jiachen Li, Konstantinos Karydis 외

Preference-based reinforcement learning (RL) offers a promising approach for aligning policies with human intent but is often constrained by the high cost of human feedback. In this work, we introduce PrefVLM, a framewor…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Transfer Learning

Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignment

2025-12-10 · Zixuan Liu, Siavash H. Khajavi, Guangkai Jiang, Xinru Liu arxiv

Reward-model-based fine-tuning is a central paradigm in aligning Large Language Models with human preferences. However, such approaches critically rely on the assumption that proxy reward models accurately reflect intend…