Continual SFT Matches Multimodal RLHF with Negative Supervision

2024-11-22 · CVPR 2025 1 · Ke Zhu, Yu Wang, Yanpeng Sun, Qiang Chen, JiangJiang Liu, Gang Zhang, Jingdong Wang

Multimodal RLHF usually happens after supervised finetuning (SFT) stage to continually improve vision-language models' (VLMs) comprehension. Conventional wisdom holds its superiority over continual SFT during this preference alignment stage. In this paper, we observe that the inherent value of multimodal RLHF lies in its negative supervision, the logit of the rejected responses. We thus propose a novel negative supervised finetuning (nSFT) approach that fully excavates these information resided. Our nSFT disentangles this negative supervision in RLHF paradigm, and continually aligns VLMs with a simple SFT loss. This is more memory efficient than multimodal RLHF where 2 (e.g., DPO) or 4 (e.g., PPO) large VLMs are strictly required. The effectiveness of nSFT is rigorously proved by comparing it with various multimodal RLHF approaches, across different dataset sources, base VLMs and evaluation metrics. Besides, fruitful of ablations are provided to support our hypothesis. We hope this paper will stimulate further research to properly align large vision language models.

📄 PDF Abstract BibTeX arXiv:2411.14797

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

SFT Shrink and Fine-Tune, or SFT, is a type of distillation that avoids explicit distillation by copying parameters to a student student model and then fine-tuning.…

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

BASE 설명 없음

Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

2025-01-14 · Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt

Language model (LM) post-training relies on two stages of human supervision: task demonstrations for supervised finetuning (SFT), followed by preference comparisons for reinforcement learning from human feedback (RLHF). …

Instruction Following Math

MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge

2026-03-16 · Baochen Fu, Yuntao Du, Cheng Chang, Baihao Jin 외 arxiv

As real-world knowledge continues to evolve, the parametric knowledge acquired by multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on m…

Reinforcement Learning knowledge editing

Reason to Contrast: A Cascaded Multimodal Retrieval Framework

2025-12-21 · Xuanming Cui, Hong-You Chen, Hao Yu, Hao Yuan 외 arxiv

Traditional multimodal retrieval systems rely primarily on bi-encoder architectures, where performance is closely tied to embedding dimensionality. Recent work, Think-Then-Embed (TTE), shows that incorporating multimodal…

Multimodal Reasoning

Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization

2026-05-29 · Haolin Deng, Xin Zou, Zhiwei Jin, Chen Chen 외 arxiv

Multimodal hallucination remains a persistent challenge for Vision-Language Models (VLMs). Standard textual Direct Preference Optimization (DPO) often fails to mitigate it due to a lack of explicit visual supervision. Wh…

Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE

2026-03-31 · Hejin Huang, Jusheng Zhang, Kaitong Cai, Jian Wang 외 arxiv

Preference-based alignment objectives have been widely adopted, from RLHF-style pairwise learning in large language models to emerging applications in recommender systems. Yet, existing work rarely examines how Direct Pr…

Sequential Recommendation