Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations

2025-09-14 · Shresth Grover, Akshay Gopalkrishnan, Bo Ai, Henrik I. Christensen, Hao Su, Xuanlin Li arxiv

Vision-language-action (VLA) models finetuned from vision-language models (VLMs) hold the promise of leveraging rich pretrained representations to build generalist robots across diverse tasks and environments. However, direct fine-tuning on robot data often disrupts these representations and limits generalization. We present a framework that better preserves pretrained features while adapting them for robot manipulation. Our approach introduces three components: (i) a dual-encoder design with one frozen vision encoder to retain pretrained features and another trainable for task adaptation, (ii) a string-based action tokenizer that casts continuous actions into character sequences aligned with the model's pretraining domain, and (iii) a co-training strategy that combines robot demonstrations with vision-language datasets emphasizing spatial reasoning and affordances. Evaluations in simulation and on real robots show that our method improves robustness to visual perturbations, generalization to novel instructions and environments, and overall task success compared to baselines.

📄 PDF Abstract BibTeX arXiv:2509.11417

Code (0)

등록된 구현이 없습니다.

Tasks

Robot Manipulation Spatial Reasoning

Hierarchical Cross-modal Prompt Learning for Vision-Language Models

2025-07-20 · Hao Zheng, Shunzhi Yang, Zhuoxin He, Jinfeng Yang 외

Pre-trained Vision-Language Models (VLMs) such as CLIP have shown excellent generalization abilities. However, adapting these large-scale models to downstream tasks while preserving their generalization capabilities rema…

Prompt Learning

Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models

2025-08-07 · Zane Xu, Jason Sun arxiv

This report synthesizes eight seminal papers on the zero-shot adversarial robustness of vision-language models (VLMs) like CLIP. A central challenge in this domain is the inherent trade-off between enhancing adversarial …

Zero-shot Generalization Adversarial Robustness

MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalization

2025-11-25 · Chengyue Huang, Mellon M. Zhang, Robert Azarcon, Glen Chou 외 arxiv

Vision-Language-Action (VLA) models inherit strong priors from pretrained Vision-Language Models (VLMs), but naive fine-tuning often disrupts these representations and harms generalization. Existing fixes -- freezing mod…

Semantic-Preserving Cross-Style Visual Reasoning for Robust Multi-Modal Understanding in Large Vision-Language Models

2025-10-26 · Aya Nakayama, Brian Wong, Yuji Nishimura, Kaito Tanaka arxiv

The "style trap" poses a significant challenge for Large Vision-Language Models (LVLMs), hindering robust semantic understanding across diverse visual styles, especially in in-context learning (ICL). Existing methods oft…

Visual Question Answering Contrastive Learning Visual Reasoning

BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation

2026-08-05 · Peiyan Li, Yuze Zhu, Yixiang Chen, Qisen Ma 외 hf

Leveraging pre-trained vision-language models (VLMs) to construct vision-language-action (VLA) models has emerged as a promising paradigm for 3D robot manipulation. However, existing 3D VLA methods remain data-hungry, ex…

Robot Manipulation Point Clouds