LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition

2026-07-22 · Jiajun Cheng, Subarna Tripathi, Sainan Liu, Xiaofan Yu, Shan Lin arxiv

Understanding instrument-tissue interactions is essential for context-aware surgical AI and autonomous robotic surgery. Pretrained vision-language models (VLMs) and vision encoders offer an alternative to conventional interaction classifiers by transferring broad visual and semantic knowledge. However, adapting them to fine-grained surgical interactions remains challenging: (1) freezing the vision encoder depends entirely on pretrained representations that may retain noise and provide weak spatial localization, while (2) full fine-tuning can improve global semantic alignment without ensuring that the encoder learns meaningful features in the correct action region. We address these limitations by introducing LAViFiT, an end-to-end latent-action-guided framework for vision-language fine-tuning. An inverse dynamics model captures the visual changes induced by each action, while a forward world model drives the encoder to represent action-relevant regions. A patch-level SIG Regularizer further prevents local feature collapse without additional supervision, such as bounding boxes or pseudo-labels. Experiments across multiple encoders and datasets improve recognition and image-text alignment, while representation analyses show stronger grounding over the complete instrument-tissue interaction region and more spatially coherent features.

📄 PDF Abstract BibTeX arXiv:2607.19889

Code (0)

등록된 구현이 없습니다.

PearlVLA: Progressive Embodied Action-Plan Refinement in Latent Space

2026-06-16 · Bochen Yang, Lianlei Shan arxiv

Current Vision-Language-Action (VLA) models face a trade-off between efficient action generation and explicit deliberation. Directly decoding actions from vision-language backbone representations enables low-latency cont…

Visual Grounding

MPCoT: Reward-Guided Multi-Path Latent Reasoning for Test-Time Scalable Vision-Language-Action

2026-06-04 · Boyang Zhang, Lianlei Shan arxiv

Vision-Language-Action (VLA) policies remain brittle in long-horizon control, where one-pass action decoding offers limited inference-time deliberation. Explicit chain-of-thought adds reasoning depth but incurs token-gen…

Distraction-free Embeddings for Robust VQA

2023-08-31 · Atharvan Dogra, Deeksha Varshney, Ashwin Kalyan, Ameet Deshpande 외

The generation of effective latent representations and their subsequent refinement to incorporate precise information is an essential prerequisite for Vision-Language Understanding (VLU) tasks such as Video Question Answ…

Question Answering Video Question Answering Visual Question Answering (VQA)

SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space

2026-08-02 · Ruiteng Zhao, Zhengshen Zhang, Yue Su, Wenshuo Wang 외 hf

World Action Models (WAMs) couple action generation with prediction of future states. Their effectiveness depends on whether future dynamics are modeled in a space that is both aligned with action generation and sufficie…

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

2026-03-23 · Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan 외 arxiv

Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temp…

Trajectory Prediction General Knowledge