An analysis of HOI: using a training-free method with multimodal visual foundation models when only the test set is available, without the training set

2024-08-11 · Chaoyi Ai

Human-Object Interaction (HOI) aims to identify the pairs of humans and objects in images and to recognize their relationships, ultimately forming $\langle human, object, verb \rangle$ triplets. Under default settings, HOI performance is nearly saturated, with many studies focusing on long-tail distribution and zero-shot/few-shot scenarios. Let us consider an intriguing problem:``What if there is only test dataset without training dataset, using multimodal visual foundation model in a training-free manner? '' This study uses two experimental settings: grounding truth and random arbitrary combinations. We get some interesting conclusion and find that the open vocabulary capabilities of the multimodal visual foundation model are not yet fully realized. Additionally, replacing the feature extraction with grounding DINO further confirms these findings.

📄 PDF Abstract BibTeX arXiv:2408.05772

Code (0)

등록된 구현이 없습니다.

Tasks

Human-Object Interaction Detection

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Multi-Head Attention 설명 없음

Attention 설명 없음

Vision Transformer The Vision Transformer, or ViT, is a model for image classification that employs a Transformer-like architecture over…

VIP5: Towards Multimodal Foundation Models for Recommendation

2023-05-23 · Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu 외

Computer Vision (CV), Natural Language Processing (NLP), and Recommender Systems (RecSys) are three prominent AI applications that have traditionally developed independently, resulting in disparate modeling and engineeri…

Recommendation Systems

MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge

2026-08-24 · Liangtao Shi, Jinxia Xie, Xiantao Hu, Ting Liu arxiv

In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models. We decompose the ta…

Video Object Segmentation Multimodal Reasoning Video Segmentation

Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation

2025-06-12 · Zhiyang Xu, Jiuhai Chen, Zhaojiang Lin, Xichen Pan 외

Recent advances in large language models (LLMs) have enabled multimodal foundation models to tackle both image understanding and generation within a unified framework. Despite these gains, unified models often underperfo…

Image Generation multimodal generation

freePruner: A Training-free Approach for Large Multimodal Model Acceleration

2024-11-23 · Bingxin Xu, Yuzhang Shang, Yunhao Ge, Qian Lou 외

Large Multimodal Models (LMMs) have demonstrated impressive capabilities in visual-language tasks but face significant deployment challenges due to their high computational demands. While recent token reduction methods s…

Quantization Question Answering Token Reduction Visual Question Answering

Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation

2026-05-13 · Chao Hao, Jun Xu, Ji Du, Shuo Ye 외 arxiv

Language-guided segmentation transcends the scope limitations of traditional semantic segmentation, enabling models to segment arbitrary target regions based on natural language instructions. Existing approaches typicall…

Semantic Segmentation Multimodal Reasoning Visual Reasoning