From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoning
Our key innovation lies in bridging the semantic gap between object detection and event understanding without requiring expensive task-specific training. The proposed plug-and-play framework interfaces with any open-vocabulary detector while extending their inherent capabilities across architectures. At its core, our approach combines (i) a symbolic regression mechanism exploring relationship patterns among detected entities and (ii) a LLM-guided strategically guiding the search toward meaningful expressions. These discovered symbolic rules transform low-level visual perception into interpretable event understanding, providing a transparent reasoning path from objects to events with strong transferability across domains.We compared our training-free framework against specialized event recognition systems across diverse application domains. Experiments demonstrate that our framework enhances multiple object detector architectures to recognize complex events such as illegal fishing activities (75% AUROC, +8.36% improvement), construction safety violations (+15.77%), and abnormal crowd behaviors (+23.16%). The code will be released soon.
Code (0)
등록된 구현이 없습니다.
Tasks
object-detectionObject DetectionSymbolic RegressionSimilar Papers 제목 키워드 기반
PolarVLM: Bridging the Semantic-Physical Gap in Vision-Language Models
Mainstream vision-language models (VLMs) fundamentally struggle with severe optical ambiguities, such as reflections and transparent objects, due to the inherent limitations of standard RGB inputs. While polarization ima…
Debiasing Event Understanding for Visual Commonsense Tasks
We study event understanding as a critical step towards visual commonsense tasks.Meanwhile, we argue that current object-based event understanding is purely likelihood-based, leading to incorrect event prediction, due to…
Multimodal Reference Visual Grounding
Visual grounding focuses on detecting objects from images based on language expressions. Recent Large Vision-Language Models (LVLMs) have significantly advanced visual grounding performance by training large models with …
Few-Shot Object DetectionVisual GroundingCLEVRER: CoLlision Events for Video REpresentation and Reasoning
The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, in…
counterfactualDescriptiveDiagnosticVisual ReasoningMoments in Time Dataset: one million videos for event understanding
We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics ev…
Action RecognitionDiversityMultimodal Activity RecognitionTemporal Action Localization