LLM-EvRep: Learning an LLM-Compatible Event Representation Using a Self-Supervised Framework
Recent advancements in event-based recognition have demonstrated significant promise, yet most existing approaches rely on extensive training, limiting their adaptability for efficient processing of event-driven visual content. Meanwhile, large language models (LLMs) have exhibited remarkable zero-shot capabilities across diverse domains, but their application to event-based visual recognition remains largely unexplored. To bridge this gap, we propose \textbf{LLM-EvGen}, an event representation generator that produces LLM-compatible event representations \textbf{LLM-EvRep}, thereby enhancing the performance of LLMs on event recognition tasks. The generator is trained using a self-supervised framework, aligning the generated representations with semantic consistency and structural fidelity. Comprehensive experiments were conducted on three datasets: N-ImageNet, N-Caltech101, and N-MNIST. The results demonstrate that our method, \textbf{LLM-EvRep}, outperforms the event-to-video method, E2VID, by 15.93\%, 0.82\%, and 50.21\%, respectively, in recognition tasks when evaluated using GPT-4o.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
EvRepSL: Event-Stream Representation via Self-Supervised Learning for Event-Based Vision
Event-stream representation is the first step for many computer vision tasks using event cameras. It converts the asynchronous event-streams into a formatted structure so that conventional machine learning models can be …
Event-based visionOptical Flow EstimationSelf-Supervised LearningiQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning
Learning representations for reinforcement learning (RL) has shown much promise for continuous control. We propose an efficient representation learning method using only a self-supervised latent-state consistency loss. O…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+2Self-Supervised Models are Continual Learners
Self-supervised models have been shown to produce comparable or better visual representations than their supervised counterparts when trained offline on unlabeled data at scale. However, their efficacy is catastrophicall…
Continual LearningRepresentation LearningImproving out-of-distribution generalization via multi-task self-supervised pretraining
Self-supervised feature representations have been shown to be useful for supervised classification, few-shot learning, and adversarial robustness. We show that features obtained using self-supervised learning are compara…
Adversarial RobustnessDomain GeneralizationFew-Shot LearningMulti-Task Learning+2Polynomial Contrastive Learning for Privacy-Preserving Representation Learning on Graphs
Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations on graph data without requiring manual labels. However, leading SSL methods like GRACE are fundamentally incompatible with pr…
Graph Representation LearningSelf-Supervised LearningContrastive Learning