Event-Based Video Reconstruction Using Transformer
Event cameras, which output events by detecting spatio-temporal brightness changes, bring a novel paradigm to image sensors with high dynamic range and low latency. Previous works have achieved impressive performances on event-based video reconstruction by introducing convolutional neural networks (CNNs). However, intrinsic locality of convolutional operations is not capable of modeling long-range dependency, which is crucial to many vision tasks. In this paper, we present a hybrid CNN-Transformer network for event-based video reconstruction (ET-Net), which merits the fine local information from CNN and global contexts from Transformer. In addition, we further propose a Token Pyramid Aggregation strategy to implement multi-scale token integration for relating internal and intersected semantic concepts in the token-space. Experimental results demonstrate that our proposed method achieves superior performance over state-of-the-art methods on multiple real-world event datasets. The code is available at https://github.com/WarranWeng/ET-Net
Code (1)
Tasks
Event-based Object SegmentationEvent-Based Video ReconstructionVideo ReconstructionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
EventMamba: Enhancing Spatio-Temporal Locality with State Space Models for Event-Based Video Reconstruction
Leveraging its robust linear global modeling capability, Mamba has notably excelled in computer vision. Despite its success, existing Mamba-based vision models have overlooked the nuances of event-driven tasks, especiall…
Event-Based Video ReconstructionMambaState Space ModelsVideo ReconstructionEvent-to-Video Reconstruction using Spatio-Temporal and Frequency-Enhanced Deep Neural Networks
Event cameras offer significant advantages over conventional frame-based counterparts, including high temporal resolution, low latency, and energy efficiency. These characteristics make them suitable for high-speed and h…
Video ReconstructionScene UnderstandingSensing Diversity and Sparsity Models for Event Generation and Video Reconstruction from Events
Events-to-video (E2V) reconstruction and video-to-events (V2E) simulation are two fundamental research topics in event-based vision. Current deep neural networks for E2V reconstruction are usually complex and difficult t…
DiversityEvent-based visionVideo ReconstructionMulti-Contextual Predictions with Vision Transformer for Video Anomaly Detection
Video Anomaly Detection(VAD) has been traditionally tackled in two main methodologies: the reconstruction-based approach and the prediction-based one. As the reconstruction-based methods learn to generalize the input ima…
Anomaly DetectionVideo Anomaly DetectionEVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction
Event cameras are a new type of vision sensor that incorporates asynchronous and independent pixels, offering advantages over traditional frame-based cameras such as high dynamic range and minimal motion blur. However, t…
Event-Based Video ReconstructionEvent-based visionVideo Reconstruction