Context-Enhanced Memory-Refined Transformer for Online Action Detection
Online Action Detection (OAD) detects actions in streaming videos using past observations. State-of-the-art OAD approaches model past observations and their interactions with an anticipated future. The past is encoded using short- and long-term memories to capture immediate and long-range dependencies, while anticipation compensates for missing future context. We identify a training-inference discrepancy in existing OAD methods that hinders learning effectiveness. The training uses varying lengths of short-term memory, while inference relies on a full-length short-term memory. As a remedy, we propose a Context-enhanced Memory-Refined Transformer (CMeRT). CMeRT introduces a context-enhanced encoder to improve frame representations using additional near-past context. It also features a memory-refined decoder to leverage near-future generation to enhance performance. CMeRT achieves state-of-the-art in online detection and anticipation on THUMOS'14, CrossTask, and EPIC-Kitchens-100.
Code (1)
Tasks
Action DetectionDecoderOnline Action DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
History-Enhanced Two-Stage Transformer for Aerial Vision-and-Language Navigation
Aerial Vision-and-Language Navigation (AVLN) requires Unmanned Aerial Vehicle (UAV) agents to localize targets in large-scale urban environments based on linguistic instructions. While successful navigation demands both …
Context Enhanced Transformer for Single Image Object Detection
With the increasing importance of video data in real-world applications, there is a rising need for efficient object detection methods that utilize temporal information. While existing video object detection (VOD) techni…
Objectobject-detectionObject DetectionVideo Object DetectionATLAS: Learning to Optimally Memorize the Context at Test Time
Transformers have been established as the most popular backbones in sequence modeling, mainly due to their effectiveness in in-context retrieval tasks and the ability to learn at scale. Their quadratic memory and time co…
Common Sense ReasoningLanguage ModelingLanguage ModellingLong-Context UnderstandingEDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation
Handling complex or nonlinear motion patterns has long posed challenges for video frame interpolation. Although recent advances in diffusion-based methods offer improvements over traditional optical flow-based approaches…
Optical Flow EstimationVideo Frame InterpolationLM2: Large Memory Models
This paper introduces the Large Memory Model (LM2), a decoder-only Transformer architecture enhanced with an auxiliary memory module that aims to address the limitations of standard Transformers in multi-step reasoning, …
DecoderMMLUQuestion Answering