Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding
Multimodal foundation models (MFMs) have demonstrated significant success in tasks such as visual captioning, question answering, and image-text retrieval. However, these models face inherent limitations due to their finite internal capacity, which restricts their ability to process extended temporal sequences, a crucial requirement for comprehensive video and audio analysis. To overcome these challenges, we introduce a specialized cognitive module, temporal working memory (TWM), which aims to enhance the temporal modeling capabilities of MFMs. It selectively retains task-relevant information across temporal dimensions, ensuring that critical details are preserved throughout the processing of video and audio content. The TWM uses a query-guided attention approach to focus on the most informative multimodal segments within temporal sequences. By retaining only the most relevant content, TWM optimizes the use of the model's limited capacity, enhancing its temporal modeling ability. This plug-and-play module can be easily integrated into existing MFMs. With our TWM, nine state-of-the-art models exhibit significant performance improvements across tasks such as video captioning, question answering, and video-text retrieval. By enhancing temporal modeling, TWM extends the capability of MFMs to handle complex, time-sensitive data effectively. Our code is available at https://github.com/xid32/NAACL_2025_TWM.
Code (1)
Tasks
Image CaptioningImage-text RetrievalQuestion AnsweringRetrievalTemporal SequencesText RetrievalVideo CaptioningVideo-Text RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SAM-DAQ: Segment Anything Model with Depth-guided Adaptive Queries for RGB-D Video Salient Object Detection
Recently segment anything model (SAM) has attracted widespread concerns, and it is often treated as a vision foundation model for universal segmentation. Some researchers have attempted to directly apply the foundation m…
Video Salient Object DetectionLocal-Global Context Aware Transformer for Language-Guided Video Segmentation
We explore the task of language-guided video segmentation (LVS). Previous algorithms mostly adopt 3D CNNs to learn video representation, struggling to capture long-term context and easily suffering from visual-linguistic…
Referring Expression SegmentationReferring Video Object SegmentationSegmentationSemantic Segmentation+4TEDi: Temporal Memory-Enhanced and Denoising Transformer for Surgical Instrument Segmentation
Query-based segmentation methods have shown promising potential for surgical instrument segmentation and recognition, which is essential for scene understanding and downstream tasks in computer assisted surgery. However,…
Scene UnderstandingAddressing Issues with Working Memory in Video Object Segmentation
Contemporary state-of-the-art video object segmentation (VOS) models compare incoming unannotated images to a history of image-mask relations via affinity or cross-attention to predict object masks. We refer to the inter…
Inductive BiasObjectSemantic SegmentationVideo Object Segmentation+1DeVOS: Flow-Guided Deformable Transformer for Video Object Segmentation
The recent works on Video Object Segmentation achieved remarkable results by matching dense semantic and instance-level features between the current and previous frames for long-time propagation. Nevertheless, global fea…
Optical Flow EstimationSemantic SegmentationVideo Object SegmentationVideo Semantic Segmentation