Encoding and Controlling Global Semantics for Long-form Video Question Answering

2024-05-30 · Thong Thanh Nguyen, Zhiyuan Hu, Xiaobao Wu, Cong-Duy T Nguyen, See-Kiong Ng, Anh Tuan Luu

Seeking answers effectively for long videos is essential to build video question answering (videoQA) systems. Previous methods adaptively select frames and regions from long videos to save computations. However, this fails to reason over the whole sequence of video, leading to sub-optimal performance. To address this problem, we introduce a state space layer (SSL) into multi-modal Transformer to efficiently integrate global semantics of the video, which mitigates the video information loss caused by frame and region selection modules. Our SSL includes a gating unit to enable controllability over the flow of global semantics into visual representations. To further enhance the controllability, we introduce a cross-modal compositional congruence (C^3) objective to encourage global semantics aligned with the question. To rigorously evaluate long-form videoQA capacity, we construct two new benchmarks Ego-QA and MAD-QA featuring videos of considerably long length, i.e. 17.5 minutes and 1.9 hours, respectively. Extensive experiments demonstrate the superiority of our framework on these new as well as existing datasets. The code, model, and data have been made available at https://nguyentthong.github.io/Long_form_VideoQA.

📄 PDF Abstract BibTeX arXiv:2405.19723

Code (1)

zhiyuanhubj/long_form_videoqa 공식 구현 pytorch

Tasks

FormQuestion Answering Video Question Answering

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Residual Connection 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Multi-Head Attention 설명 없음

Enriching and Controlling Global Semantics for Text Summarization

2021-09-22 · EMNLP 2021 11 · Thong Nguyen, Anh Tuan Luu, Truc Lu, Tho Quan

Recently, Transformer-based models have been proven effective in the abstractive summarization task by creating fluent and informative summaries. Nevertheless, these models still suffer from the short-range dependency pr…

Abstractive Text Summarization Text Generation Text Summarization

REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion

2025-12-18 · Giorgos Petsangourakis, Christos Sgouropoulos, Bill Psomas, Theodoros Giannakopoulos 외 arxiv

Latent diffusion models (LDMs) achieve state-of-the-art image synthesis, yet their reconstruction-style denoising objective provides only indirect semantic supervision: high-level semantics emerge slowly, requiring longe…

SIMMF: Semantics-aware Interactive Multiagent Motion Forecasting for Autonomous Vehicle Driving

2023-06-26 · Vidyaa Krishnan Nivash, Ahmed H. Qureshi

Autonomous vehicles require motion forecasting of their surrounding multiagents (pedestrians and vehicles) to make optimal decisions for navigation. The existing methods focus on techniques to utilize the positions and v…

Autonomous Driving Autonomous Vehicles Motion Forecasting motion prediction

Using ASP(Q) to Handle Inconsistent Prioritized Data

2026-04-23 · Meghyn Bienvenu, Camille Bourgaux, Robin Jean, Giuseppe Mazzotta arxiv

We explore the use of answer set programming (ASP) and its extension with quantifiers, ASP(Q), for inconsistency-tolerant querying of prioritized data, where a priority relation between conflicting facts is exploited to …

HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation

2024-10-28 · Yuhan Chen, Ang Lv, Jian Luan, Bin Wang 외

Many positional encodings (PEs) are designed to exhibit long-term decay, based on an entrenched and long-standing inductive opinion: tokens farther away from the current position carry less relevant information. We argue…

Position