SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering

2024-11-07 · Tianyu Yang, Yiyang Nan, Lisen Dai, Zhenwen Liang, Yapeng Tian, Xiangliang Zhang

Audio-Visual Question Answering (AVQA) is a challenging task that involves answering questions based on both auditory and visual information in videos. A significant challenge is interpreting complex multi-modal scenes, which include both visual objects and sound sources, and connecting them to the given question. In this paper, we introduce the Source-aware Semantic Representation Network (SaSR-Net), a novel model designed for AVQA. SaSR-Net utilizes source-wise learnable tokens to efficiently capture and align audio-visual elements with the corresponding question. It streamlines the fusion of audio and visual information using spatial and temporal attention mechanisms to identify answers in multi-modal scenes. Extensive experiments on the Music-AVQA and AVQA-Yang datasets show that SaSR-Net outperforms state-of-the-art AVQA methods.

📄 PDF Abstract BibTeX arXiv:2411.04933

Code (0)

등록된 구현이 없습니다.

Tasks

Audio-visual Question AnsweringAudio-Visual Question Answering (AVQA)Question Answering Visual Question Answering

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

eSASRec: Enhancing Transformer-based Recommendations in a Modular Fashion

2025-08-08 · Daria Tikhonovich, Nikita Zelinskiy, Aleksandr V. Petrov, Mayya Spirina 외 arxiv

Since their introduction, Transformer-based models, such as SASRec and BERT4Rec, have become common baselines for sequential recommendations, surpassing earlier neural and non-neural methods. A number of following public…

A Challenging Benchmark of Anime Style Recognition

2022-04-29 · Haotang Li, Shengtao Guo, Kailin Lyu, Xiao Yang 외

Given two images of different anime roles, anime style recognition (ASR) aims to learn abstract painting style to determine whether the two images are from the same work, which is an interesting but challenging problem. …

Art Analysis Face Recognition Person Re-Identification

TiSASRec: Time Interval Aware Self-Attention for Sequential Recommendation

2020-01-01 · Jiacheng Li, Yujie Wang, Julian McAuley

Sequential recommender systems seek to exploit the order of users' interactions, in order to predict their next action based on the context of what they have done recently. Traditionally, Markov Chains(MCs), and more rec…

Recommendation Systems Sequential Recommendation

When Transformers Meet Recommenders: Integrating Self-Attentive Sequential Recommendation with Fine-Tuned LLMs

2025-07-08 · Kechen Liu arxiv

Self-Attentive Sequential Recommendation (SASRec) effectively captures long-term user preferences by applying attention mechanisms to historical interactions. Concurrently, the rise of Large Language Models (LLMs) has mo…

Sequential Recommendation Collaborative Filtering

Scaling Session-Based Transformer Recommendations using Optimized Negative Sampling and Loss Functions

2023-07-27 · Timo Wilm, Philipp Normann, Sophie Baumeister, Paul-Vincent Kobow

This work introduces TRON, a scalable session-based Transformer Recommender using Optimized Negative-sampling. Motivated by the scalability and performance limitations of prevailing models such as SASRec and GRU4Rec+, TR…

Product Recommendation Recommendation Systems Session-Based Recommendations