Should we hard-code the recurrence concept or learn it instead ? Exploring the Transformer architecture for Audio-Visual Speech Recognition

2020-05-19 · George Sterpu, Christian Saam, Naomi Harte

The audio-visual speech fusion strategy AV Align has shown significant performance improvements in audio-visual speech recognition (AVSR) on the challenging LRS2 dataset. Performance improvements range between 7% and 30% depending on the noise level when leveraging the visual modality of speech in addition to the auditory one. This work presents a variant of AV Align where the recurrent Long Short-term Memory (LSTM) computation block is replaced by the more recently proposed Transformer block. We compare the two methods, discussing in greater detail their strengths and weaknesses. We find that Transformers also learn cross-modal monotonic alignments, but suffer from the same visual convergence problems as the LSTM model, calling for a deeper investigation into the dominant modality problem in machine learning.

📄 PDF Abstract BibTeX arXiv:2005.09297

Code (1)

georgesterpu/Taris tf

Tasks

Audio-Visual Speech Recognition speech-recognition Speech Recognition Visual Speech Recognition

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Multi-Head Attention 설명 없음

Adam 설명 없음

Sigmoid Activation 설명 없음

Mining Recurrent Concepts in Data Streams using the Discrete Fourier Transform

2014-06-24 · Sakthithasan Sripirakas, Russel Pears

In this research we address the problem of capturing recurring concepts in a data stream environment. Recurrence capture enables the re-use of previously learned classifiers without the need for re-learning while providi…

Meta-Learning

Integrating Pathology and CT Imaging for Personalized Recurrence Risk Prediction in Renal Cancer

2025-08-29 · Daniël Boeke, Cedrik Blommestijn, Rebecca N. Wray, Kalina Chupetlovska 외 arxiv

Recurrence risk estimation in clear cell renal cell carcinoma (ccRCC) is essential for guiding postoperative surveillance and treatment. The Leibovich score remains widely used for stratifying distant recurrence risk but…

Simple vs complex temporal recurrences for video saliency prediction

2019-07-03 · Panagiotis Linardos, Eva Mohedano, Juan Jose Nieto, Noel E. O'Connor 외

This paper investigates modifying an existing neural network architecture for static saliency prediction using two types of recurrences that integrate information from the temporal domain. The first modification is the a…

Prediction Saliency Prediction Video Saliency Detection Video Saliency Prediction

Going in circles is the way forward: the role of recurrence in visual inference

2020-03-26 · Ruben S. van Bergen, Nikolaus Kriegeskorte

Biological visual systems exhibit abundant recurrent connectivity. State-of-the-art neural network models for visual recognition, by contrast, rely heavily or exclusively on feedforward computation. Any finite-time recur…

Sliding Window Recurrences for Sequence Models

2025-12-15 · Dragos Secrieru, Garyk Brixi, Yoshua Bengio, Taiji Suzuki 외 arxiv

Multi-hybrid architectures are poised to take over language modeling due to better quality and performance. We introduce a hierarchical decomposition framework for linear recurrences that allows us to develop algorithms …