Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading
The goal of this work is to recognize words, phrases, and sentences being spoken by a talking face without given the audio. Current deep learning approaches for lip reading focus on exploring the appearance and optical flow information of videos. However, these methods do not fully exploit the characteristics of lip motion. In addition to appearance and optical flow, the mouth contour deformation usually conveys significant information that is complementary to others. However, the modeling of dynamic mouth contour has received little attention than that of appearance and optical flow. In this work, we propose a novel model of dynamic mouth contours called Adaptive Semantic-Spatio-Temporal Graph Convolution Network (ASST-GCN), to go beyond previous methods by automatically learning both the spatial and temporal information from videos. To combine the complementary information from appearance and mouth contour, a two-stream visual front-end network is proposed. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art lip reading methods on several large-scale lip reading benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Landmark-based LipreadingLip ReadingOptical Flow EstimationSimilar Papers 제목 키워드 기반
Ada-TransGNN: An Air Quality Prediction Model Based On Adaptive Graph Convolutional Networks
Accurate air quality prediction is becoming increasingly important in the environmental field. To address issues such as low prediction accuracy and slow real-time updates in existing models, which lead to lagging predic…
Graph structure learningDD-GCN: Directed Diffusion Graph Convolutional Network for Skeleton-based Human Action Recognition
Graph Convolutional Networks (GCNs) have been widely used in skeleton-based human action recognition. In GCN-based methods, the spatio-temporal graph is fundamental for capturing motion patterns. However, existing approa…
Action RecognitionTemporal Action LocalizationSpatio-Temporal Joint Graph Convolutional Networks for Traffic Forecasting
Recent studies have shifted their focus towards formulating traffic forecasting as a spatio-temporal graph modeling problem. Typically, they constructed a static spatial graph at each time step and then connected each no…
DSTSA-GCN: Advancing Skeleton-Based Gesture Recognition with Semantic-Aware Spatio-Temporal Topology Modeling
Graph convolutional networks (GCNs) have emerged as a powerful tool for skeleton-based action and gesture recognition, thanks to their ability to model spatial and temporal dependencies in skeleton data. However, existin…
Action RecognitionGesture RecognitionHand Gesture RecognitionSkeleton Based Action RecognitionMegaCRN: Meta-Graph Convolutional Recurrent Network for Spatio-Temporal Modeling
Spatio-temporal modeling as a canonical task of multivariate time series forecasting has been a significant research topic in AI community. To address the underlying heterogeneity and non-stationarity implied in the grap…
DecoderGraph LearningGraph structure learningMultivariate Time Series Forecasting+4