Gaussian Kernel-based Cross Modal Network for Spatio-Temporal Video Grounding

2022-07-02 · Zeyu Xiong, Daizong Liu, Pan Zhou

Spatial-Temporal Video Grounding (STVG) is a challenging task which aims to localize the spatio-temporal tube of the interested object semantically according to a natural language query. Most previous works not only severely rely on the anchor boxes extracted by Faster R-CNN, but also simply regard the video as a series of individual frames, thus lacking their temporal modeling. Instead, in this paper, we are the first to propose an anchor-free framework for STVG, called Gaussian Kernel-based Cross Modal Network (GKCMN). Specifically, we utilize the learned Gaussian Kernel-based heatmaps of each video frame to locate the query-related object. A mixed serial and parallel connection network is further developed to leverage both spatial and temporal relations among frames for better grounding. Experimental results on VidSTG dataset demonstrate the effectiveness of our proposed GKCMN.

📄 PDF Abstract BibTeX arXiv:2207.00744

Code (0)

등록된 구현이 없습니다.

Tasks

Spatio-Temporal Video Grounding Video Grounding

Methods 이 논문이 사용한 방법론

RoIPool 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

RPN A Region Proposal Network, or RPN, is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Faster R-CNN Faster R-CNN is an object detection model that improves on Fast R-CNN by utilising a region proposal network…

Non-separable Spatio-temporal Graph Kernels via SPDEs

2021-11-16 · Alexander Nikitin, ST John, Arno Solin, Samuel Kaski

Gaussian processes (GPs) provide a principled and direct approach for inference and learning on graphs. However, the lack of justified graph kernels for spatio-temporal modelling has held back their use in graph problems…

Gaussian Processes

Scalable high-resolution forecasting of sparse spatiotemporal events with kernel methods: a winning solution to the NIJ "Real-Time Crime Forecasting Challenge"

2018-01-09 · Seth Flaxman, Michael Chirico, Pau Pereira, Charles Loeffler

We propose a generic spatiotemporal event forecasting method, which we developed for the National Institute of Justice's (NIJ) Real-Time Crime Forecasting Challenge. Our method is a spatiotemporal forecasting model combi…

Density Estimation Gaussian Processes

Probabilistic Spatiotemporal Modeling of Day-Ahead Wind Power Generation with Input-Warped Gaussian Processes

2024-09-10 · Qiqi Li, Mike Ludkovski

We design a Gaussian Process (GP) spatiotemporal model to capture features of day-ahead wind power forecasts. We work with hourly-scale day-ahead forecasts across hundreds of wind farm locations, with the main aim of con…

Gaussian Processes

Imitation Learning of Neural Spatio-Temporal Point Processes

2019-06-13 · Shixiang Zhu, Shuang Li, Zhigang Peng, Yao Xie

We present a novel Neural Embedding Spatio-Temporal (NEST) point process model for spatio-temporal discrete event data and develop an efficient imitation learning (a type of reinforcement learning) based approach for mod…

Computational Efficiency Imitation Learning Point Processes

Non-separable Covariance Kernels for Spatiotemporal Gaussian Processes based on a Hybrid Spectral Method and the Harmonic Oscillator

2023-02-19 · Dionissios T. Hristopulos

Gaussian processes provide a flexible, non-parametric framework for the approximation of functions in high-dimensional spaces. The covariance kernel is the main engine of Gaussian processes, incorporating correlations th…

Computational Efficiency Gaussian Processes