Learning Scalable Deep Kernels with Recurrent Structure
Many applications in speech, robotics, finance, and biology deal with sequential data, where ordering matters and recurrent structures are common. However, this structure cannot be easily captured by standard kernel functions. To model such structure, we propose expressive closed-form kernel functions for Gaussian processes. The resulting model, GP-LSTM, fully encapsulates the inductive biases of long short-term memory (LSTM) recurrent networks, while retaining the non-parametric probabilistic advantages of Gaussian processes. We learn the properties of the proposed kernels by optimizing the Gaussian process marginal likelihood using a new provably convergent semi-stochastic gradient procedure and exploit the structure of these kernels for scalable training and prediction. This approach provides a practical representation for Bayesian LSTMs. We demonstrate state-of-the-art performance on several benchmarks, and thoroughly investigate a consequential autonomous driving application, where the predictive uncertainties provided by GP-LSTM are uniquely valuable.
Code (2)
Tasks
Autonomous DrivingGaussian ProcessesSmart Grid PredictionSimilar Papers 제목 키워드 기반
Deriving Neural Architectures from Sequence and Graph Kernels
The design of neural architectures for structured objects is typically guided by experimental insights rather than a formal process. In this work, we appeal to kernels over combinatorial structures, such as sequences and…
Graph RegressionLanguage ModelingLanguage ModellingregressionDeep Kernel Learning
We introduce scalable deep kernels, which combine the structural properties of deep learning architectures with the non-parametric flexibility of kernel methods. Specifically, we transform the inputs of a spectral mixtur…
Gaussian ProcessesFrom Node Embedding to Graph Embedding: Scalable Global Graph Kernel via Random Features
Graph kernels are one of the most important methods for graph data analysis and have been successfully applied in diverse applications. We can generally categorize existing graph kernels into two groups: kernels based on…
Graph EmbeddingScalable Gaussian Processes for Characterizing Multidimensional Change Surfaces
We present a scalable Gaussian process model for identifying and characterizing smooth multidimensional changepoints, and automatically learning changes in expressive covariance structure. We use Random Kitchen Sink feat…
Gaussian ProcessesReservoir Computing meets Recurrent Kernels and Structured Transforms
Reservoir Computing is a class of simple yet efficient Recurrent Neural Networks where internal weights are fixed at random and only a linear output layer is trained. In the large size limit, such random neural networks …
Time SeriesTime Series AnalysisTime Series Prediction