Continuous Learning in a Hierarchical Multiscale Neural Network
We reformulate the problem of encoding a multi-scale representation of a sequence in a language model by casting it in a continuous learning framework. We propose a hierarchical multi-scale language model in which short time-scale dependencies are encoded in the hidden state of a lower-level recurrent neural network while longer time-scale dependencies are encoded in the dynamic of the lower-level network by having a meta-learner update the weights of the lower-level neural network in an online meta-learning fashion. We use elastic weights consolidation as a higher-level to prevent catastrophic forgetting in our continuous learning framework.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMeta-LearningSimilar Papers 제목 키워드 기반
Hierarchical Multiscale Recurrent Neural Networks
Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolv…
Language ModellingMultiscale Combinatorial Grouping
We propose a unified approach for bottom-up hierarchical image segmentation and object candidate generation for recognition, called Multiscale Combinatorial Grouping (MCG). For this purpose, we first develop a fast norma…
Image SegmentationObjectSegmentationSemantic SegmentationEfficient Multiscale Gaussian Process Regression using Hierarchical Clustering
Standard Gaussian Process (GP) regression, a powerful machine learning tool, is computationally expensive when it is applied to large datasets, and potentially inaccurate when data points are sparsely distributed in a hi…
ClusteringregressionMultiscale Graph Construction Using Non-local Cluster Features
This paper presents a multiscale graph construction method using both graph and signal features. Multiscale graph is a hierarchical representation of the graph, where a node at each level indicates a cluster in a finer r…
ClusteringGraph Clusteringgraph constructionPoint Cloud SegmentationA Multiscale Graph Convolutional Network Using Hierarchical Clustering
The information contained in hierarchical topology, intrinsic to many networks, is currently underutilised. A novel architecture is explored which exploits this information through a multiscale decomposition. A dendrogra…
ClusteringMolecular Property PredictionPredictionProperty Prediction+1