Terminating Differentiable Tree Experts
We advance the recently proposed neuro-symbolic Differentiable Tree Machine, which learns tree operations using a combination of transformers and Tensor Product Representations. We investigate the architecture and propose two key components. We first remove a series of different transformer layers that are used in every step by introducing a mixture of experts. This results in a Differentiable Tree Experts model with a constant number of parameters for any arbitrary number of steps in the computation, compared to the previous method in the Differentiable Tree Machine with a linear growth. Given this flexibility in the number of steps, we additionally propose a new termination algorithm to provide the model the power to choose how many steps to make automatically. The resulting Terminating Differentiable Tree Experts model sluggishly learns to predict the number of steps without an oracle. It can do so while maintaining the learning capabilities of the model, converging to the optimal amount of steps.
Code (0)
등록된 구현이 없습니다.
Tasks
Mixture-of-ExpertsSimilar Papers 제목 키워드 기반
Densities of Almost Surely Terminating Probabilistic Programs are Differentiable Almost Everywhere
We study the differential properties of higher-order statistical probabilistic programs with recursion and conditioning. Our starting point is an open problem posed by Hongseok Yang: what class of statistical probabilist…
Differentiable Scaffolding Tree for Molecule Optimization
The structural design of functional molecules, also called molecular optimization, is an essential chemical science and engineering task with important applications, such as drug discovery. Deep generative models and com…
Combinatorial OptimizationDrug DiscoveryGraph Neural NetworkDifferentiable Scaffolding Tree for Molecular Optimization
The structural design of functional molecules, also called molecular optimization, is an essential chemical science and engineering task with important applications, such as drug discovery. Deep generative models and com…
Combinatorial OptimizationDrug DiscoveryGraph Neural NetworkMoET: Interpretable and Verifiable Reinforcement Learning via Mixture of Expert Trees
Deep Reinforcement Learning (DRL) has led to many recent breakthroughs on complex control tasks, such as defeating the best human player in the game of Go. However, decisions made by the DRL agent are not explainable, hi…
Deep Reinforcement LearningGame of GoImitation LearningMixture-of-Experts+3MoËT: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learning
Rapid advancements in deep learning have led to many recent breakthroughs. While deep learning models achieve superior performance, often statistically better than humans, their adoption into safety-critical settings, su…
Game of GoImitation LearningMixture-of-ExpertsOpenAI Gym+4