Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network

2021-04-28 · SeungHyun Lee, Byung Cheol Song

Knowledge distillation (KD) is one of the most useful techniques for light-weight neural networks. Although neural networks have a clear purpose of embedding datasets into the low-dimensional space, the existing knowledge was quite far from this purpose and provided only limited information. We argue that good knowledge should be able to interpret the embedding procedure. This paper proposes a method of generating interpretable embedding procedure (IEP) knowledge based on principal component analysis, and distilling it based on a message passing neural network. Experimental results show that the student network trained by the proposed KD method improves 2.28% in the CIFAR100 dataset, which is higher performance than the state-of-the-art (SOTA) method. We also demonstrate that the embedding procedure knowledge is interpretable via visualization of the proposed KD process. The implemented code is available at https://github.com/sseung0703/IEPKT.

📄 PDF Abstract BibTeX arXiv:2104.13561

Code (1)

sseung0703/IEPKT tf

Tasks

Graph Neural Network Knowledge Distillation Transfer Learning

Physiological Signal Embeddings (PHASE) via Interpretable Stacked Models

2019-05-01 · ICLR 2019 5 · Hugh Chen, Scott Lundberg, Gabe Erion, Su-In Lee

In health, machine learning is increasingly common, yet neural network embedding (representation) learning is arguably under-utilized for physiological signals. This inadequacy stands out in stark contrast to more tradi…

Network Embedding Representation Learning

Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games

2020-10-22 · NeurIPS 2020 12 · Yunqiu Xu, Meng Fang, Ling Chen, Yali Du 외

We study reinforcement learning (RL) for text-based games, which are interactive simulations in the context of natural language. While different methods have been developed to represent the environment information and la…

Decision Making Deep Reinforcement Learning Knowledge Graphs reinforcement-learning+2

Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Search

2021-11-15 · Zixiang Ding, Yaran Chen, Nannan Li, Dongbin Zhao 외

Different from other deep scalable architecture-based NAS approaches, Broad Neural Architecture Search (BNAS) proposes a broad scalable architecture which consists of convolution and enhancement blocks, dubbed Broad Conv…

GPU Neural Architecture Search

Interpretable ICD Code Embeddings with Self- and Mutual-Attention Mechanisms

2019-06-13 · Dixin Luo, Hongteng Xu, Lawrence Carin

We propose a novel and interpretable embedding method to represent the international statistical classification codes of diseases and related health problems (i.e., ICD codes). This method considers a self-attention mech…

Knowledge transfer for surgical activity prediction

2017-11-15 · Olga Dergachyova, Xavier Morandi, Pierre Jannin

Lack of training data hinders automatic recognition and prediction of surgical activities necessary for situation-aware operating rooms. We propose using knowledge transfer to compensate for data deficit and improve pred…

Activity Prediction Prediction Transfer Learning