Facilitate the Parametric Dimension Reduction by Gradient Clipping
We extend a well-known dimension reduction method, t-distributed stochastic neighbor embedding (t-SNE), from non-parametric to parametric by training neural networks. The main advantage of a parametric technique is the generalization of handling new data, which is particularly beneficial for streaming data exploration. However, training a neural network to optimize the t-SNE objective function frequently fails. Previous methods overcome this problem by pre-training and then fine-tuning the network. We found that the training failure comes from the gradient exploding problem, which occurs when data points distant in high-dimensional space are projected to nearby embedding positions. Accordingly, we applied the gradient clipping method to solve the problem. Since the networks are trained by directly optimizing the t-SNE objective function, our method achieves an embedding quality that is compatible with the non-parametric t-SNE while enjoying the ability of generalization. Due to mini-batch network training, our parametric dimension reduction method is highly efficient. We further extended other non-parametric state-of-the-art approaches, such as LargeVis and UMAP, to the parametric versions. Experiment results demonstrate the feasibility of our method. Considering its practicability, we will soon release the codes for public use.
Code (0)
등록된 구현이 없습니다.
Tasks
Dimensionality ReductionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dimension Reduction and MARS
The multivariate adaptive regression spline (MARS) is one of the popular estimation methods for nonparametric multivariate regressions. However, as MARS is based on marginal splines, to incorporate interactions of covari…
Dimensionality ReductionregressionInference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descent
Differentially private stochastic gradient descent (DP-SGD) is known to have poorer training and test performance on large neural networks, compared to ordinary stochastic gradient descent (SGD). In this paper, we perfor…
Dimensionality ReductionAn Adaptive Differentially Private Federated Learning Framework
Federated learning enables collaborative model training across distributed clients while preserving data privacy. However, in practical deployments, device heterogeneity and non-independent and identically distributed (N…
Dimensionality ReductionFederated LearningTo Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions
The success of modern machine learning is due in part to the adaptive optimization methods that have been developed to deal with the difficulties of training large models over complex datasets. One such method is gradien…
SchedulingSoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training
Differential privacy (DP) provides strong protection for sensitive data, but often reduces model performance and fairness, especially for underrepresented groups. One major reason is gradient clipping in DP-SGD, which ca…