Performance-guaranteed regularization in maximum likelihood method: Gauge symmetry in Kullback -- Leibler divergence
The maximum likelihood method is the best-known method for estimating the probabilities behind the data. However, the conventional method obtains the probability model closest to the empirical distribution, resulting in overfitting. Then regularization methods prevent the model from being excessively close to the wrong probability, but little is known systematically about their performance. The idea of regularization is similar to error-correcting codes, which obtain optimal decoding by mixing suboptimal solutions with an incorrectly received code. The optimal decoding in error-correcting codes is achieved based on gauge symmetry. We propose a theoretically guaranteed regularization in the maximum likelihood method by focusing on a gauge symmetry in Kullback -- Leibler divergence. In our approach, we obtain the optimal model without the need to search for hyperparameters frequently appearing in regularization.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Feasible Invertibility Conditions for Maximum Likelihood Estimation for Observation-Driven Models
Invertibility conditions for observation-driven time series models often fail to be guaranteed in empirical applications. As a result, the asymptotic theory of maximum likelihood and quasi-maximum likelihood estimators m…
Time SeriesTime Series AnalysisTowards Understanding Distributional Reinforcement Learning: Regularization, Optimization, Acceleration and Sinkhorn Algorithm
Distributional reinforcement learning~(RL) is a class of state-of-the-art algorithms that estimate the whole distribution of the total return rather than only its expectation. Despite the remarkable performance of distri…
Atari GamesDistributional Reinforcement Learningreinforcement-learningReinforcement Learning (RL)A globally optimal fast iterative linear maximum likelihood classifier
A novel iterative linear classification algorithm is developed from a maximum likelihood (ML) linear classifier. The main contribution of this paper is the discovery that a well-known maximum likelihood linear classifier…
ClassificationNeural KEM: A Kernel Method with Deep Coefficient Prior for PET Image Reconstruction
Image reconstruction of low-count positron emission tomography (PET) data is challenging. Kernel methods address the challenge by incorporating image prior information in the forward model of iterative PET image reconstr…
Image ReconstructionKnown Unknowns: Uncertainty Quality in Bayesian Neural Networks
We evaluate the uncertainty quality in neural networks using anomaly detection. We extract uncertainty measures (e.g. entropy) from the predictions of candidate models, use those measures as features for an anomaly detec…
Anomaly DetectionKnown Unknowns