Maximum Entropy Auto-Encoding
In this paper, it is shown that an auto-encoder using optimal reconstruction significantly outperforms a conventional auto-encoder. Optimal reconstruction uses the conditional mean of the input given the features, under a maximum entropy prior distribution. The optimal reconstruction network, which is called deterministic projected belied network (D-PBN), resembles a standard reconstruction network, but with special non-linearities that mist be iteratively solved. The method, which can be seen as a generalization of maximum entropy image reconstruction, extends to multiple layers. In experiments, mean square reconstruction error reduced by up to a factor of two. The performance improvement diminishes for deeper networks, or for input data with unconstrained values (Gaussian assumption).
Code (0)
등록된 구현이 없습니다.
Tasks
Image ReconstructionSimilar Papers 제목 키워드 기반
Matrix Information Theory for Self-Supervised Learning
The maximum entropy encoding framework provides a unified perspective for many non-contrastive learning methods like SimSiam, Barlow Twins, and MEC. Inspired by this framework, we introduce Matrix-SSL, a novel approach t…
Contrastive LearningGSM8KLanguage ModelingLanguage Modelling+4Encoding Weights of Irregular Sparsity for Fixed-to-Fixed Model Compression
Even though fine-grained pruning techniques achieve a high compression ratio, conventional sparsity representations (such as CSR) associated with irregular sparsity degrade parallelism significantly. Practical pruning me…
Model CompressionA Scheme for Molecular Computation of Maximum Likelihood Estimators for Log-Linear Models
We propose a novel molecular computing scheme for statistical inference. We focus on the much-studied statistical inference problem of computing maximum likelihood estimators for log-linear models. Our scheme takes log-l…
MIM: Mutual Information Machine
We introduce the Mutual Information Machine (MIM), a probabilistic auto-encoder for learning joint distributions over observations and latent variables. MIM reflects three design principles: 1) low divergence, to encoura…
ClusteringDecoderEfficient Learned Image Compression without Entropy Coding
Entropy coding is widely used in typical learned image compression (LIC) that converts latents into a compact bitstream. However, entropy coding is typically sequential and becomes the coding latency bottleneck. To overc…
Image Compression