홈 › Papers

Gibbs randomness-compression proposition: An efficient deep learning

A proposition that connects randomness and compression put forward via Gibbs entropy over set of measurement vectors associated with a compression process. The proposition states that a lossy compression process is equivalent to {\it directed randomness} that preserves information content. The proposition originated from the observed behaviour in newly proposed {\it Dual Tomographic Compression} (DTC) compress-train framework. This is akin to tomographic reconstruction of layer weight matrices via building compressed sensed projections, so called {\it weight rays}. This tomographic approach is applied to previous and next layers in a dual fashion, that triggers neuronal-level pruning. This novel model compress-train scheme appear in iterative fashion and act as smart neural architecture search, Experiments demonstrated utility of this dual-tomography producing state-of-the-art performance with efficient compression during training, accelerating and supporting lottery ticket hypothesis. However, random compress-train iterations having similar performance demonstrated the connection between randomness and compression from statistical physics perspective, we formulated so called {\it Gibbs randomness-compression proposition}, signifying randomness-compression relationship via Gibbs entropy. Practically, DTC framework provides a promising approach for massively energy and resource efficient deep learning training approach.

Code (1)

Tasks

Methods 이 논문이 사용한 방법론

Similar Papers 제목 키워드 기반