Lattice Representation Learning
In this article we introduce theory and algorithms for learning discrete representations that take on a lattice that is embedded in an Euclidean space. Lattice representations possess an interesting combination of properties: a) they can be computed explicitly using lattice quantization, yet they can be learned efficiently using the ideas we introduce in this paper, b) they are highly related to Gaussian Variational Autoencoders, allowing designers familiar with the latter to easily produce discrete representations from their models and c) since lattices satisfy the axioms of a group, their adoption can lead into a way of learning simple algebras for modeling binary operations between objects through symbolic formalisms, yet learn these structures also formally using differentiation techniques. This article will focus on laying the groundwork for exploring and exploiting the first two properties, including a new mathematical result linking expressions used during training and inference time and experimental validation on two popular datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationRepresentation LearningSimilar Papers 제목 키워드 기반
Lattice Representation Learning
We introduce the notion of \emph{lattice representation learning}, in which the representation for some object of interest (e.g. a sentence or an image) is a lattice point in an Euclidean space. Our main contribution is …
DecoderQuantizationRepresentation LearningSentenceA representation theorem for events within lattice structures of state-spaces
For the standard lattice model of information structures, we derive a reduced poset representation which provides the same informational content as the complete lattice structure which derives it. Rational agents can rec…
Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models
Chinese pre-trained language models usually process text as a sequence of characters, while ignoring more coarse granularity, e.g., words. In this work, we propose a novel pre-training paradigm for Chinese -- Lattice-BER…
Natural Language UnderstandingSentenceSketchLattice: Latticed Representation for Sketch Manipulation
The key challenge in designing a sketch representation lies with handling the abstract and iconic nature of sketches. Existing work predominantly utilizes either, (i) a pixelative format that treats sketches as natural i…
The Lattice Representation Hypothesis of Large Language Models
We propose the Lattice Representation Hypothesis of large language models: a symbolic backbone that grounds conceptual hierarchies and logical operations in embedding geometry. Our framework unifies the Linear Representa…