Language Models Are Implicitly Continuous
Language is typically modelled with discrete sequences. However, the most successful approaches to language modelling, namely neural networks, are continuous and smooth function approximators. In this work, we show that Transformer-based language models implicitly learn to represent sentences as continuous-time functions defined over a continuous input space. This phenomenon occurs in most state-of-the-art Large Language Models (LLMs), including Llama2, Llama3, Phi3, Gemma, Gemma2, and Mistral, and suggests that LLMs reason about language in ways that fundamentally differ from humans. Our work formally extends Transformers to capture the nuances of time and space continuity in both input and output space. Our results challenge the traditional interpretation of how LLMs understand language, with several linguistic and engineering implications.
Code (1)
Tasks
Language ModellingSimilar Papers 제목 키워드 기반
Towards Learning to Speak and Hear Through Multi-Agent Communication over a Continuous Acoustic Channel
Multi-agent reinforcement learning has been used as an effective means to study emergent communication between agents, yet little focus has been given to continuous acoustic communication. This would be more akin to huma…
Language AcquisitionMulti-agent Reinforcement LearningQ-Learningreinforcement-learning+1iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learning
Learning representations for reinforcement learning (RL) has shown much promise for continuous control. We propose an efficient representation learning method using only a self-supervised latent-state consistency loss. O…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+2Bi-directional Attention with Agreement for Dependency Parsing
We develop a novel bi-directional attention model for dependency parsing, which learns to agree on headword predictions from the forward and backward parsing directions. The parsing procedure for each direction is formul…
Dependency ParsingLearning to Caption Images through a Lifetime by Asking Questions
In order to bring artificial agents into our lives, we will need to go beyond supervised learning on closed datasets to having the ability to continuously expand knowledge. Inspired by a student learning in a classroom, …
Active LearningImage CaptioningLearning to Caption Images Through a Lifetime by Asking Questions
In order to bring artificial agents into our lives, we will need to go beyond supervised learning on closed datasets to having the ability to continuously expand knowledge. Inspired by a student learning in a classroom, …
Active Learning