NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning
Language models struggle with handling numerical data and performing arithmetic operations. We hypothesize that this limitation can be partially attributed to non-intuitive textual numbers representation. When a digit is read or generated by a causal language model it does not know its place value (e.g. thousands vs. hundreds) until the entire number is processed. To address this issue, we propose a simple adjustment to how numbers are represented by including the count of digits before each number. For instance, instead of "42", we suggest using "{2:42}" as the new format. This approach, which we term NumeroLogic, offers an added advantage in number generation by serving as a Chain of Thought (CoT). By requiring the model to consider the number of digits first, it enhances the reasoning process before generating the actual number. We use arithmetic tasks to demonstrate the effectiveness of the NumeroLogic formatting. We further demonstrate NumeroLogic applicability to general natural language modeling, improving language understanding performance in the MMLU benchmark.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMMLUSimilar Papers 제목 키워드 기반
Efficient numeracy in language models through single-token number embeddings
To drive progress in science and engineering, large language models (LLMs) must be able to process large amounts of numerical data and solve long calculations efficiently. This is currently only possible through the use …
DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving
Large language models (LLMs) have shown great promise for autonomous driving. However, discretizing numbers into tokens limits precise numerical reasoning, fails to reflect the positional significance of digits in the tr…
Trajectory PredictionAutonomous DrivingChatMol: A Versatile Molecule Designer Based on the Numerically Enhanced Large Language Model
Goal-oriented de novo molecule design, namely generating molecules with specific property or substructure constraints, is a crucial yet challenging task in drug discovery. Existing methods, such as Bayesian optimization …
Bayesian OptimizationDrug DiscoveryLanguage ModelingLanguage Modelling+3xVal: A Continuous Numerical Tokenization for Scientific Language Models
Due in part to their discontinuous and discrete default encodings for numbers, Large Language Models (LLMs) have not yet been commonly used to process numerically-dense scientific datasets. Rendering datasets as text, ho…
Computational EfficiencyInductive BiasOut-of-Distribution GeneralizationInterleaving Text and Number Embeddings to Solve Mathemathics Problems
Integrating text and numbers effectively is a crucial step towards enhancing Large Language Models (LLMs) capabilities in assisting in scientific tasks. While most current approaches rely on discrete tokenization of numb…
DecoderInductive Bias