Interleaving Text and Number Embeddings to Solve Mathemathics Problems
Integrating text and numbers effectively is a crucial step towards enhancing Large Language Models (LLMs) capabilities in assisting in scientific tasks. While most current approaches rely on discrete tokenization of numbers, for instance, conversion to scientific notation or base 10-decomposition, a recent approach proposed a continuous numerical encoding as an inductive bias. In this paper, we build upon this approach by introducing more expressive numerical embeddings. Our method addresses key shortcomings, including the elimination of numerical artefacts and the ability to handle a wide range of magnitudes without clipping. Our work presents two key contributions. First, we employ an MLP to assign distinct directions in the embedding space to different numbers. Our second contribution is the introduction of a routing layer that differentiates between numerical and text embeddings. We hypothesise that this combined approach enables the model to distinguish between text and number distributions while maintaining its capacity for arithmetic operations. Using only a 45 M parameter encoder-decoder architecture our method achieves a $R^2$=0.9988 over a wide range of magnitude ($10^{-3},10^{8}$). In addition, we empirically observe a reduction of the numerical artefacts and biases observed compared to the baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderInductive BiasMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ABC: Achieving Better Control of Multimodal Embeddings using VLMs
Visual embedding models excel at zero-shot tasks like visual retrieval and classification. However, these models cannot be used for tasks that contain ambiguity or require user instruction. These tasks necessitate a mult…
Image to textImage-to-Text RetrievalRetrievalText Retrieval+1Interleaving Learning, with Application to Neural Architecture Search
Interleaving learning is a human learning technique where a learner interleaves the studies of multiple topics, which increases long-term retention and improves ability to transfer learned knowledge. Inspired by the inte…
BIG-bench Machine Learningimage-classificationImage ClassificationNeural Architecture SearchSpiralThinker: Latent Reasoning through an Iterative Process with Text-Latent Interleaving
Recent advances in large reasoning models have been driven by reinforcement learning and test-time scaling, accompanied by growing interest in latent rather than purely textual reasoning. However, existing latent reasoni…
Reinforcement LearningWVEmbs with its Masking: A Method For Radar Signal Sorting
Our study proposes a novel embedding method, Wide-Value-Embeddings (WVEmbs), for processing Pulse Descriptor Words (PDWs) as normalized inputs to neural networks. This method adapts to the distribution of interleaved rad…
Adapting Interleaved Encoders with PPO for Language-Guided Reinforcement Learning in BabyAI
Deep reinforcement learning agents often struggle when tasks require understanding both vision and language. Conventional architectures typically isolate perception (for example, CNN-based visual encoders) from decision-…
Reinforcement Learning