On Linearizing Structured Data in Encoder-Decoder Language Models: Insights from Text-to-SQL
Structured data, prevalent in tables, databases, and knowledge graphs, poses a significant challenge in its representation. With the advent of large language models (LLMs), there has been a shift towards linearization-based methods, which process structured data as sequential token streams, diverging from approaches that explicitly model structure, often as a graph. Crucially, there remains a gap in our understanding of how these linearization-based methods handle structured data, which is inherently non-linear. This work investigates the linear handling of structured data in encoder-decoder language models, specifically T5. Our findings reveal the model's ability to mimic human-designed processes such as schema linking and syntax prediction, indicating a deep, meaningful learning of structure beyond simple token sequencing. We also uncover insights into the model's internal mechanisms, including the ego-centric nature of structure node encodings and the potential for model compression due to modality fusion redundancy. Overall, this work sheds light on the inner workings of linearization-based methods and could potentially provide guidance for future research.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderKnowledge GraphsModel CompressionText to SQLText-To-SQLMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models
Structured pruning methods have proven effective in reducing the model size and accelerating inference speed in various network architectures such as Transformers. Despite the versatility of encoder-decoder models in num…
DecoderDeep Learning Models for Global Coordinate Transformations that Linearize PDEs
We develop a deep autoencoder architecture that can be used to find a coordinate transformation which turns a nonlinear PDE into a linear PDE. Our architecture is motivated by the linearizing transformations provided by …
DecoderDeep LearningDomain-Adaptive Small Language Models for Structured Tax Code Prediction
Every day, multinational firms process thousands of transactions, each of which must adhere to tax regulations that vary by jurisdiction and are often nuanced. The determination of product and service tax codes, such as …
DecoderSmall Language ModelFeedback Linearization for Unknown Systems via Reinforcement Learning
We present a novel approach to control design for nonlinear systems which leverages model-free policy optimization techniques to learn a linearizing controller for a physical plant with unknown dynamics. Feedback lineari…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)SCENIC: Semantic-Conditioned Edge-Aware Neural Framework for Structured IoT Command Generation
Edge Internet of Things (IoT) agents are often constrained by memory capacity, privacy requirements, communication latency, and recurring inference cost. Current smart-home assistants commonly rely on API-level command i…