Do Language Embeddings Capture Scales?
Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense, and factual knowledge. One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects. We show that pretrained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning. We identify contextual information in pre-training and numeracy as two key factors affecting their performance and show that a simple method of canonicalizing numbers can have a significant effect on the results.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningSimilar Papers 제목 키워드 기반
Language Through a Prism: A Spectral Approach for Multiscale Language Representations
Language exhibits structure at different scales, ranging from subwords to words, sentences, paragraphs, and documents. To what extent do deep models capture information at these scales, and can we force them to better ca…
Part-Of-Speech TaggingTopic ClassificationDo Location Encoders Capture Spatial Effects? A GeoShapley Benchmark Across Scales
Location encoders transform geographic coordinates into high dimensional embeddings for downstream machine learning, but it is unclear how well these representations capture interpretable spatial effects. We benchmark wh…
Ovis: Structural Embedding Alignment for Multimodal Large Language Model
Current Multimodal Large Language Models (MLLMs) typically integrate a pre-trained LLM with another pre-trained vision transformer through a connector, such as an MLP, endowing the LLM with visual capabilities. However, …
Language ModelingMultimodal Large Language ModelVisual Question Answering (VQA)A Broader Picture of Random-walk Based Graph Embedding
Graph embedding based on random-walks supports effective solutions for many graph-related downstream tasks. However, the abundance of embedding literature has made it increasingly difficult to compare existing methods an…
Graph EmbeddingLink PredictionScale-dependent Relationships in Natural Language
Natural language exhibits statistical dependencies at a wide range of scales. For instance, the mutual information between words in natural language decays like a power law with the temporal lag between them. However, ma…