Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction

2023-05-22 · Siyu Yuan, Jiangjie Chen, Xuyang Ge, Yanghua Xiao, Deqing Yang

The vital role of analogical reasoning in human cognition allows us to grasp novel concepts by linking them with familiar ones through shared relational structures. Despite the attention previous research has given to word analogies, this work suggests that Large Language Models (LLMs) often overlook the structures that underpin these analogies, raising questions about the efficacy of word analogies as a measure of analogical reasoning skills akin to human cognition. In response to this, our paper introduces a task of analogical structure abduction, grounded in cognitive psychology, designed to abduce structures that form an analogy between two systems. In support of this task, we establish a benchmark called SCAR, containing 400 scientific analogies from 13 distinct fields, tailored for evaluating analogical reasoning with structure abduction. The empirical evidence underlines the continued challenges faced by LLMs, including ChatGPT and GPT-4, in mastering this task, signifying the need for future exploration to enhance their abilities.

📄 PDF Abstract BibTeX arXiv:2305.12660

Code (1)

siyuyuan/scar 공식 구현

Tasks

Novel ConceptsQuestion Answering

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Connotation Lexicon: A Dash of Sentiment Beneath the Surface Meaning

2013-08-01 · ACL 2013 8 · Song Feng, Jun Seok Kang, Polina Kuznetsova, Yejin Choi

Sentiment Analysis

Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

2026-09-04 · Seogyeong Jeong, Jaehui Hwang, Dongyoon Han, Geonmo Gu 외 arxiv

Reasoning in large language models unfolds through diverse functional operations, such as problem formulation, goal decomposition, and deduction. Although these operations are explicitly distinguished in text, little is …

Capacitive Sensor Based 2D Subsurface Imaging Technology for Non Destructive Evaluation of Building Surfaces

2019-07-22

Understanding the underlying structure of building surfaces like walls and floors is essential when carrying out building maintenance and modification work. To facilitate such work, this paper introduces a capacitive sen…

Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLMs

2024-08-20 · Maxim Ifergan, Leshem Choshen, Roee Aharoni, Idan Szpektor 외

The veracity of a factoid is largely independent of the language it is written in. However, language models are inconsistent in their ability to answer the same factual question across languages. This raises questions ab…

knowledge editing

Going Beneath the Surface: Evaluating Image Captioning for Grammaticality, Truthfulness and Diversity

2019-12-19 · Huiyuan Xie, Tom Sherborne, Alexander Kuhnle, Ann Copestake

Image captioning as a multimodal task has drawn much interest in recent years. However, evaluation for this task remains a challenging problem. Existing evaluation metrics focus on surface similarity between a candidate …

Diagnostic Diversity Image Captioning