Reflection-Bench: probing AI intelligence with reflection

2024-10-21 · Lingyu Li, Yixu Wang, Haiquan Zhao, Shuqi Kong, Yan Teng, Chunbo Li, Yingchun Wang

The ability to adapt beliefs or behaviors in response to unexpected outcomes, reflection, is fundamental to intelligent systems' interaction with the world. From a cognitive science perspective, this serves as a core principle of intelligence applicable to both human and AI systems. To address the debate on the intelligence of large language models (LLMs), we propose Reflection-Bench, a comprehensive benchmark comprising 7 tasks spanning core cognitive functions crucial for reflection, including perception, memory, belief updating, decision-making, prediction, counterfactual thinking, and meta-reflection. We evaluate the performances of 13 prominent LLMs such as OpenAI o1, GPT-4, Claude 3.5 Sonnet, etc. The results indicate that current LLMs still lack satisfactory reflection ability. We discuss the underlying causes of these results and suggest potential avenues for future research. In conclusion, Reflection-Bench offers both evaluation tools and inspiration for developing AI capable of reliably interacting with the environment. Our data and code are available at https://github.com/YabYum/ReflectionBench.

📄 PDF Abstract BibTeX arXiv:2410.16270

Code (1)

yabyum/reflectionbench 공식 구현

Tasks

counterfactualDecision Making

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Residual Connection 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

From Emergence to Control: Probing and Modulating Self-Reflection in Language Models

2025-06-13 · XUDONG ZHU, Jiachen Jiang, Mohammad Mahdi Khalili, Zhihui Zhu

Self-reflection -- the ability of a large language model (LLM) to revisit, evaluate, and revise its own reasoning -- has recently emerged as a powerful behavior enabled by reinforcement learning with verifiable rewards (…

Large Language Model Navigate

Intelligent Reflecting Surface-Aided Electromagnetic Stealth over Extended Regions

2025-03-07 · Qingjie Wu, Beixiong Zheng, Guangchi Zhang, Derrick Wing Kwan Ng 외

Compared to traditional electromagnetic stealth (ES) materials, which are effective only within specific frequencies and orientations, intelligent reflecting surface (IRS) technology introduces a novel paradigm for achie…

From Synthesis to Removal: Physics-Grounded Reflection Simulation and Diffusion-Based Video Dereflection

2026-08-12 · Zepeng Wang, Jiagao Hu, Fuhao Li, Yuxuan Chen 외 hf

Videos captured through glass often contain reflections that degrade visual quality and interfere with downstream vision tasks. Although single-image reflection removal has been extensively studied, video reflection remo…

Reflection Removal

Reflection Generation for Composite Image Using Diffusion Model

2026-04-02 · Haonan Zhao, Qingyang Liu, Jiaxuan Chen, Li Niu arxiv

Image composition involves inserting a foreground object into the background while synthesizing environment-consistent effects such as shadows and reflections. Although shadow generation has been extensively studied, ref…

Single Image Reflection Removal with Reflection Intensity Prior Knowledge

2023-12-06 · Dongshen Han, Seungkyu Lee, Chaoning Zhang, Heechan Yoon 외

Single Image Reflection Removal (SIRR) in real-world images is a challenging task due to diverse image degradations occurring on the glass surface during light transmission and reflection. Many existing methods rely on s…