Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game

2024-06-16 · Prisha Samadarshi, Mariam Mustafa, Anushka Kulkarni, Raven Rothkopf, Tuhin Chakrabarty, Smaranda Muresan

The New York Times Connections game has emerged as a popular and challenging pursuit for word puzzle enthusiasts. We collect 438 Connections games to evaluate the performance of state-of-the-art large language models (LLMs) against expert and novice human players. Our results show that even the best performing LLM, Claude 3.5 Sonnet, which has otherwise shown impressive reasoning abilities on a wide variety of benchmarks, can only fully solve 18% of the games. Novice and expert players perform better than Claude 3.5 Sonnet, with expert human players significantly outperforming it. We create a taxonomy of the knowledge types required to successfully cluster and categorize words in the Connections game. We find that while LLMs perform relatively well on categorizing words based on semantic relations they struggle with other types of knowledge such as Encyclopedic Knowledge, Multiword Expressions or knowledge that combines both Word Form and Meaning. Our results establish the New York Times Connections game as a challenging benchmark for evaluating abstract reasoning capabilities in AI systems.

📄 PDF Abstract BibTeX arXiv:2406.11012

Code (1)

mustafamariam/llm-connections-solver 공식 구현

Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue

2026-05-31 · Jingjie Lin, Bingbing Wang, Zihan Wang, Zhengda Jin 외 arxiv

Despite substantial progress in long-context modeling, existing benchmarks remain confined to factual memory for explicit recall, failing to measure the reflective memory required to synthesize fragmented, multimodal cue…

Explore Brain-Inspired Machine Intelligence for Connecting Dots on Graphs Through Holographic Blueprint of Oscillatory Synchronization

2026-01-20 · Tingting Dan, Jiaqi Ding, Guorong Wu arxiv

Neural coupling in both neuroscience and artificial intelligence emerges as dynamic oscillatory patterns that encode abstract concepts. To this end, we hypothesize that a deeper understanding of the neural mechanisms gov…

CodaRAG: Connecting the Dots with Associativity Inspired by Complementary Learning

2026-04-12 · Cheng-Yen Li, Xuanjun Chen, Claire Lin, Wei-Yu Chen 외 arxiv

Large Language Models (LLMs) struggle with knowledge-intensive tasks due to hallucinations and fragmented reasoning over dispersed information. While Retrieval-Augmented Generation (RAG) grounds generation in external so…

Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning

2025-11-24 · Liqin Luo, Guangyao Chen, Xiawu Zheng, Yongxing Dai 외 arxiv

Visual grounding, the task of linking textual queries to specific regions within images, plays a pivotal role in vision-language integration. Existing methods typically rely on extensive task-specific annotations and fin…

Visual Grounding

Large Language Models Are Not Strong Abstract Reasoners

2023-05-31 · Gaël Gendron, Qiming Bao, Michael Witbrock, Gillian Dobbie

Large Language Models have shown tremendous performance on a large variety of natural language processing tasks, ranging from text comprehension to common sense reasoning. However, the mechanisms responsible for this suc…

Common Sense Reasoning Memorization Reading Comprehension