Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents

2025-03-11 · Rui Xu, Mingyu Wang, Xintao Wang, Dakuan Lu, Xiaoyu Tan, Wei Chu, Yinghui Xu

Recent advances in LLM-based role-playing language agents (RPLAs) have attracted broad attention in various applications. While chain-of-thought reasoning has shown importance in many tasks for LLMs, the internal thinking processes of RPLAs remain unexplored. Understanding characters' inner thoughts is crucial for developing advanced RPLAs. In this paper, we introduce ROLETHINK, a novel benchmark constructed from literature for evaluating character thought generation. We propose the task of inner thought reasoning, which includes two sets: the gold set that compares generated thoughts with original character monologues, and the silver set that uses expert synthesized character analyses as references. To address this challenge, we propose MIRROR, a chain-of-thought approach that generates character thoughts by retrieving memories, predicting character reactions, and synthesizing motivations. Through extensive experiments, we demonstrate the importance of inner thought reasoning for RPLAs, and MIRROR consistently outperforms existing methods. Resources are available at https://github.com/airaer1998/RPA_Thought.

📄 PDF Abstract BibTeX arXiv:2503.08193

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Do LLMs Really Need 10+ Thoughts for "Find the Time 1000 Days Later"? Towards Structural Understanding of LLM Overthinking

2025-10-09 · Xinliang Frederick Zhang, Anhad Mohananey, Alexandra Chronopoulou, Pinelopi Papalampidi 외 arxiv

Models employing long chain-of-thought (CoT) reasoning have shown superior performance on complex reasoning tasks. Yet, this capability introduces a critical and often overlooked inefficiency -- overthinking -- models of…

Reinforcement Learning for Chain of Thought Compression with One-Domain-to-All Generalization

2025-12-19 · Hanyu Li, Jiangshan Duo, Bofei Gao, Hailin Zhang 외 arxiv

Chain-of-thought reasoning in large language models can trigger an "overthinking trap": longer rollouts raise cost and latency yet often yield unreliable accuracy gains. Existing methods use global, static controls that …

Reinforcement Learning Instruction Following

Do Thought Streams Matter? Evaluating Reasoning in Gemini Vision-Language Models for Video Scene Understanding

2026-04-13 · Shivam Sharma, Sankalp Nagaonkar, Ashish Choithani, Ashutosh Trivedi arxiv

We benchmark how internal reasoning traces, which we call thought streams, affect video scene understanding in vision-language models. Using four configurations of Google's Gemini 2.5 Flash and Flash Lite across scenes e…

Scene Understanding

Proactive Conversational Agents with Inner Thoughts

2024-12-31 · Xingyu Bruce Liu, Shitao Fang, Weiyan Shi, Chien-Sheng Wu 외

One of the long-standing aspirations in conversational AI is to allow them to autonomously take initiatives in conversations, i.e., being proactive. This is especially challenging for multi-party conversations. Prior NLP…

Chatbot

StyleBench: Evaluating thinking styles in Large Language Models

2025-09-25 · Junyu Guo, Shangding Gu, Ming Jin, Costas Spanos 외 arxiv

Structured reasoning can improve the inference performance of large language models (LLMs), but it also introduces computational cost and control constraints. When additional reasoning structure helps, and when it instea…