PRISM: Prosody-Integrated Multi-Agent Reasoning Framework for Empathetic Spoken Dialogue

2026-06-11 · Wen Zhang, Xiaocui Yang, Zhuoyue Gao, Shi Feng, Daling Wang, Yifei Zhang arxiv

Empathetic spoken dialogue systems require not only semantically appropriate responses but also emotionally aligned prosodic expression. However, cascade pipelines often discard acoustic cues during speech-to-text conversion, while end-to-end speech models lack interpretable control over emotion and knowledge integration. To address these challenges, we propose PRISM, a multi-agent framework for empathetic spoken dialogue that decouples speech perception, response generation, and speech synthesis into coordinated components. PRISM introduces a prosody-to-language translation mechanism to stabilize large language model reasoning and enables on-demand invocation of external knowledge tools for empathetic dialogue generation. Experimental results demonstrate that PRISM achieves consistent improvements in empathy, prosodic appropriateness, and text response generation quality across objective and subjective metrics. Our code is available at: https://github.com/Bxzfrm/PRISM.

📄 PDF Abstract BibTeX arXiv:2606.12902

Code (0)

등록된 구현이 없습니다.

Tasks

Dialogue Generation Response Generation Speech Synthesis

PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodality

2025-08-26 · Nanxi Li, Zhengyue Zhao, G. Edward Suh, Marco Pavone 외 arxiv

Safeguarding vision-language models (VLMs) is a critical challenge, as existing methods often suffer from over-defense, which harms utility, or rely on shallow alignment, failing to detect complex threats that require de…

PRiSM: An Agentic Multimodal Benchmark for Scientific Reasoning via Python-Grounded Evaluation

2025-12-05 · Shima Imani, Seungwhan Moon, Adel Ahmadyan, Lu Zhang 외 arxiv

Evaluating vision-language models (VLMs) in scientific domains like mathematics and physics poses unique challenges that go far beyond predicting final answers. These domains demand conceptual understanding, symbolic rea…

Program Synthesis

PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments

2026-05-12 · Yunn Kang Lim, Pengzhan Sun, Ziyi Bai, Xun Xu 외 arxiv

When an LLM-based embodied agent fails at a household task, the culprit could be misidentified objects, forgotten sub-goals, or poor action sequencing -- yet existing benchmarks report only a single success rate, making …

PrismAgent: Illuminating Harm in Memes via a Zero-Shot Interpretable Multi-Agent Framework

2026-05-01 · Zihan Ding, Ziyuan Yang, Yi Zhang arxiv

The rapid spread of memes makes harmful content detection increasingly crucial, as effective identification can curb the circulation of misinformation. However, existing methods rely heavily on high-volume annotated data…

PRISMA: Reinforcement Learning Guided Two-Stage Policy Optimization in Multi-Agent Architecture for Open-Domain Multi-Hop Question Answering

2026-01-09 · Yu Liu, Wenxiao Zhang, Cong Cao, Wenxuan Lu 외 arxiv

Answering real-world open-domain multi-hop questions over massive corpora is a critical challenge in Retrieval-Augmented Generation (RAG) systems. Recent research employs reinforcement learning (RL) to end-to-end optimiz…

Multi-hop Question Answering Reinforcement Learning