PSYCHE: A Multi-faceted Patient Simulation Framework for Evaluation of Psychiatric Assessment Conversational Agents
Recent advances in large language models (LLMs) have accelerated the development of conversational agents capable of generating human-like responses. Since psychiatric assessments typically involve complex conversational interactions between psychiatrists and patients, there is growing interest in developing LLM-based psychiatric assessment conversational agents (PACAs) that aim to simulate the role of psychiatrists in clinical evaluations. However, standardized methods for benchmarking the clinical appropriateness of PACAs' interaction with patients still remain underexplored. Here, we propose PSYCHE, a novel framework designed to enable the 1) clinically relevant, 2) ethically safe, 3) cost-efficient, and 4) quantitative evaluation of PACAs. This is achieved by simulating psychiatric patients based on a multi-faceted psychiatric construct that defines the simulated patients' profiles, histories, and behaviors, which PACAs are expected to assess. We validate the effectiveness of PSYCHE through a study with 10 board-certified psychiatrists, supported by an in-depth analysis of the simulated patient utterances.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingSimilar Papers 제목 키워드 기반
PsychePass: Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments
While large language models show promise in mental healthcare, evaluating their therapeutic competence remains challenging due to the unstructured and longitudinal nature of counseling. We argue that current evaluation p…
Reinforcement LearningNeuroplasticity and Psychedelics: a comprehensive examination of classic and non-classic compounds in pre and clinical models
Neuroplasticity, the ability of the nervous system to adapt throughout an organism's lifespan, offers potential as both a biomarker and treatment target for neuropsychiatric conditions. Psychedelics, a burgeoning categor…
The serotonergic psychedelic N,N-dipropyltryptamine alters information-processing dynamics in cortical neural circuits
Most of the recent work in psychedelic neuroscience has been done using non-invasive neuroimaging, with data recorded from the brains of adult volunteers under the influence of a variety of drugs. While this data provide…
How to set up a psychedelic study: Unique considerations for research involving human participants
Setting up a psychedelic study can be a long, arduous, and kafkaesque process. This rapidly-developing field poses several unique challenges for researchers, necessitating a range of considerations that have not yet been…
AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding
Evaluating large language models (LLMs) has recently emerged as a critical issue for safe and trustworthy application of LLMs in the medical domain. Although a variety of static medical question-answering (QA) benchmarks…