Large language models enabled multiagent ensemble method for efficient EHR data labeling
This study introduces a novel multiagent ensemble method powered by LLMs to address a key challenge in ML - data labeling, particularly in large-scale EHR datasets. Manual labeling of such datasets requires domain expertise and is labor-intensive, time-consuming, expensive, and error-prone. To overcome this bottleneck, we developed an ensemble LLMs method and demonstrated its effectiveness in two real-world tasks: (1) labeling a large-scale unlabeled ECG dataset in MIMIC-IV; (2) identifying social determinants of health (SDOH) from the clinical notes of EHR. Trading off benefits and cost, we selected a pool of diverse open source LLMs with satisfactory performance. We treat each LLM's prediction as a vote and apply a mechanism of majority voting with minimal winning threshold for ensemble. We implemented an ensemble LLMs application for EHR data labeling tasks. By using the ensemble LLMs and natural language processing, we labeled MIMIC-IV ECG dataset of 623,566 ECG reports with an estimated accuracy of 98.2%. We applied the ensemble LLMs method to identify SDOH from social history sections of 1,405 EHR clinical notes, also achieving competitive performance. Our experiments show that the ensemble LLMs can outperform individual LLM even the best commercial one, and the method reduces hallucination errors. From the research, we found that (1) the ensemble LLMs method significantly reduces the time and effort required for labeling large-scale EHR data, automating the process with high accuracy and quality; (2) the method generalizes well to other text data labeling tasks, as shown by its application to SDOH identification; (3) the ensemble of a group of diverse LLMs can outperform or match the performance of the best individual LLM; and (4) the ensemble method substantially reduces hallucination errors. This approach provides a scalable and efficient solution to data-labeling challenges.
Code (0)
등록된 구현이 없습니다.
Tasks
HallucinationSimilar Papers 제목 키워드 기반
Dual Ensembled Multiagent Q-Learning with Hypernet Regularizer
Overestimation in single-agent reinforcement learning has been extensively studied. In contrast, overestimation in the multiagent setting has received comparatively little attention although it increases with the number …
Q-LearningSMACSMAC+Real-World Deployment of a Hierarchical Uncertainty-Aware Collaborative Multiagent Planning System
We would like to enable a collaborative multiagent team to navigate at long length scales and under uncertainty in real-world environments. In practice, planning complexity scales with the number of agents in the team, w…
NavigateImplicit Multiagent Coordination at Unsignalized Intersections via Multimodal Inference Enabled by Topological Braids
We focus on navigation among rational, non-communicating agents at unsignalized street intersections. Following collision-free motion under such settings demands nuanced implicit coordination among agents. Often, the str…
Decision MakingTrajectory PredictionTowards Safe Control of Continuum Manipulator Using Shielded Multiagent Reinforcement Learning
Continuum robotic manipulators are increasingly adopted in minimal invasive surgery. However, their nonlinear behavior is challenging to model accurately, especially when subject to external interaction, potentially lead…
reinforcement-learningReinforcement Learning (RL)Continuification control of large-scale multiagent systems under limited sensing and structural perturbations
We investigate the stability and robustness properties of a continuification-based strategy for the control of large-scale multiagent systems. Within continuation-based strategy, one transforms the microscopic, agent-lev…