paper-with-me

홈 › Papers

Measuring the practice of shared-decision making (OPTION12): An Investigation into Open-sourced Smaller LLMs (OS-sLLMs) for Better Privacy and Sustainability

2026-07-07 · Tamara Wit, Lifeng Han, Carly Heipon, David Lindevelt, Anne Stiggelbout, Suzan Verberne arxiv

We present LLM4SDM, the first study of open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making (SDM) using the Observer OPTION12 framework. Unlike previous work that relies on large commercial models and the shorter OPTION5 instrument, our study focuses on privacy-preserving locally deployable models and Dutch melanoma consultation transcripts. Using expert-annotated clinical consultations, we evaluate three general-domain and two medical-domain OS-sLLMs during a development-phase pilot study. Results show that general-domain models outperform medical-domain models, which exhibit substantial hallucination and instruction-following failures. Gemma3:12b achieves the strongest agreement with human annotations (Pearson r=0.51, Spearman \r{ho}=0.59). Item-level and qualitative analyses reveal systematic challenges related to temporal discourse reasoning, conversational role attribution, and evidence grounding. We further introduce a Judge-LLM consensus framework designed to support disagreement resolution among multiple models. Our findings suggest that while current OS-sLLMs cannot replace human annotators, they offer a promising foundation for privacy-preserving human-in-the-loop SDM assessment.

📄 PDF Abstract BibTeX arXiv:2607.06127

Code (0)

등록된 구현이 없습니다.

Tasks

Decision Making

Similar Papers 제목 키워드 기반

Interpretable Stochastic Block Influence Model: measuring social influence among homophilous communities

2020-06-01 · Yan Leng, Tara Sowrirajan, Alex Pentland

Decision-making on networks can be explained by both homophily and social influence. While homophily drives the formation of communities with similar characteristics, social influence occurs both within and between commu…

Decision MakingMarketing

AI for Requirements Engineering: Industry adoption and Practitioner perspectives

2025-11-03 · Lekshmi Murali Rani, Richard Berntsson Svensson, Robert Feldt arxiv

The integration of AI for Requirements Engineering (RE) presents significant benefits but also poses real challenges. Although RE is fundamental to software engineering, limited research has examined AI adoption in RE. W…

Decision Making

FairSight: Visual Analytics for Fairness in Decision Making

2019-08-01 · Yongsu Ahn, Yu-Ru Lin

Data-driven decision making related to individuals has become increasingly pervasive, but the issue concerning the potential discrimination has been raised by recent studies. In response, researchers have made efforts to…

Decision MakingDiagnosticFairness

ADAPT: Action-aware Driving Caption Transformer

2023-02-01 · Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li 외

End-to-end autonomous driving has great potential in the transportation industry. However, the lack of transparency and interpretability of the automatic decision-making process hinders its industrial adoption in practic…

Autonomous DrivingDecision Making

Measuring Agreeableness Bias in Multimodal Models

2024-08-17 · JaeHyuk Lim, Bruce W. Lee

This paper examines a phenomenon in multimodal language models where pre-marked options in question images can significantly influence model responses. Our study employs a systematic methodology to investigate this effec…

Decision MakingMultiple-choice