Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison

2026-05-28 · Tiancheng Yang, Matthias Schonlau, Ilia Sucholutsky arxiv

Emerging personal AI agents are moving toward persistent, multi-source memory. This creates an evaluation problem: systems must decide how to use conflicting or incomplete evidence; they cannot just retrieve facts from one clean history. Existing benchmarks rarely show whether an error came from the evidence given to a method or from the method's conflict-resolution step. We study this as selective QA over conflicting multi-source personal memory: systems answer based on conflicting, sometimes incomplete sources, or abstain when evidence is insufficient. We develop a benchmark containing 18 question templates across 8 reasoning types, 480 personas, 4 random seeds, and 34,560 instances, with controlled source distortions and deterministic ground truth. We evaluate the performance of baselines without access to any source, access to a single source, structured fusion methods, and frontier LLMs. The best trained fusion resolver reaches 80.3% accuracy, while the strongest prompt-only LLM baseline reaches 70.0%. With abstention, the same resolver reaches 85.3% selective accuracy at 78.3% coverage and the best LLM reaches 71.0% selective accuracy at 95.4% coverage. Different models have different strengths across reasoning types. We release the data, code, cached model outputs, and data-generating process for reuse.

📄 PDF Abstract BibTeX arXiv:2605.30087

Code (0)

등록된 구현이 없습니다.

According to Me: Long-Term Personalized Referential Memory QA

2026-03-02 · Jingbiao Mei, Jinghong Chen, Guangyu Yang, Xinyu Hou 외 arxiv

Personalized AI assistants must recall and reason over long-term user memory, which naturally spans multiple modalities and sources such as images, videos, and emails. However, existing Long-term Memory benchmarks focus …

Answer Generation

FedBiCross: Personalized One-Shot Federated Learning on Medical Images

2026-01-05 · Yuexuan Xia, Yinghao Zhang, Yalin Liu, Hong-Ning Dai 외 arxiv

Data-free knowledge distillation-based one-shot federated learning (OSFL) trains a model in a single communication round without sharing raw data, making OSFL attractive for privacy-sensitive medical applications. Howeve…

Data-free Knowledge Distillation Federated Learning

Can Language Models Follow Multiple Turns of Entangled Instructions?

2025-03-17 · Chi Han

Despite significant achievements in improving the instruction-following capabilities of large language models (LLMs), the ability to process multiple potentially entangled or conflicting instructions remains a considerab…

Instruction Following Memorization

FedEmbed: Personalized Private Federated Learning

2022-02-18 · Andrew Silva, Katherine Metcalf, Nicholas Apostoloff, Barry-John Theobald

Federated learning enables the deployment of machine learning to problems for which centralized data collection is impractical. Adding differential privacy guarantees bounds on privacy while data are contributed to a glo…

Federated Learning

TRACE-Memory: Public-Conditioned Retrieval and Utility-Aware Evidence Admission for Personalized Generation

2026-08-09 · Jing Wang, Zhu Wang, Yifan Guo, Yulong Yang 외 arxiv

Personalized generation systems retrieve user history by request--memory relevance and inject it into the model context. Yet relevant history may concern the wrong preference aspect, duplicate public information, or prov…

Semantic Retrieval