DiffuseAgent-MI: Distributionally-Grounded,Tool-Integrated Self-Evolving Agents for Faithful Visual Reasoning

2026-08-01 · An Lanji, Dawei Liu, Jin Li, Haoran Xu, Mei Chen, Yu Tian arxiv

Tool-integrated vision-language agents have made remarkable progress on compositional and multi-step visual reasoning. Yet their outputs frequently exhibit unfaithfulness: the stated reasoning path diverges from the computation that actually produced the answer, undermining reliability in safety-critical applications. We present DiffuseAgent-MI, a self-evolving agent whose perceptual grounding is governed by a KL-minimal energy model over feature units, providing a distributional view of visual mechanistic interpretability. The agent learns an energy landscape that softly constrains generated samples to lie near the native prior conditioned on the chosen interpretable unit, closing the gap between the explanation and the internal representation. A verifier then supplies trajectory-level faithfulness rewards, and a repair branch re-conditions the energy when the verifier flags an unfaithful step. On GeoQA, SciVis, VQA-v2 and an in-house multimodal reasoning set, DiffuseAgent-MI improves accuracy by up to 5.1 points over prior self-evolving agents while more than doubling mutual-information faithfulness and human-interpretability agreement. Our analysis shows the energy term and the verifier are complementary: the former guarantees distributional faithfulness, the latter trajectory-level faithfulness, and only their combination closes both gaps.

📄 PDF Abstract BibTeX arXiv:2608.00540

Code (0)

등록된 구현이 없습니다.

Tasks

Multimodal Reasoning Visual Reasoning

Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning

2025-11-25 · Jiaqi Liu, Kaiwen Xiong, Peng Xia, Yiyang Zhou 외 arxiv

Vision-language agents have achieved remarkable progress in a variety of multimodal reasoning tasks; however, their learning remains constrained by the limitations of human-annotated supervision. Recent self-rewarding ap…

Reinforcement Learning Multimodal Reasoning Visual Reasoning

Robust Generalization with Adaptive Optimal Transport Priors for Decision-Focused Learning

2026-02-01 · Haixiang Sun, Andrew L. Liu arxiv

Few-shot learning requires models to generalize under limited supervision while remaining robust to distribution shifts. Existing Sinkhorn Distributionally Robust Optimization (DRO) methods provide theoretical guarantees…

Few-Shot Learning

Distributionally Robust Chance-Constrained Flexibility Planning for Integrated Energy System

2021-05-17 · Sen Zhan, Peng Hou, Guangya Yang

Inflexible combined heat and power (CHP) plants and uncertain wind power production result in excess power in distribution networks, which leads to inverse power flow challenging grid operations. Power-to-X facilities su…

Distributionally Robust Classifiers in Sentiment Analysis

2021-10-20 · Shilun Li, Renee Li, Carina Zhang

In this paper, we propose sentiment classification models based on BERT integrated with DRO (Distributionally Robust Classifiers) to improve model performance on datasets with distributional shifts. We added 2-Layer Bi-L…

Sentiment Analysis Sentiment Classification

Distributionally Ambiguous Optimization Techniques for Batch Bayesian Optimization

2017-07-13 · Nikitas Rontsis, Michael A. Osborne, Paul J. Goulart

We propose a novel, theoretically-grounded, acquisition function for Batch Bayesian optimization informed by insights from distributionally ambiguous optimization. Our acquisition function is a lower bound on the well-kn…

Bayesian Optimization