LLM+Reasoning+Planning for supporting incomplete user queries in presence of APIs

2024-05-21 · Sudhir Agarwal, Anu Sreepathy, David H. Alonso, Prarit Lamba

Recent availability of Large Language Models (LLMs) has led to the development of numerous LLM-based approaches aimed at providing natural language interfaces for various end-user tasks. These end-user tasks in turn can typically be accomplished by orchestrating a given set of APIs. In practice, natural language task requests (user queries) are often incomplete, i.e., they may not contain all the information required by the APIs. While LLMs excel at natural language processing (NLP) tasks, they frequently hallucinate on missing information or struggle with orchestrating the APIs. The key idea behind our proposed approach is to leverage logical reasoning and classical AI planning along with an LLM for accurately answering user queries including identification and gathering of any missing information in these queries. Our approach uses an LLM and ASP (Answer Set Programming) solver to translate a user query to a representation in Planning Domain Definition Language (PDDL) via an intermediate representation in ASP. We introduce a special API "get_info_api" for gathering missing information. We model all the APIs as PDDL actions in a way that supports dataflow between the APIs. Our approach then uses a classical AI planner to generate an orchestration of API calls (including calls to get_info_api) to answer the user query. Our evaluation results show that our approach significantly outperforms a pure LLM based approach by achieving over 95\% success rate in most cases on a dataset containing complete and incomplete single goal and multi-goal queries where the multi-goal queries may or may not require dataflow among the APIs.

📄 PDF Abstract BibTeX arXiv:2405.12433

Code (0)

등록된 구현이 없습니다.

Tasks

Logical Reasoning

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Faithful Embeddings for Knowledge Base Queries

2020-04-07 · NeurIPS 2020 12 · Haitian Sun, Andrew O. Arnold, Tania Bedrax-Weiss, Fernando Pereira 외

The deductive closure of an ideal knowledge base (KB) contains exactly the logical queries that the KB can answer. However, in practice KBs are both incomplete and over-specified, failing to answer some queries that have…

Question Answering

Towards Building General Purpose Embedding Models for Industry 4.0 Agents

2025-06-14 · Christodoulos Constantinides, Shuxin Lin, Dhaval Patel

In this work we focus on improving language models' understanding for asset maintenance to guide the engineer's decisions and minimize asset downtime. Given a set of tasks expressed in natural language for Industry 4.0 d…

GR-Agent: Adaptive Graph Reasoning Agent under Incomplete Knowledge

2025-12-16 · Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou 외 arxiv

Large language models (LLMs) achieve strong results on knowledge graph question answering (KGQA), but most benchmarks assume complete knowledge graphs (KGs) where direct supporting triples exist. This reduces evaluation …

Graph Question Answering Knowledge Graphs

NeSy-RAG: Neuro-Symbolic RAG for Explainable Question Answering

2026-08-06 · Jonas Gann, Michael Gertz arxiv

Retrieval-augmented generation (RAG) improves question answering by grounding large language models (LLMs) in external knowledge such as text corpora. However, its reasoning process remains largely opaque: intermediate r…