On Monotonic Aggregation for Open-domain QA
Question answering (QA) is a critical task for speech-based retrieval from knowledge sources, by sifting only the answers without requiring to read supporting documents. Specifically, open-domain QA aims to answer user questions on unrestricted knowledge sources. Ideally, adding a source should not decrease the accuracy, but we find this property (denoted as "monotonicity") does not hold for current state-of-the-art methods. We identify the cause, and based on that we propose Judge-Specialist framework. Our framework consists of (1) specialist retrievers/readers to cover individual sources, and (2) judge, a dedicated language model to select the final answer. Our experiments show that our framework not only ensures monotonicity, but also outperforms state-of-the-art multi-source QA methods on Natural Questions. Additionally, we show that our models robustly preserve the monotonicity against noise from speech recognition. We publicly release our code and setting.
Code (1)
Tasks
Language ModelingLanguage ModellingNatural QuestionsQuestion AnsweringRetrievalspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Dangerous Liaisons of Convex Learning and Non-Affine Aggregation
Last-iterate convergence and generalization guarantees in first-order convex learning hinge on the monotonicity of the update operator. While linear averaging preserves the monotonicity of gradient updates, this property…
Weak comonotonicity
The classical notion of comonotonicity has played a pivotal role when solving diverse problems in economics, finance, and insurance. In various practical problems, however, this notion of extreme positive dependence stru…
Weakly monotone averaging functions
Monotonicity with respect to all arguments is fundamental to the definition of aggregation functions. It is also a limiting property that results in many important non-monotonic averaging functions being excluded from th…
Multi-Teacher Ensemble Distillation: A Mathematical Framework for Probability-Domain Knowledge Aggregation
Building on the probability-domain distillation framework of Sparse-KD, we develop an axiomatic, operator-theoretic framework for multi-teacher ensemble knowledge distillation. Rather than prescribing a specific aggregat…
Knowledge DistillationOn the Correspondence Between Monotonic Max-Sum GNNs and Datalog
Although there has been significant interest in applying machine learning techniques to structured data, the expressivity (i.e., a description of what can be learned) of such techniques is still poorly understood. In thi…