Towards leveraging LLMs for Conditional QA
This study delves into the capabilities and limitations of Large Language Models (LLMs) in the challenging domain of conditional question-answering. Utilizing the Conditional Question Answering (CQA) dataset and focusing on generative models like T5 and UL2, we assess the performance of LLMs across diverse question types. Our findings reveal that fine-tuned LLMs can surpass the state-of-the-art (SOTA) performance in some cases, even without fully encoding all input context, with an increase of 7-8 points in Exact Match (EM) and F1 scores for Yes/No questions. However, these models encounter challenges in extractive question answering, where they lag behind the SOTA by over 10 points, and in mitigating the risk of injecting false information. A study with oracle-retrievers emphasizes the critical role of effective evidence retrieval, underscoring the necessity for advanced solutions in this area. Furthermore, we highlight the significant influence of evaluation metrics on performance assessments and advocate for a more comprehensive evaluation framework. The complexity of the task, the observed performance discrepancies, and the need for effective evidence retrieval underline the ongoing challenges in this field and underscore the need for future work focusing on refining training tasks and exploring prompt-based techniques to enhance LLM performance in conditional question-answering tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Extractive Question-AnsweringQuestion AnsweringRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods
The rapid scaling of large language models (LLMs) has raised concerns about the transparency and fair use of the pretraining data used for training them. Detecting such content is challenging due to the scale of the data…
Inference AttackLanguage ModelingLanguage ModellingMembership Inference AttackDiffusion LLMs are Natural Adversaries for any LLM
We introduce a novel framework that transforms the resource-intensive (adversarial) prompt optimization problem into an \emph{efficient, amortized inference task}. Our core insight is that pretrained, non-autoregressive …
Red TeamingFollowing the Autoregressive Nature of LLM Embeddings via Compression and Alignment
A new trend uses LLMs as dense text encoders via contrastive learning. However, since LLM embeddings predict the probability distribution of the next token, they are inherently generative and distributive, conflicting wi…
Contrastive LearningLeveraging LLMs to Create a Haptic Devices' Recommendation System
Haptic technology has seen significant growth, yet a lack of awareness of existing haptic device design knowledge hinders development. This paper addresses these limitations by leveraging advancements in Large Language M…
BenchmarkingLayoutCoT: Unleashing the Deep Reasoning Potential of Large Language Models for Layout Generation
Conditional layout generation aims to automatically generate visually appealing and semantically coherent layouts from user-defined constraints. While recent methods based on generative models have shown promising result…
In-Context LearningLayout GenerationRAGRetrieval+1