AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization
Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale.
Code (1)
Tasks
Abstractive Text SummarizationDomain AdaptationIn-Context LearningText SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
AdaptEval: A Benchmark for Evaluating Large Language Models on Code Snippet Adaptation
Recent advancements in large language models (LLMs) have automated various software engineering tasks, with benchmarks emerging to evaluate their capabilities. However, for adaptation, a critical activity during code reu…
Domain Adaptation in MT Using Titles in Wikipedia as a Parallel Corpus: Resources and Evaluation
This paper presents how an state-of-the-art SMT system is enriched by using an extra in-domain parallel corpora extracted from Wikipedia. We collect corpora from parallel titles and from parallel fragments in comparable …
ArticlesDomain AdaptationQuestion Generation for Evaluating Cross-Dataset Shifts in Multi-modal Grounding
Visual question answering (VQA) is the multi-modal task of answering natural language questions about an input image. Through cross-dataset adaptation methods, it is possible to transfer knowledge from a source dataset w…
Question AnsweringQuestion GenerationQuestion-GenerationVisual Question Answering+1Linguistically Informed Masking for Representation Learning in the Patent Domain
Domain-specific contextualized language models have demonstrated substantial effectiveness gains for domain-specific downstream tasks, like similarity matching, entity recognition or information retrieval. However succes…
Domain AdaptationInformation RetrievalRepresentation LearningRetrievalText to Trust: Evaluating Fine-Tuning and LoRA Trade-offs in Language Models for Unfair Terms of Service Detection
Large Language Models (LLMs) have transformed text understanding, yet their adaptation to specialized legal domains remains constrained by the cost of full fine-tuning. This study provides a systematic evaluation of fine…