Rethinking STS and NLI in Large Language Models
Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. However, when applied to semantic textual similarity (STS) and natural language inference (NLI), the effectiveness of LLMs turns out to be limited by low-resource domain accuracy, model overconfidence, and difficulty to capture the disagreements between human judgements. With this in mind, here we try to rethink STS and NLI in the era of LLMs. We first evaluate the performance of STS and NLI in the clinical/biomedical domain, and then we assess LLMs' predictive confidence and their capability of capturing collective human opinions. We find that these old problems are still to be properly addressed in the era of LLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceSemantic Textual SimilaritySTSSimilar Papers 제목 키워드 기반
Rethinking with Retrieval: Faithful Large Language Model Inference
Despite the success of large language models (LLMs) in various natural language processing (NLP) tasks, the stored knowledge in these models may inevitably be incomplete, out-of-date, or incorrect. This motivates the nee…
Language ModelingLanguage ModellingLarge Language Modelmodel+2Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization
This work suggests fundamentally rethinking the current practice of pruning large language models (LLMs). The way it is done is by divide and conquer: split the model into submodels, sequentially prune them, and reconstr…
One-shot Entropy Minimization
We trained 13,440 large language models and found that entropy minimization requires only a single unlabeled data and 10 steps optimization to achieve performance improvements comparable to or even greater than those obt…
reinforcement-learningReinforcement LearningImproving Low-resource Reading Comprehension via Cross-lingual Transposition Rethinking
Extractive Reading Comprehension (ERC) has made tremendous advances enabled by the availability of large-scale high-quality ERC training data. Despite of such rapid progress and widespread application, the datasets in la…
Reading ComprehensionGRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism
Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open ab…
Mixture-of-Experts