CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations
Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task. In contrast, humans have the ability to learn new concepts from language. Here, we explore training zero-shot classifiers for structured data purely from language. For this, we introduce CLUES, a benchmark for Classifier Learning Using natural language ExplanationS, consisting of a range of classification tasks over structured data along with natural language supervision in the form of explanations. CLUES consists of 36 real-world and 144 synthetic classification tasks. It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks. To model the influence of explanations in classifying an example, we develop ExEnt, an entailment-based model that learns classifiers using explanations. ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations. We delineate key challenges for automated learning from explanations, addressing which can lead to progress on CLUES in the future. Code and datasets are available at: https://clues-benchmark.github.io.
Code (0)
등록된 구현이 없습니다.
Tasks
Inductive LearningSimilar Papers 제목 키워드 기반
Training Classifiers with Natural Language Explanations
Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in whi…
Binary ClassificationGeneral ClassificationRelation ExtractionLet the CAT out of the bag: Contrastive Attributed explanations for Text
Contrastive explanations for understanding the behavior of black box models has gained a lot of attention recently as they provide potential for recourse. In this paper, we propose a method Contrastive Attributed explana…
AttributeLanguage ModelingLanguage ModellingDynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering
Recent advances in multimodal large language models (LLMs) have shown extreme effectiveness in visual question answering (VQA). However, the design nature of these end-to-end models prevents them from being interpretable…
Question AnsweringVisual Question AnsweringVisual Question Answering (VQA)Benchmarking Post-Hoc Interpretability Approaches for Transformer-based Misogyny Detection
Transformer-based Natural Language Processing models have become the standard for hate speech detection. However, the unconscious use of these techniques for such a critical task comes with negative consequences. Various…
BenchmarkingHate Speech DetectionDISCERN: Decoding Systematic Errors in Natural Language for Text Classifiers
Despite their high predictive accuracies, current machine learning systems often exhibit systematic biases stemming from annotation artifacts or insufficient support for certain classes in the dataset. Recent work propos…
Active Learningtext-classificationText Classification