Regex-augmented Domain Transfer Topic Classification based on a Pre-trained Language Model: An application in Financial Domain
A common way to use large pre-trained language models for downstream tasks is to fine tune them using additional layers. This may not work well if downstream domain is a specialized domain whereas the large language model has been pre-trained on a generic corpus. In this paper, we discuss the use of regular expression patterns employed as features for domain knowledge during the process of fine tuning, in addition to domain specific text. Our experiments on real scenario production data show that this method of fine tuning improves the downstream text classification tasks as compared to fine tuning only on domain specific text. We also show that the use of attention network for fine tuning improves results compared to simple linear layers.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLarge Language Modeltext-classificationText ClassificationTopic ClassificationSimilar Papers 제목 키워드 기반
Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs
The analysis of public affairs documents is crucial for citizens as it promotes transparency, accountability, and informed decision-making. It allows citizens to understand government policies, participate in public disc…
Decision MakingTopic ClassificationCooperative Semi-Supervised Transfer Learning of Machine Reading Comprehension
Pretrained language models have significantly improved the performance of down-stream language understanding tasks, including extractive question answering, by providing high-quality contextualized word embeddings. Howev…
Extractive Question-AnsweringMachine Reading ComprehensionQuestion AnsweringQuestion Generation+4Towards Faithful Explanations for Text Classification with Robustness Improvement and Explanation Guided Training
Feature attribution methods highlight the important input tokens as explanations to model predictions, which have been widely applied to deep neural networks towards trustworthy AI. However, recent works show that explan…
text-classificationText ClassificationSplitRegex: Faster Regex Synthesis via Neural Example Splitting
Due to the practical importance of regular expressions (regexes, for short), there has been a lot of research to automatically generate regexes from positive and negative string examples. A basic idea of learning a regex…
Neuro-Symbolic Regex Synthesis Framework via Neural Example Splitting
Due to the practical importance of regular expressions (regexes, for short), there has been a lot of research to automatically generate regexes from positive and negative string examples. We tackle the problem of learnin…