Large Scale Legal Text Classification Using Transformer Models
Large multi-label text classification is a challenging Natural Language Processing (NLP) problem that is concerned with text classification for datasets with thousands of labels. We tackle this problem in the legal domain, where datasets, such as JRC-Acquis and EURLEX57K labeled with the EuroVoc vocabulary were created within the legal information systems of the European Union. The EuroVoc taxonomy includes around 7000 concepts. In this work, we study the performance of various recent transformer-based models in combination with strategies such as generative pretraining, gradual unfreezing and discriminative learning rates in order to reach competitive classification performance, and present new state-of-the-art results of 0.661 (F1) for JRC-Acquis and 0.754 for EURLEX57K. Furthermore, we quantify the impact of individual steps, such as language model fine-tuning or gradual unfreezing in an ablation study, and provide reference dataset splits created with an iterative stratification algorithm.
Code (1)
Tasks
ClassificationGeneral ClassificationLanguage ModelingLanguage ModellingMulti Label Text ClassificationMulti-Label Text Classificationtext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classification
Despite advances in legal NLP, no comprehensive evaluation of Transformer-based models customized for legal tasks (referred to as `legal-specific' models in this paper) exists for contract classification tasks. To addres…
Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrieval
The rapid growth of statutory corpora and judicial decisions requires scalable legal AI systems capable of classification and retrieval over extremely long contexts. Transformer-based architectures (e.g., Longformer, DeB…
uOttawa at LegalLens-2024: Transformer-based Classification Experiments
This paper presents the methods used for LegalLens-2024 shared task, which focused on detecting legal violations within unstructured textual data and associating these violations with potentially affected individuals. Th…
Classificationnamed-entity-recognitionNamed Entity RecognitionNatural Language Inference+1LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-Training
Large Transformer-based language models such as BERT have led to broad performance improvements on many NLP tasks. Domain-specific variants of these models have demonstrated excellent performance on a variety of speciali…
Language ModelingLanguage ModellingMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATION+3LegalPro-BERT: Classification of Legal Provisions by fine-tuning BERT Large Language Model
A contract is a type of legal document commonly used in organizations. Contract review is an integral and repetitive process to avoid business risk and liability. Contract analysis requires the identification and classif…
Language Modeling