Legal Transformer Models May Not Always Help

2021-09-14 · Saibo Geng, Rémi Lebret, Karl Aberer

Deep learning-based Natural Language Processing methods, especially transformers, have achieved impressive performance in the last few years. Applying those state-of-the-art NLP methods to legal activities to automate or simplify some simple work is of great value. This work investigates the value of domain adaptive pre-training and language adapters in legal NLP tasks. By comparing the performance of language models with domain adaptive pre-training on different tasks and different dataset splits, we show that domain adaptive pre-training is only helpful with low-resource downstream tasks, thus far from being a panacea. We also benchmark the performance of adapters in a typical legal NLP task and show that they can yield similar performance to full model tuning with much smaller training costs. As an additional result, we release LegalRoBERTa, a RoBERTa model further pre-trained on legal corpora.

📄 PDF Abstract BibTeX arXiv:2109.06862

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Weight Decay 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classification

2025-08-11 · Amrita Singh, H. Suhan Karaca, Aditya Joshi, Hye-young Paik 외 arxiv

Despite advances in legal NLP, no comprehensive evaluation of Transformer-based models customized for legal tasks (referred to as `legal-specific' models in this paper) exists for contract classification tasks. To addres…

Analysing similarities between legal court documents using natural language processing approaches based on Transformers

2022-04-14 · Raphael Souza de Oliveira, Erick Giovani Sperandio Nascimento

Recent advances in Artificial Intelligence (AI) have leveraged promising results in solving complex problems in the area of Natural Language Processing (NLP), being an important tool to help in the expeditious resolution…

When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset

2021-04-18 · Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson 외

While self-supervised learning has made rapid advances in natural language processing, it remains unclear when researchers should engage in resource-intensive domain-specific pretraining (domain pretraining). The law, pu…

Multiple-choice Question Answering Self-Supervised Learning Specificity+1

Can Domain Pre-training Help Interdisciplinary Researchers from Data Annotation Poverty? A Case Study of Legal Argument Mining with BERT-based Transformers

2021-12-01 · NLP4DH (ICON) 2021 12 · Gechuan Zhang, David Lillis, Paul Nulty

Interdisciplinary Natural Language Processing (NLP) research traditionally suffers from the requirement for costly data annotation. However, transformer frameworks with pre-training have shown their ability on many downs…

Argument Mining

Privacy-Preserving Models for Legal Natural Language Processing

2022-11-05 · Ying Yin, Ivan Habernal

Pre-training large transformer models with in-domain data improves domain adaptation and helps gain performance on the domain-specific downstream tasks. However, sharing models pre-trained on potentially sensitive data i…

Domain Adaptation Privacy Preserving Self-Supervised Learning