Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
The ability of semantic reasoning over the sentence pair is essential for many natural language understanding tasks, e.g., natural language inference and machine reading comprehension. A recent significant improvement in these tasks comes from BERT. As reported, the next sentence prediction (NSP) in BERT, which learns the contextual relationship between two sentences, is of great significance for downstream problems with sentence-pair input. Despite the effectiveness of NSP, we suggest that NSP still lacks the essential signal to distinguish between entailment and shallow correlation. To remedy this, we propose to augment the NSP task to a 3-class categorization task, which includes a category for previous sentence prediction (PSP). The involvement of PSP encourages the model to focus on the informative semantics to determine the sentence order, thereby improves the ability of semantic understanding. This simple modification yields remarkable improvement against vanilla BERT. To further incorporate the document-level information, the scope of NSP and PSP is expanded into a broader range, i.e., NSP and PSP also include close but nonsuccessive sentences, the noise of which is mitigated by the label-smoothing technique. Both qualitative and quantitative experimental results demonstrate the effectiveness of the proposed method. Our method consistently improves the performance on the NLI and MRC benchmarks, including the challenging HANS dataset \cite{hans}, suggesting that the document-level task is still promising for the pre-training.
Code (1)
Tasks
Machine Reading ComprehensionNatural Language InferenceNatural Language UnderstandingReading ComprehensionSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Spectral Analysis of Symmetric and Anti-Symmetric Pairwise Kernels
We consider the problem of learning regression functions from pairwise data when there exists prior knowledge that the relation to be learned is symmetric or anti-symmetric. Such prior knowledge is commonly enforced by s…
regressionExplicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks
In English semantic similarity tasks, classic word embedding-based approaches explicitly model pairwise "interactions" between the word representations of a sentence pair. Transformer-based pretrained language models dis…
Semantic SimilaritySemantic Textual SimilaritySentenceDual Asymmetric Deep Hashing Learning
Due to the impressive learning power, deep learning has achieved a remarkable performance in supervised hash function learning. In this paper, we propose a novel asymmetric supervised deep hashing method to preserve the …
Deep HashingRetrievalAsyInst: Asymmetric Affinity with DepthGrad and Color for Box-Supervised Instance Segmentation
The weakly supervised instance segmentation is a challenging task. The existing methods typically use bounding boxes as supervision and optimize the network with a regularization loss term such as pairwise color affinity…
Box-supervised Instance SegmentationInstance SegmentationSegmentationSemantic Segmentation+1Cross-Modal Learning via Pairwise Constraints
In multimedia applications, the text and image components in a web document form a pairwise constraint that potentially indicates the same semantic concept. This paper studies cross-modal learning via the pairwise constr…
ClusteringRetrieval