Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the `next sentence prediction (classification)' heuristic can be derived in a principled way from our mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables. The proposed approach not only is effective at increasing the mutual information of segments under the learned model but more importantly, leads to a higher likelihood on holdout data, and improved generation quality. Code is released at https://github.com/BorealisAI/BMI.
Code (1)
Tasks
General ClassificationInductive BiasLanguage ModellingMutual Information EstimationSentenceSimilar Papers 제목 키워드 기반
Rethinking the long-range dependency in Mamba/SSM and transformer models
Long-range dependency is one of the most desired properties of recent sequence models such as state-space models (particularly Mamba) and transformer models. New model architectures are being actively developed and bench…
Joint Bootstrapping Machines for High Confidence Relation Extraction
Semi-supervised bootstrapping techniques for relationship extraction from text iteratively expand a set of initial seed instances. Due to the lack of labeled data, a key challenge in bootstrapping is semantic drift: if a…
RelationRelation ExtractionRelationship Extraction (Distant Supervised)Vocal Bursts Intensity PredictionReduced Spatial Dependency for More General Video-level Deepfake Detection
As one of the prominent AI-generated content, Deepfake has raised significant safety concerns. Although it has been demonstrated that temporal consistency cues offer better generalization capability, existing methods bas…
DeepFake DetectionFace SwappingArabic Tweets Treebanking and Parsing: A Bootstrapping Approach
In this paper, we propose using a {``}bootstrapping{''} method for constructing a dependency treebank of Arabic tweets. This method uses a rule-based parser to create a small treebank of one thousand Arabic tweets and a …
Domain AdaptationA Mutual Bootstrapping Model for Automated Skin Lesion Segmentation and Classification
Automated skin lesion segmentation and classification are two most essential and related tasks in the computer-aided diagnosis of skin cancer. Despite their prevalence, deep learning models are usually designed for only …
ClassificationGeneral ClassificationLesion ClassificationLesion Segmentation+1