Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling
Given the claims of improved text generation quality across various pre-trained neural models, we consider the coherence evaluation of machine generated text to be one of the principal applications of coherence models that needs to be investigated. Prior work in neural coherence modeling has primarily focused on devising new architectures for solving the permuted document task. We instead use a basic model architecture and show significant improvements over state of the art within the same training regime. We then design a harder self-supervision objective by increasing the ratio of negative samples within a contrastive learning setup, and enhance the model further through automatic hard negative mining coupled with a large global negative queue encoded by a momentum encoder. We show empirically that increasing the density of negative samples improves the basic model, and using a global negative queue further improves and stabilizes the model while training with hard negative samples. We evaluate the coherence model on task-independent test sets that resemble real-world applications and show significant improvements in coherence evaluations of downstream tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Coherence EvaluationContrastive LearningText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Catching the Imposter: Self-Supervised Learning of Physical Coherence with Cross-Entity Feature Permutations
Scientific data often describe entities whose features are jointly governed by the laws of physics, yet existing self-supervised learning (SSL) objectives largely ignore this physical coherence. We introduce imposter, a …
Self-Supervised LearningLaxMotion: Rethinking Supervision Granularity for 3D Human Motion Generation
Recent 3D human motion generation models demonstrate remarkable reconstruction accuracy yet struggle to generalize beyond training distributions. This limitation arises partly from the use of precise 3D supervision, whic…
MonoSelfRecon: Purely Self-Supervised Explicit Generalizable 3D Reconstruction of Indoor Scenes from Monocular RGB Views
Current monocular 3D scene reconstruction (3DR) works are either fully-supervised, or not generalizable, or implicit in 3D representation. We propose a novel framework - MonoSelfRecon that for the first time achieves exp…
3D Reconstruction3D Scene ReconstructionDepth EstimationNeRFRethinking Deep Clustering Paradigms: Self-Supervision Is All You Need
The recent advances in deep clustering have been made possible by significant progress in self-supervised and pseudo-supervised learning. However, the trade-off between self-supervision and pseudo-supervision can give ri…
AllClusteringDeep ClusteringSkip-Clip: Self-Supervised Spatiotemporal Representation Learning by Future Clip Order Ranking
Deep neural networks require collecting and annotating large amounts of data to train successfully. In order to alleviate the annotation bottleneck, we propose a novel self-supervised representation learning approach for…
Action RecognitionFuture predictionRepresentation LearningSelf-Supervised Action Recognition