Contrastive Document Representation Learning with Graph Attention Networks
Recent progress in pretrained Transformer-based language models has shown great success in learning contextual representation of text. However, due to the quadratic self-attention complexity, most of the pretrained Transformers models can only handle relatively short text. It is still a challenge when it comes to modeling very long documents. In this work, we propose to use a graph attention network on top of the available pretrained Transformers model to learn document embeddings. This graph attention network allows us to leverage the high-level semantic structure of the document. In addition, based on our graph document model, we design a simple contrastive learning strategy to pretrain our models on a large amount of unlabeled corpus. Empirically, we demonstrate the effectiveness of our approaches in document classification and document retrieval tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Contrastive LearningDocument ClassificationGraph AttentionRepresentation LearningRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Graph Contrastive Topic Model
Existing NTMs with contrastive learning suffer from the sample bias problem owing to the word frequency-based sampling strategy, which may result in false negative samples with similar semantics to the prototypes. In thi…
Contrastive LearningmodelRepresentation LearningSkim-Aware Contrastive Learning for Efficient Document Representation
Although transformer-based models have shown strong performance in word- and sentence-level tasks, effectively representing long documents, especially in fields like law and medicine, remains difficult. Sparse attention …
Natural Language InferenceContrastive LearningContrastive Hierarchical Discourse Graph for Scientific Document Summarization
The extended structural context has made scientific paper summarization a challenging task. This paper proposes CHANGES, a contrastive hierarchical graph neural network for extractive scientific paper summarization. CHAN…
Contrastive LearningDocument SummarizationGraph Neural NetworkScientific Document Summarization+1StructCoh: Structured Contrastive Learning for Context-Aware Text Semantic Matching
Text semantic matching requires nuanced understanding of both structural relationships and fine-grained semantic distinctions. While pre-trained language models excel at capturing token-level interactions, they often ove…
Contrastive LearningDependency ParsingAttention-wise masked graph contrastive learning for predicting molecular property
Accurate and efficient prediction of the molecular properties of drugs is one of the fundamental problems in drug research and development. Recent advancements in representation learning have been shown to greatly improv…
Contrastive LearningGraph AttentionMolecular Property Predictionmolecular representation+3