An Empirical Exploration of Local Ordering Pre-training for Structured Prediction
Recently, pre-training contextualized encoders with language model (LM) objectives has been shown an effective semi-supervised method for structured prediction. In this work, we empirically explore an alternative pre-training method for contextualized encoders. Instead of predicting words in LMs, we {``}mask out{''} and predict word order information, with a local ordering strategy and word-selecting objectives. With evaluations on three typical structured prediction tasks (dependency parsing, POS tagging, and NER) over four languages (English, Finnish, Czech, and Italian), we show that our method is consistently beneficial. We further conduct detailed error analysis, including one that examines a specific type of parsing error where the head is misidentified. The results show that pre-trained contextual encoders can bring improvements in a structured way, suggesting that they may be able to capture higher-order patterns and feature combinations from unlabeled data.
Code (1)
Tasks
Dependency ParsingLanguage ModelingLanguage ModellingNERPOSPOS TaggingStructured PredictionSimilar Papers 제목 키워드 기반
HAMMER: Hamiltonian Curiosity Augmented Large Language Model Reinforcement
Recent curriculum reinforcement learning for large language models (LLMs) typically rely on difficulty-based annotations for data filtering and ordering. However, such methods suffer from local optimization, where contin…
Reinforcement LearningCan Graph Reordering Speed Up Graph Neural Network Training? An Experimental Study
Graph neural networks (GNNs) are a type of neural network capable of learning on graph-structured data. However, training GNNs on large-scale graphs is challenging due to iterative aggregations of high-dimensional featur…
CPUGPUGraph Neural NetworkCached Operator Reordering: A Unified View for Fast GNN Training
Graph Neural Networks (GNNs) are a powerful tool for handling structured graph data and addressing tasks such as node classification, graph classification, and clustering. However, the sparse nature of GNN computation po…
Graph AttentionGraph ClassificationNode ClassificationData Ordering Patterns for Neural Machine Translation: An Empirical Study
Recent works show that ordering of the training data affects the model performance for Neural Machine Translation. Several approaches involving dynamic data ordering and data sharding based on curriculum learning have be…
Machine TranslationTranslationGraph Spectral Regularization For Neural Network Interpretability
Deep neural networks can learn meaningful representations of data. However, these representations are hard to interpret. For example, visualizing a latent layer is generally only possible for at most three dimensions. Ne…