Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing
Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization.
Code (1)
Tasks
Abstract Meaning RepresentationAMR ParsingDecoderSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing
Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and …
Abstract Meaning RepresentationAMR ParsingDecoderSentenceUnified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning
Unified Sequence Labeling that articulates different sequence labeling problems such as Named Entity Recognition, Relation Extraction, Semantic Role Labeling, etc. in a generalized sequence-to-sequence format opens up th…
In-Context LearningLanguage ModelingLanguage ModellingLarge Language Model+6TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusion
Reinforcement learning with stochastic optimal control offers a promising framework for diffusion fine-tuning, where a pre-trained diffusion model is optimized to generate paths that lead to a reward-tilted distribution.…
Reinforcement LearningSwarms of Large Language Model Agents for Protein Sequence Design with Experimental Validation
Designing proteins de novo with tailored structural, physicochemical, and functional properties remains a grand challenge in biotechnology, medicine, and materials science, due to the vastness of sequence space and the c…
Protein DesignLong Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity
The adaptation of pre-trained large language models (LLMs) to diverse downstream tasks via fine-tuning is critical for numerous applications. However, the inefficiency of parameter-efficient fine-tuning (PEFT) techniques…
parameter-efficient fine-tuning