Unfreeze with Care: Space-Efficient Fine-Tuning of Semantic Parsing Models
Semantic parsing is a key NLP task that maps natural language to structured meaning representations. As in many other NLP tasks, SOTA performance in semantic parsing is now attained by fine-tuning a large pretrained language model (PLM). While effective, this approach is inefficient in the presence of multiple downstream tasks, as a new set of values for all parameters of the PLM needs to be stored for each task separately. Recent work has explored methods for adapting PLMs to downstream tasks while keeping most (or all) of their parameters frozen. We examine two such promising techniques, prefix tuning and bias-term tuning, specifically on semantic parsing. We compare them against each other on two different semantic parsing datasets, and we also compare them against full and partial fine-tuning, both in few-shot and conventional data settings. While prefix tuning is shown to do poorly for semantic parsing tasks off the shelf, we modify it by adding special token embeddings, which results in very strong performance without compromising parameter savings.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSemantic ParsingSimilar Papers 제목 키워드 기반
Positional Prompt Tuning for Efficient 3D Representation Learning
Point cloud analysis has achieved significant development and is well-performed in multiple downstream tasks like point cloud classification and segmentation, etc. Being conscious of the simplicity of the position encodi…
3D Parameter-Efficient Fine-Tuning for Classification3D Point Cloud Classificationparameter-efficient fine-tuningPoint Cloud Classification+2Collaborative Semantic Alignment in Recommendation Systems
Traditional recommender systems primarily leverage identity-based (ID) representations for users and items, while the advent of pre-trained language models (PLMs) has introduced rich semantic modeling of item description…
Collaborative FilteringLanguage ModellingRecommendation SystemsInvestigating Transfer Learning Capabilities of Vision Transformers and CNNs by Fine-Tuning a Single Trainable Block
In recent developments in the field of Computer Vision, a rise is seen in the use of transformer-based architectures. They are surpassing the state-of-the-art set by CNN architectures in accuracy but on the other hand, t…
Transfer LearningLinMU: Multimodal Understanding Made Linear
Modern Vision-Language Models (VLMs) achieve impressive performance but are limited by the quadratic complexity of self-attention, which prevents their deployment on edge devices and makes their understanding of high-res…
Multimodal ReasoningCogAdapt: Adapting Clinical ECG Foundation Models for Wearable Cognitive Load Assessment
Assessing cognitive load continuously and at low latency would help adaptive human-computer interaction, but it remains hard because labeled data are scarce and models generalize poorly across subjects. Recent ECG founda…