AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain
During the fine-tuning phase of transfer learning, the pretrained vocabulary remains unchanged, while model parameters are updated. The vocabulary generated based on the pretrained data is suboptimal for downstream data when domain discrepancy exists. We propose to consider the vocabulary as an optimizable parameter, allowing us to update the vocabulary by expanding it with domain-specific vocabulary based on a tokenization statistic. Furthermore, we preserve the embeddings of the added words from overfitting to downstream data by utilizing knowledge learned from a pretrained language model with a regularization term. Our method achieved consistent performance improvements on diverse domains (i.e., biomedical, computer science, news, and reviews).
Code (1)
Tasks
Language ModelingLanguage ModellingTransfer LearningSimilar Papers 제목 키워드 기반
Domain Adaptation for Large-Vocabulary Object Detectors
Large-vocabulary object detectors (LVDs) aim to detect objects of many categories, which learn super objectness features and can locate objects accurately while applied to various downstream data. However, LVDs often str…
Domain AdaptationKnowledge GraphsObjectObject LocalizationEvaluating Pretraining Strategies for Clinical BERT Models
Research suggests that using generic language models in specialized domains may be sub-optimal due to significant domain differences. As a result, various strategies for developing domain-specific language models have be…
Language ModelingLanguage ModellingGraph Neural Networks for Adapting Off-the-shelf General Domain Language Models to Low-Resource Specialised Domains
Language models encode linguistic proprieties and are used as input for more specific models. Using their word representations as-is for specialised and low-resource domains might be less efficient. Methods of adapting t…
Avocado Price Prediction Using a Hybrid Deep Learning Model: TCN-MLP-Attention Architecture
With the growing demand for healthy foods, agricultural product price forecasting has become increasingly important. Hass avocados, as a high-value crop, exhibit complex price fluctuations influenced by factors such as s…
ImputationTime Series ForecastingAVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
Audiovisual video captioning aims to generate semantically rich descriptions with temporal alignment between visual and auditory events, thereby benefiting both video understanding and generation. In this paper, we prese…
Video Captioning