QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search
In light of the success of the pre-trained language models (PLMs), continual pre-training of generic PLMs has been the paradigm of domain adaption. In this paper, we propose QUERT, A Continual Pre-trained Language Model for QUERy Understanding in Travel Domain Search. QUERT is jointly trained on four tailored pre-training tasks to the characteristics of query in travel domain search: Geography-aware Mask Prediction, Geohash Code Prediction, User Click Behavior Learning, and Phrase and Token Order Prediction. Performance improvement of downstream tasks and ablation experiment demonstrate the effectiveness of our proposed pre-training tasks. To be specific, the average performance of downstream tasks increases by 2.02% and 30.93% in supervised and unsupervised settings, respectively. To check on the improvement of QUERT to online business, we deploy QUERT and perform A/B testing on Fliggy APP. The feedback results show that QUERT increases the Unique Click-Through Rate and Page Click-Through Rate by 0.89% and 1.03% when applying QUERT as the encoder. Our code and downstream task data will be released for future research.
Code (1)
Tasks
Domain AdaptationLanguage ModelingLanguage ModellingPredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-training
The advancement of Large Language Models (LLMs) has significantly transformed the field of natural language processing, although the focus on English-centric models has created a noticeable research gap for specific lang…
Language ModelingLanguage ModellingLarge Language ModelQuestion Answering+3Affordance-First Decomposition for Continual Learning in Video-Language Understanding
Continual learning for video--language understanding is increasingly important as models face non-stationary data, domains, and query styles, yet prevailing solutions blur what should stay stable versus what should adapt…
Continual LearningReal-Time Trend Prediction via Continually-Aligned LLM Query Generation
Trending news detection in low-traffic search environments faces a fundamental cold-start problem, where a lack of query volume prevents systems from identifying emerging or long-tail trends. Existing methods relying on …
Continual LearningWeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens
Recent progress in multimodal large language models (MLLMs) has highlighted the challenge of efficiently bridging pre-trained Vision-Language Models (VLMs) with Diffusion Models. While methods using a fixed number of lea…
Computational EfficiencyContinual LearningInjecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG
The injection of domain-specific knowledge is crucial for adapting language models (LMs) to specialized fields such as biomedicine. While most current approaches rely on unstructured text corpora, this study explores two…
Continual PretrainingQuestion Answering