Distributionally Robust Finetuning BERT for Covariate Drift in Spoken Language Understanding
In this study, we investigate robustness against covariate drift in spoken language understanding (SLU). Covariate drift can occur in SLUwhen there is a drift between training and testing regarding what users request or how they request it. To study this we propose a method that exploits natural variations in data to create a covariate drift in SLU datasets. Experiments show that a state-of-the-art BERT-based model suffers performance loss under this drift. To mitigate the performance loss, we investigate distributionally robust optimization (DRO) for finetuning BERT-based models. We discuss some recent DRO methods, propose two new variants and empirically show that DRO improves robustness under drift.
Code (0)
등록된 구현이 없습니다.
Tasks
Spoken Language UnderstandingSimilar Papers 제목 키워드 기반
Detecting covariate drift in text data using document embeddings and dimensionality reduction
Detecting covariate drift in text data is essential for maintaining the reliability and performance of text analysis models. In this research, we investigate the effectiveness of different document embeddings, dimensiona…
ClassificationDimensionality ReductionDrift DetectionDistributionally Robust Policy Learning under Concept Drifts
Distributionally robust policy learning aims to find a policy that performs well under the worst-case distributional shift, and yet most existing methods for robust policy learning consider the worst-case joint distribut…
Distributionally Robust Bayesian Optimization
Robustness to distributional shift is one of the key challenges of contemporary machine learning. Attaining such robustness is the goal of distributionally robust optimization, which seeks a solution to an optimization p…
Bayesian OptimizationTraining data reduction for multilingual Spoken Language Understanding systems
Fine-tuning self-supervised pre-trained language models such as BERT has significantly improved state-of-the-art performance on natural language processing tasks. Similar finetuning setups can also be used in commercial …
intent-classificationIntent ClassificationSpoken Language UnderstandingCRUMB: Efficient Prior Fitted Network Inference via Distributionally Matched Context Batching
Prior-fitted networks (PFNs) are a promising class of tabular foundation models that perform in-context learning, whereby the entire labelled training set is supplied as context, and predictions for test queries are prod…