Graph Neural Networks for Adapting Off-the-shelf General Domain Language Models to Low-Resource Specialised Domains
Language models encode linguistic proprieties and are used as input for more specific models. Using their word representations as-is for specialised and low-resource domains might be less efficient. Methods of adapting them exist, but these models often overlook global information about how words, terms, and concepts relate to each other in a corpus due to their strong reliance on attention. We consider that global information can influence the results of the downstream tasks, and combination with contextual information is performed using graph convolution networks or GCN built on vocabulary graphs. By outperforming baselines, we show that this architecture is profitable for domain-specific tasks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Online Learning via Memory: Retrieval-Augmented Detector Adaptation
This paper presents a novel way of online adapting any off-the-shelf object detection model to a novel domain without retraining the detector model. Inspired by how humans quickly learn knowledge of a new subject (e.g., …
Memorizationobject-detectionObject DetectionRetrievalAdapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Large pre-trained models have achieved great success in many natural language processing tasks. However, when they are applied in specific domains, these models suffer from domain shift and bring challenges in fine-tunin…
Knowledge DistillationVLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events
The rapid growth of ego-centric dashcam footage presents a major challenge for detecting safety-critical events such as collisions and near-collisions, scenarios that are brief, rare, and difficult for generic vision mod…
Visual Question AnsweringAutonomous DrivingAnomaly DetectionShelfRectNet: Single View Shelf Image Rectification with Homography Estimation
Estimating homography from a single image remains a challenging yet practically valuable task, particularly in domains like retail, where only one viewpoint is typically available for shelf monitoring and product alignme…
Homography EstimationToward Accurate Long-Horizon Robotic Manipulation: Language-to-Action with Foundation Models via Scene Graphs
This paper presents a framework that leverages pre-trained foundation models for robotic manipulation without domain-specific training. The framework integrates off-the-shelf models, combining multimodal perception from …