TBGC: Task-level Backbone-Oriented Gradient Clip for Multi-Task Foundation Model Learning
The AllInOne training paradigm squeezes a wide range of tasks into a unified model in a multi-task learning manner. However, optimization in multi-task learning is more challenge than single-task learning, as the gradient norm from different tasks may vary greatly, making the backbone overly biased towards one specific task. To address this issue, we propose the task-level backbone-oriented gradient clip paradigm, compared with the vanilla gradient clip method, it has two points of emphasis:1) gradient clip is performed independently for each task. 2) backbone gradients generated from each task are rescaled to the same norm scale. Based on the experimental results, we argue that the task-level backbone-oriented gradient clip paradigm can relieve the gradient bias problem to some extent. We also propose a novel multi-branch data augmentation strategy where conflict augmentations are placed in different branches. Our approach has been shown to be effective and finally achieve 1st place in the Leaderboard A and 2nd place in the Leaderboard B of the CVPR2023 Foundation Model Challenge. It's worth noting that instead of evaluating all three tasks(detection, segmentation and fine-grained classification) in Leaderboard A, the segmentation task is not evaluated in Leaderboard B, in which our team has a huge advantage.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationMulti-Task LearningTask 2Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-training
Existing reasoning data curation pipelines score whole samples, treating every intermediate step as equally valuable. In reality, steps within a trace contribute very unevenly, and selecting reasoning data well requires …
Distributional Soft Actor-Critic with Harmonic Gradient for Safe and Efficient Autonomous Driving in Multi-lane Scenarios
Reinforcement learning (RL), known for its self-evolution capability, offers a promising approach to training high-level autonomous driving systems. However, handling constraints remains a significant challenge for exist…
Autonomous DrivingReinforcement Learning (RL)Contextual Gradient Scaling for Few-Shot Learning
Model-agnostic meta-learning (MAML) is a well-known optimization-based meta-learning algorithm that works well in various computer vision tasks, e.g., few-shot classification. MAML is to learn an initialization so that a…
Cross-Domain Few-ShotFew-Shot LearningMeta-LearningFULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibration
Multi-modality fusion and multi-task learning are becoming trendy in 3D autonomous driving scenario, considering robust prediction and computation budget. However, naively extending the existing framework to the domain o…
Autonomous DrivingMulti-Task LearningDeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer
Recent advances in knowledge distillation have emphasized the importance of decoupling different knowledge components. While existing methods utilize momentum mechanisms to separate task-oriented and distillation gradien…
DenoisingKnowledge DistillationTransfer Learning