PILE: Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation

2022-11-11 · Lianshang Cai, Linhao Zhang, Dehong Ma, Jun Fan, Daiting Shi, Yi Wu, Zhicong Cheng, Simiu Gu, Dawei Yin

Pre-trained language models have become a crucial part of ranking systems and achieved very impressive effects recently. To maintain high performance while keeping efficient computations, knowledge distillation is widely used. In this paper, we focus on two key questions in knowledge distillation for ranking models: 1) how to ensemble knowledge from multi-teacher; 2) how to utilize the label information of data in the distillation process. We propose a unified algorithm called Pairwise Iterative Logits Ensemble (PILE) to tackle these two questions simultaneously. PILE ensembles multi-teacher logits supervised by label information in an iterative way and achieved competitive performance in both offline and online experiments. The proposed method has been deployed in a real-world commercial search system.

📄 PDF Abstract BibTeX arXiv:2211.06059

Code (0)

등록된 구현이 없습니다.

Tasks

Knowledge Distillation

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Ensemble-based Adversarial Defense Using Diversified Distance Mapping

2021-01-01 · Ehsan Kazemi, Mohamed E. Hussein, Wael AbdAlmgaeed

We propose an ensemble-based defense against adversarial examples using distance map layers (DMLs). Similar to linear (map) layers, DMLs can be used to output logits for a multi-class classification model. We show in thi…

Adversarial Defense Diversity Multi-class Classification

Compiler-Guided Adaptive Proof Search with Cross-Model Synergy on Context-Dependent Theorem Proving

2026-06-04 · Zhuo Liu, Ding Yu, Hangfeng He arxiv

Theorem proving in real-world Lean 4 projects is challenging because proofs often depend on project-specific context. While iterative refinement can use compiler errors to repair failed proofs, reusing failed attempts re…

Embedded Self-Distillation in Compact Multi-Branch Ensemble Network for Remote Sensing Scene Classification

2021-04-01 · Qi Zhao, Yujing Ma, Shuchang Lyu, Lijiang Chen

Remote sensing (RS) image scene classification task faces many challenges due to the interference from different characteristics of different geographical elements. To solve this problem, we propose a multi-branch ensemb…

General Classification Scene Classification

GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation

2024-03-25 · CVPR 2024 1 · Weiming Zhang, Yexin Liu, Xu Zheng, Lin Wang

This paper tackles a novel yet challenging problem: how to transfer knowledge from the emerging Segment Anything Model (SAM) -- which reveals impressive zero-shot instance segmentation capacity -- to learn a compact pano…

Domain Adaptation Instance Segmentation Semantic Segmentation Transfer Learning+1

Multi-label ensemble based on variable pairwise constraint projection

2014-03-08 · Ping Li, Hong Li, Min Wu

Multi-label classification has attracted an increasing amount of attention in recent years. To this end, many algorithms have been developed to classify multi-label data in an effective manner. However, they usually do n…

Classification General Classification Multi-Label Classification MUlTI-LABEL-ClASSIFICATION