Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

2022-05-12 · ACL 2022 5 · Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, Monojit Choudhury

Massively Multilingual Transformer based Language Models have been observed to be surprisingly effective on zero-shot transfer across languages, though the performance varies from language to language depending on the pivot language(s) used for fine-tuning. In this work, we build upon some of the existing techniques for predicting the zero-shot performance on a task, by modeling it as a multi-task learning problem. We jointly train predictive models for different tasks which helps us build more accurate predictors for tasks where we have test data in very few languages to measure the actual performance of the model. Our approach also lends us the ability to perform a much more robust feature selection and identify a common set of features that influence zero-shot performance across a variety of tasks.

📄 PDF Abstract BibTeX arXiv:2205.06130

Code (0)

등록된 구현이 없습니다.

Tasks

feature selectionMulti-Task Learning

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Feature Selection Feature selection, also known as variable selection, attribute selection or variable subset selection, is the process of selecting a subset of relevant features (variables,…

Adam 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Residual Connection 설명 없음

A Generalization Theory for Zero-Shot Prediction

2025-07-12 · Ronak Mehta, Zaid Harchaoui arxiv

A modern paradigm for generalization in machine learning and AI consists of pre-training a task-agnostic foundation model, generally obtained using self-supervised and multimodal contrastive learning. The resulting repre…

Contrastive Learning

The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge

2024-07-06 · Longfei Huang, Feng Yu, Zhihao Guan, Zhonghua Wan 외

This report presents a solution for the zero-shot referring expression comprehension task. Visual-language multimodal base models (such as CLIP, SAM) have gained significant attention in recent years as a cornerstone of …

Referring Expression Referring Expression Comprehension

A Simple and Effective Framework for Strict Zero-Shot Hierarchical Classification

2023-05-24 · Rohan Bhambhoria, Lei Chen, Xiaodan Zhu

In recent years, large language models (LLMs) have achieved strong performance on benchmark tasks, especially in zero or few-shot settings. However, these benchmarks often do not adequately address the challenges posed i…

Bilingual Alignment Pre-Training for Zero-Shot Cross-Lingual Transfer

2021-06-03 · EMNLP (MRQA) 2021 11 · Ziqing Yang, Wentao Ma, Yiming Cui, Jiani Ye 외

Multilingual pre-trained models have achieved remarkable performance on cross-lingual transfer learning. Some multilingual models such as mBERT, have been pre-trained on unlabeled corpora, therefore the embeddings of dif…

Cross-Lingual Transfer Language Modelling Machine Reading Comprehension Reading Comprehension+2

TabFM: A Zero-Shot Foundation Model for Tabular Data

2026-09-29 · Weihao Kong, Erez Louidor Ilan, Shuxin Nie, Taman Narayan 외 hf

Tabular machine learning typically relies on per-dataset workflows, fitting tree ensembles or running AutoML searches from scratch for every task. We present TabFM, a 400M-parameter tabular foundation model that formulat…

Feature Engineering