홈 › Papers

XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation

2020-04-03 · Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou

In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in English for natural language understanding tasks only, XGLUE has two main advantages: (1) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (2) for each task, it provides labeled data in multiple languages. We extend a recent cross-lingual pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison.

📄 PDF Abstract BibTeX arXiv:2004.01401

Code (2)

microsoft/Unicoder pytorch

microsoft/XGLUE

Tasks

Natural Language UnderstandingXLM-R

Methods 이 논문이 사용한 방법론

XLM-R XLM-R

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Weight Decay 설명 없음

WordPiece 설명 없음

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

BERT BERT, or Bidirectional Encoder Representations from Transformers, improves upon standard Transformers by removing the…

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation

2020-11-01 · EMNLP 2020 11 · Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu 외

In this paper, we introduce XGLUE, a new benchmark dataset to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora, and evaluate their performance across a diverse set of cross-ling…

Natural Language Understanding XLM-R

MultiLegalPile: A 689GB Multilingual Legal Corpus

2023-06-03 · Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis 외

Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, there are few datasets available for specialized critical domains such as law and the available ones are often only for…

CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

2021-02-09 · Shuai Lu, Daya Guo, Shuo Ren, JunJie Huang 외

Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understandin…

BIG-bench Machine Learning Clone Detection Cloze Test Code Completion+9

ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark

2023-03-09 · Ilias Chalkidis

Following the hype around OpenAI's ChatGPT conversational agent, the last straw in the recent development of Large Language Models (LLMs) that demonstrate emergent unprecedented zero-shot capabilities, we audit the lates…

Instruction Following

Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models

2023-07-12 · James O' Neill, Sourav Dutta

We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models. We present a new method called self-distilled quantization (SDQ) that minimiz…

Quantization XLM-R