Example-based Hypernetworks for Out-of-Distribution Generalization

2022-03-27 · Tomer Volk, Eyal Ben-David, Ohad Amosy, Gal Chechik, Roi Reichart

As Natural Language Processing (NLP) algorithms continually achieve new milestones, out-of-distribution generalization remains a significant challenge. This paper addresses the issue of multi-source adaptation for unfamiliar domains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-based Hypernetwork adaptation: a T5 encoder-decoder initially generates a unique signature from an input example, embedding it within the source domains' semantic space. This signature is subsequently utilized by a Hypernetwork to generate the task classifier's weights. We evaluated our method across two tasks - sentiment classification and natural language inference - in 29 adaptation scenarios, where it outpaced established algorithms. In an advanced version, the signature also enriches the input example's representation. We also compare our finetuned architecture to few-shot GPT-3, demonstrating its effectiveness in essential use cases. To our knowledge, this marks the first application of Hypernetworks to the adaptation for unknown domains.

📄 PDF Abstract BibTeX arXiv:2203.14276

Code (1)

tomervolk/hyper-pada 공식 구현 pytorch

Tasks

Decoder Domain Adaptation Natural Language InferenceOut-of-Distribution GeneralizationSentiment AnalysisSentiment Classification

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Multi-Head Attention 설명 없음

Adafactor Adafactor is a stochastic optimization method based on Adam that reduces memory usage while retaining the empirical benefits of…

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Gated Linear Unit A Gated Linear Unit, or GLU computes: $$ \mathrm{GLU}(a, b) = a \otimes \sigma(b) $$ It is used in natural language processing architectures, for example the Gated CNN,…

Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models

2026-07-21 · Nischay Dhankhar, Dos Baha, Abulhair Saparov arxiv

Injecting factual knowledge into large language models (LLMs) reliably and at scale remains an open challenge. Hypernetworks provide a promising solution to large-scale knowledge injection. Although hypernetworks are typ…

Test-time Adaptation

Bayesian Hypernetworks

2017-10-13 · ICLR 2018 1 · David Krueger, Chin-wei Huang, Riashat Islam, Ryan Turner 외

We study Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks. A Bayesian hypernetwork $\h$ is a neural network which learns to transform a simple noise distribution, $p(\vec\epsilon)…

Active Learning Anomaly Detection Bayesian Inference Variational Inference

Hypernetworks in Meta-Reinforcement Learning

2022-10-20 · Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, Shimon Whiteson

Training a reinforcement learning (RL) agent on a real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim to improve sample efficiency by generalizing over a distr…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Investigating the Effectiveness of HyperTuning via Gisting

2024-02-26 · Jason Phang

Gisting (Mu et al., 2023) is a simple method for training models to compress information into fewer token representations using a modified attention mask, and can serve as an economical approach to training Transformer-b…

Character-level HyperNetworks for Hate Speech Detection

2021-11-11 · Tomer Wullach, Amir Adler, Einat Minkov

The massive spread of hate speech, hateful content targeted at specific subpopulations, is a problem of critical social importance. Automated methods of hate speech detection typically employ state-of-the-art deep learni…

Data Augmentation Hate Speech Detection