Sample Compression Hypernetworks: From Generalization Bounds to Meta-Learning

2024-10-17 · Benjamin Leblanc, Mathieu Bazinet, Nathaniel D'Amours, Alexandre Drouin, Pascal Germain

Reconstruction functions are pivotal in sample compression theory, a framework for deriving tight generalization bounds. From a small sample of the training set (the compression set) and an optional stream of information (the message), they recover a predictor previously learned from the whole training set. While usually fixed, we propose to learn reconstruction functions. To facilitate the optimization and increase the expressiveness of the message, we derive a new sample compression generalization bound for real-valued messages. From this theoretical analysis, we then present a new hypernetwork architecture that outputs predictors with tight generalization guarantees when trained using an original meta-learning framework. The results of promising preliminary experiments are then reported.

📄 PDF Abstract BibTeX arXiv:2410.13577

Code (0)

등록된 구현이 없습니다.

Tasks

Generalization BoundsMeta-Learning

Methods 이 논문이 사용한 방법론

HyperNetwork A HyperNetwork is a network that generates weights for a main network. The behavior of the main network is the same with any usual neural network: it learns to map some raw…

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Sample Compression Unleashed: New Generalization Bounds for Real Valued Losses

2024-09-26 · Mathieu Bazinet, Valentina Zantedeschi, Pascal Germain

The sample compression theory provides generalization guarantees for predictors that can be fully defined using a subset of the training dataset and a (short) message string, generally defined as a binary sequence. Previ…

Generalization Bounds valid

Hypernetworks in Meta-Reinforcement Learning

2022-10-20 · Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, Shimon Whiteson

Training a reinforcement learning (RL) agent on a real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim to improve sample efficiency by generalizing over a distr…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Hypernetworks build Implicit Neural Representations of Sounds

2023-02-09 · Filip Szatkowski, Karol J. Piczak, Przemysław Spurek, Jacek Tabor 외

Implicit Neural Representations (INRs) are nowadays used to represent multimedia signals across various real-life applications, including image super-resolution, image compression, or 3D rendering. Existing methods that …

Image Compression Image Super-Resolution Meta-Learning Super-Resolution

HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworks

2022-11-03 · Filip Szatkowski, Karol J. Piczak, Przemysław Spurek, Jacek Tabor 외

Implicit neural representations (INRs) are a rapidly growing research field, which provides alternative ways to represent multimedia signals. Recent applications of INRs include image super-resolution, compression of hig…

Image Super-Resolution Meta-Learning Super-Resolution

The Pick-to-Learn Algorithm: Empowering Compression for Tight Generalization Bounds and Improved Post-training Performance

2023-09-21 · NeurIPS 2023 11

Generalization bounds are valuable both for theory and applications. On the one hand, they shed light on the mechanisms that underpin the learning processes; on the other, they certify how well a learned model performs a…