Spectral Probing
Linguistic information is encoded at varying timescales (subwords, phrases, etc.) and communicative levels, such as syntax and semantics. Contextualized embeddings have analogously been found to capture these phenomena at distinctive layers and frequencies. Leveraging these findings, we develop a fully learnable frequency filter to identify spectral profiles for any given task. It enables vastly more granular analyses than prior handcrafted filters, and improves on efficiency. After demonstrating the informativeness of spectral probing over manual filters in a monolingual setting, we investigate its multilingual characteristics across seven diverse NLP tasks in six languages. Our analyses identify distinctive spectral profiles which quantify cross-task similarity in a linguistically intuitive manner, while remaining consistent across languages-highlighting their potential as robust, lightweight task descriptors.
Code (1)
Tasks
InformativenessSimilar Papers 제목 키워드 기반
Channel Performance Estimations with Extended Channel Probing
We test the concept of extended channel probing in an Optical Spectrum as a Service scenario in coherent optimized flex-grid long-haul and 10Gbit/s OOK optimized 100-GHz fixed-grid dispersion-managed legacy DWDM networks…
Spectral Temporal Contrastive Learning
Learning useful data representations without requiring labels is a cornerstone of modern deep learning. Self-supervised learning methods, particularly contrastive learning (CL), have proven successful by leveraging data …
Contrastive LearningSelf-Supervised LearningActive Beam Learning for Full-Duplex Wireless Systems
In this paper, we present a novel active beam learning method for in-band full-duplex wireless systems, that aims to design transmit and receive beams which suppress self-interference and maximize the sum spectral effici…
Spectral Aliasing Pretext: A novel task for Self-Supervised fault diagnosis in rotating machinery
Deep learning is a new way for machinery fault diagnosis but requires extensive labeled data, a scarce resource in industrial settings. We propose Spectral Aliasing Pretext (SAP), a self-supervised learning method that p…
Self-Supervised LearningFault DiagnosisMachine learning applied to quantum synchronization-assisted probing
A probing scheme is considered with an accessible and controllable qubit, used to probe an out-of equilibrium system consisting of a second qubit interacting with an environment. Quantum spontaneous synchronization betwe…
BIG-bench Machine Learning