The Unreasonable Effectiveness of Structured Random Orthogonal Embeddings
We examine a class of embeddings based on structured random matrices with orthogonal rows which can be applied in many machine learning applications including dimensionality reduction and kernel approximation. For both the Johnson-Lindenstrauss transform and the angular kernel, we show that we can select matrices yielding guaranteed improved performance in accuracy and/or speed compared to earlier methods. We introduce matrices with complex entries which give significant further accuracy improvement. We provide geometric and Markov chain-based perspectives to help understand the benefits, and empirical results which suggest that the approach is helpful in a wider range of applications.
Code (2)
Tasks
BIG-bench Machine LearningDimensionality ReductionSimilar Papers 제목 키워드 기반
The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation
Continuous-output neural machine translation (CoNMT) replaces the discrete next-word prediction problem with an embedding prediction. The semantic structure of the target embedding space (i.e., closeness of related words…
Machine TranslationPredictionTranslationOrthogonal Random Features
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel a…
The unreasonable effectiveness of pattern matching
We report on an astonishing ability of large language models (LLMs) to make sense of "Jabberwocky" language in which most or all content words have been randomly replaced by nonsense strings, e.g., translating "He dwushe…
The Unreasonable Effectiveness of Large Language-Vision Models for Source-free Video Domain Adaptation
Source-Free Video Unsupervised Domain Adaptation (SFVUDA) task consists in adapting an action recognition model, trained on a labelled source dataset, to an unlabelled target dataset, without accessing the actual source …
Action RecognitionDomain AdaptationUnsupervised Domain AdaptationThe Unreasonable Effectiveness of Randomized Representations in Online Continual Graph Learning
Catastrophic forgetting is one of the main obstacles for Online Continual Graph Learning (OCGL), where nodes arrive one by one, distribution drifts may occur at any time and offline training on task-specific subgraphs is…
Graph Learning