MATE: Matryoshka Audio-Text Embeddings for Open-Vocabulary Keyword Spotting

2026-01-20 · Youngmoon Jung, Myunghun Jung, Joon-Young Yang, Yong-Hyeok Lee, Jaeyoung Roh, Hoon-Young Cho arxiv

Open-vocabulary keyword spotting (KWS) with text-based enrollment has emerged as a flexible alternative to fixed-phrase triggers. Prior utterance-level matching methods, from an embedding-learning standpoint, learn embeddings at a single fixed dimensionality. We depart from this design and propose Matryoshka Audio-Text Embeddings (MATE), a dual-encoder framework that encodes multiple embedding granularities within a single vector via nested sub-embeddings ("prefixes"). Specifically, we introduce a PCA-guided prefix alignment: PCA-compressed versions of the full text embedding for each prefix size serve as teacher targets to align both audio and text prefixes. This alignment concentrates salient keyword cues in lower-dimensional prefixes, while higher dimensions add detail. MATE is trained with standard deep metric learning objectives for audio-text KWS, and is loss-agnostic. To our knowledge, this is the first application of matryoshka-style embeddings to KWS, achieving state-of-the-art results on WSJ and LibriPhrase without any inference overhead.

📄 PDF Abstract BibTeX arXiv:2601.14012

Code (0)

등록된 구현이 없습니다.

Tasks

Keyword Spotting Metric Learning

WavLink: Compact Audio-Text Embeddings with a Global Whisper Token

2026-01-21 · Gokul Karthik Kumar, Ludovick Lepauloux, Hakim Hacid arxiv

Whisper has become the de-facto encoder for extracting general-purpose audio features in large audio-language models, where a 30-second clip is typically represented by 1500 frame features projected into an LLM. In contr…

ML-Embed: Inclusive and Efficient Embeddings for a Multilingual World

2026-05-14 · Ziyin Zhang, Zihan Liao, Hang Yu, Peng Di 외 arxiv

The development of high-quality text embeddings is increasingly drifting toward an exclusionary future, defined by three critical barriers: prohibitive computational costs, a narrow linguistic focus that neglects most of…

Representation Learning

SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression

2025-10-14 · Biao Zhang, Lixin Chen, Tong Liu, Bo Zheng arxiv

Large language models (LLMs) generate high-dimensional embeddings that capture rich semantic and syntactic information. However, high-dimensional embeddings exacerbate computational complexity and storage requirements, t…

Dimensionality Reduction Representation Learning

Hierarchical Level-Wise News Article Clustering via Multilingual Matryoshka Embeddings

2025-05-30 · Hans W. A. Hanley, Zakir Durumeric

Contextual large language model embeddings are increasingly utilized for topic modeling and clustering. However, current methods often scale poorly, rely on opaque similarity metrics, and struggle in multilingual setting…

Articles Clustering Language Modeling Language Modelling+1

Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs

2025-03-09 · Umberto Cappellazzo, Minsu Kim, Stavros Petridis

Audio-Visual Speech Recognition (AVSR) leverages both audio and visual modalities to enhance speech recognition robustness, particularly in noisy environments. Recent advancements in Large Language Models (LLMs) have dem…

Audio-Visual Speech Recognition Computational Efficiency Representation Learning speech-recognition+2