Constrained Density Matching and Modeling for Effective Contextualized Alignment
Multilingual representations pre-trained with monolingual data offer unmatched task performances between languages. While this has been tackled through the lens of contextualized alignments, these techniques require large parallel data, thereby leaving under-represented language communities behind. In this work, we analyze the limitations according to which previous alignments become very resource-intensive, \emph{viz.,} (i) the inability to sufficiently leverage data and (ii) that alignments are not trained properly. To address them, we present density based approaches to perform alignments, and we complement them with our validation criteria accounting for downstream task performances. Our experiments encompass 16 alignment techniques (including ours), evaluated across 6 language pairs, synthetic and 4 NLP tasks. We demonstrate that our solutions are particularly effective in the scenarios of limited and no parallel data. More importantly, we show, both theoretically and empirically, the advantages of our boostrapping procedures, by which unsupervised approaches rival supervised counterparts.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Constrained Density Matching and Modeling for Cross-lingual Alignment of Contextualized Representations
Multilingual representations pre-trained with monolingual data exhibit considerably unequal task performances across languages. Previous studies address this challenge with resource-intensive contextualized alignment, wh…
AttributeShould EBMs model the energy or the score?
Recent progress in training unnormalized models through denoising score matching with Langevin dynamics (SMLD) and denoising diffusion probabilistic modeling (DDPM) has made unnormalized models a competitive model class …
DenoisingmodelGenerative Adversarial Training for Text-to-Speech Synthesis Based on Raw Phonetic Input and Explicit Prosody Modelling
We describe an end-to-end speech synthesis system that uses generative adversarial training. We train our Vocoder for raw phoneme-to-audio conversion, using explicit phonetic, pitch and duration modeling. We experiment w…
Speech Synthesistext-to-speechText to SpeechText-To-Speech Synthesis+1On the minimax optimality of Flow Matching through the connection to kernel density estimation
Flow Matching has recently gained attention in generative modeling as a simple and flexible alternative to diffusion models, the current state of the art. While existing statistical guarantees adapt tools from the analys…
Density EstimationGenerative Modeling of Quantum Distribution with Functional Flow Matching
The emergence of powerful deep generative models based on diffusion and flow matching has enabled the learning and modeling of complex distributions. Learning quantum distributions, however, remains challenging due to th…