Reducing data sparsity by generalising distributional contexts: application to specialised texts (R\'eduction de la dispersion des donn\'ees par g\'en\'eralisation des contextes distributionnels : application aux textes de sp\'ecialit\'e) [in French]

Generalising and Normalising Distributional Contexts to Reduce Data Sparsity: Application to Medical Corpora

Scale-invariant Attention

One persistent challenge in LLM research is the development of attention mechanisms that are able to generalise from training on shorter contexts to inference on longer contexts. We propose two conditions that we expect …

Robust Dialogue State Tracking with Weak Supervision and Sparse Data

Generalising dialogue state tracking (DST) to new data is especially challenging due to the strong reliance on abundant and fine-grained supervision during training. Sample sparsity, distributional shift and the occurren…

Scaling Attention via Feature Sparsity

Scaling Transformers to ultra-long contexts is bottlenecked by the $O(n^2 d)$ cost of self-attention. Existing methods reduce this cost along the sequence axis through local windows, kernel approximations, or token-level…

Conditional Distributional Treatment Effect with Kernel Conditional Mean Embeddings and U-Statistic Regression

We propose to analyse the conditional distributional treatment effect (CoDiTE), which, in contrast to the more common conditional average treatment effect (CATE), is designed to encode a treatment's distributional aspect…