HIT: A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation
Understanding linguistics and morphology of resource-scarce code-mixed texts remains a key challenge in text processing. Although word embedding comes in handy to support downstream tasks for low-resource languages, there are plenty of scopes in improving the quality of language representation particularly for code-mixed languages. In this paper, we propose HIT, a robust representation learning method for code-mixed texts. HIT is a hierarchical transformer-based framework that captures the semantic relationship among words and hierarchically learns the sentence-level semantics using a fused attention mechanism. HIT incorporates two attention modules, a multi-headed self-attention and an outer product attention module, and computes their weighted sum to obtain the attention weights. Our evaluation of HIT on one European (Spanish) and five Indic (Hindi, Bengali, Tamil, Telugu, and Malayalam) languages across four NLP tasks on eleven datasets suggests significant performance improvement against various state-of-the-art systems. We further show the adaptability of learned representation across tasks in a transfer learning setup (with and without fine-tuning).
Code (1)
Tasks
Deep AttentionRepresentation LearningSentenceTransfer LearningSimilar Papers 제목 키워드 기반
HIT - A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation
Diagonal-Tiled Mixed-Precision Attention for Efficient Low-Bit MXFP Inference
Transformer-based large language models (LLMs) have demonstrated remarkable performance across a wide range of real-world tasks, but their inference cost remains prohibitively high due to the quadratic complexity of atte…
Attention-Infused Autoencoder for Massive MIMO CSI Compression
As the number of multiple-input multiple-output (MIMO) antennas increases drastically with the development towards 6G systems, channel state information (CSI) compression becomes crucial for mitigating feedback overhead.…
Harmonizing Code-mixed Conversations: Personality-assisted Code-mixed Response Generation in Dialogues
Code-mixing, the blending of multiple languages within a single conversation, introduces a distinctive challenge, particularly in the context of response generation. Capturing the intricacies of code-mixing proves to be …
Response GenerationRetrieval Augmented Enhanced Dual Co-Attention Framework for Target Aware Multimodal Bengali Hateful Meme Detection
Hateful content on social media increasingly appears as multimodal memes that combine images and text to convey harmful narratives. In low-resource languages such as Bengali, automated detection remains challenging due t…
Hate Speech DetectionSemantic Similarity