홈 › Papers

Arabic Multi-Dialect Segmentation: bi-LSTM-CRF vs. SVM

2017-08-19 · Mohamed Eldesouki, Younes Samih, Ahmed Abdelali, Mohammed Attia, Hamdy Mubarak, Kareem Darwish, Kallmeyer Laura

Arabic word segmentation is essential for a variety of NLP applications such as machine translation and information retrieval. Segmentation entails breaking words into their constituent stems, affixes and clitics. In this paper, we compare two approaches for segmenting four major Arabic dialects using only several thousand training examples for each dialect. The two approaches involve posing the problem as a ranking problem, where an SVM ranker picks the best segmentation, and as a sequence labeling problem, where a bi-LSTM RNN coupled with CRF determines where best to segment words. We are able to achieve solid segmentation results for all dialects using rather limited training data. We also show that employing Modern Standard Arabic data for domain adaptation and assuming context independence improve overall results.

📄 PDF Abstract BibTeX arXiv:1708.05891

Code (2)

qcri/dialectal_arabic_segmenter tf

qcri/dialectal_arabic_tools

Tasks

Domain Adaptation Information Retrieval Machine Translation Retrieval Segmentation Sentiment Analysis Translation

Methods 이 논문이 사용한 방법론

SVM A Support Vector Machine, or SVM, is a non-parametric supervised learning model. For non-linear classification and regression, they utilise the kernel trick to map inputs…

CRF Conditional Random Fields or CRFs are a type of probabilistic graph model that take neighboring sample context into account for tasks like classification. Prediction is…

Learning from Relatives: Unified Dialectal Arabic Segmentation

2017-08-01 · CONLL 2017 8 · Younes Samih, Mohamed Eldesouki, Mohammed Attia, Kareem Darwish 외

Arabic dialects do not just share a common koin{\'e}, but there are shared pan-dialectal linguistic phenomena that allow computational models for dialects to learn from each other. In this paper we build a unified segmen…

Dialect Identification Information Retrieval Machine Translation Segmentation

Morphology-aware Word-Segmentation in Dialectal Arabic Adaptation of Neural Machine Translation

2019-08-01 · WS 2019 8 · Ahmed Tawfik, Mahitab Emam, Khaled Essam, Robert Nabil 외

Parallel corpora available for building machine translation (MT) models for dialectal Arabic (DA) are rather limited. The scarcity of resources has prompted the use of Modern Standard Arabic (MSA) abundant resources to c…

Machine Translation Segmentation Translation

Pre-trained Models or Feature Engineering: The Case of Dialectal Arabic

2022-06-01 · OSACT (LREC) 2022 6 · Kathrein Abu Kwaik, Stergios Chatzikyriakidis, Simon Dobnik

The usage of social media platforms has resulted in the proliferation of work on Arabic Natural Language Processing (ANLP), including the development of resources. There is also an increased interest in processing Arabic…

Dialect Identification Feature Engineering Sentiment Analysis Word Embeddings

A Deep Learning Approach for Similar Languages, Varieties and Dialects

2019-01-02 · Vidya Prasad K, Akarsh S, Vinayakumar R, Soman Kp

Deep learning mechanisms are prevailing approaches in recent days for the various tasks in natural language processing, speech recognition, image processing and many others. To leverage this we use deep learning based me…

Deep Learning Dialect Identification speech-recognition Speech Recognition

Discrimination between Similar Languages, Varieties and Dialects using CNN- and LSTM-based Deep Neural Networks

2016-12-01 · WS 2016 12 · Chinnappa Guggilla

In this paper, we describe a system (CGLI) for discriminating similar languages, varieties and dialects using convolutional neural networks (CNNs) and long short-term memory (LSTM) neural networks. We have participated i…

Dialect Identification Information Retrieval Language Identification Machine Translation+2