Self-tuning hyper-parameters for unsupervised cross-lingual tokenization
We explore the possibility of meta-learning for the language-independent unsupervised tokenization problem for English, Russian, and Chinese. We implement the meta-learning approach for automatic determination of hyper-parameters of the unsupervised tokenization model proposed in earlier works, relying on various human-independent fitness functions such as normalised anti-entropy, compression factor and cross-split F1 score, as well as additive and multiplicative composite combinations of the three metrics, testing them against the conventional F1 tokenization score. We find a fairly good correlation between the latter and the additive combination of the former three metrics for English and Russian. In case of Chinese, we find a significant correlation between the F 1 score and the compression factor. Our results suggest the possibility of robust unsupervised tokenization of low-resource and dead languages and allow us to think about human languages in terms of the evolution of efficient symbolic communication codes with different structural optimisation schemes that have evolved in different human cultures.
Code (0)
등록된 구현이 없습니다.
Tasks
Meta-LearningSimilar Papers 제목 키워드 기반
Towards Automated Self-Supervised Learning for Truly Unsupervised Graph Anomaly Detection
Self-supervised learning (SSL) is an emerging paradigm that exploits supervisory signals generated from the data itself, and many recent studies have leveraged SSL to conduct graph anomaly detection. However, we empirica…
Anomaly DetectionGraph Anomaly DetectionSelf-Supervised LearningUnsupervised Anomaly DetectionTSA on AutoPilot: Self-tuning Self-supervised Time Series Anomaly Detection
Time series anomaly detection (TSAD) finds many applications such as monitoring environmental sensors, industry KPIs, patient biomarkers, etc. A two-fold challenge for TSAD is a versatile and unsupervised model that can …
Anomaly DetectionData AugmentationTime SeriesTime Series Anomaly DetectionUnsupervised Domain Adaptation for Text Classification via Meta Self-Paced Learning
A shift in data distribution can have a significant impact on performance of a text classification model. Recent methods addressing unsupervised domain adaptation for textual tasks typically extracted domain-invariant re…
Domain AdaptationMeta-Learningtext-classificationText Classification+1On hyperparameter tuning in general clustering problemsm
Tuning hyperparameters for unsupervised learning problems is difficult in general due to the lack of ground truth for validation. However, the success of most clustering methods depends heavily on the correct choice of t…
ClusteringCommunity DetectionModel SelectionLLM can Achieve Self-Regulation via Hyperparameter Aware Generation
In the realm of Large Language Models (LLMs), users commonly employ diverse decoding strategies and adjust hyperparameters to control the generated text. However, a critical question emerges: Are LLMs conscious of the ex…
Text Generation