Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi
The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual PretrainingSimilar Papers 제목 키워드 기반
Low-power Spike-based Wearable Analytics on RRAM Crossbars
This work introduces a spike-based wearable analytics system utilizing Spiking Neural Networks (SNNs) deployed on an In-memory Computing engine based on RRAM crossbars, which are known for their compactness and energy-ef…
Activity RecognitionHuman Activity RecognitionOptimizing Binary and Ternary Neural Network Inference on RRAM Crossbars using CIM-Explorer
Using Resistive Random Access Memory (RRAM) crossbars in Computing-in-Memory (CIM) architectures offers a promising solution to overcome the von Neumann bottleneck. Due to non-idealities like cell variability, RRAM cross…
QuantizationLoR-LUT: Learning Compact 3D Lookup Tables via Low-Rank Residuals
We present LoR-LUT, a unified low-rank formulation for compact and interpretable 3D lookup table (LUT) generation. Unlike conventional 3D-LUT-based techniques that rely on fusion of basis LUTs, which are usually dense te…
Image EnhancementStyle TransferWideband Bandpass Filters Using a Novel Thick Metallization Technology
A new class of wideband bandpass filters based on using thick metallic bars as microwave resonators, instead of common microstrip lines, is presented. These bars provide a series of advantages over fully planar printed t…
Frequency and Bandwidth Design of FR3-Band Acoustic Filters
This article presents an approach to control the operating frequency and fractional bandwidth (FBW) of miniature acoustic filters in thin-film lithium niobate (TFLN). More specifically, we used the first-order antisymmet…