Papers Small Language Model
“Small Language Model” 태그가 달린 논문 109편 · 필터 해제
Embodied CoT Distillation From LLM To Off-the-shelf Agents
We address the challenge of utilizing large language models (LLMs) for complex embodied tasks, in the environment where decision-making systems operate timely on capacity-limited, off-the-shelf devices. We present DeDer,…
Decision MakingIn-Context LearningLanguage ModelingLanguage Modelling+1Small Language Model as Data Prospector for Large Language Model
The quality of instruction data directly affects the performance of fine-tuned Large Language Models (LLMs). Previously, \cite{li2023one} proposed \texttt{NUGGETS}, which identifies and selects high-quality quality data …
Language ModelingLanguage ModellingLarge Language Modelmodel+1JPPO: Joint Power and Prompt Optimization for Accelerated Large Language Model Services
Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, leading to their increasing deployment in wireless networks for a wide variety of user services. However, the growing longer prompt…
Deep Reinforcement LearningLanguage ModelingLanguage ModellingLarge Language Model+1Is Training Data Quality or Quantity More Impactful to Small Language Model Performance?
This study investigates the relative impact of training data quality versus quantity on the performance of small language models (SLMs), utilizing the TinyStories dataset for empirical analysis. Analysis of dataset varia…
Language ModelingLanguage ModellingSmall Language ModelCodeSAM: Source Code Representation Learning by Infusing Self-Attention with Multi-Code-View Graphs
Machine Learning (ML) for software engineering (SE) has gained prominence due to its ability to significantly enhance the performance of various SE applications. This progress is largely attributed to the development of …
Clone DetectionCode SearchRepresentation LearningSmall Language ModelRadPhi-3: Small Language Models for Radiology
LLM based copilot assistants are useful in everyday tasks. There is a proliferation in the exploration of AI assistant use cases to support radiology workflows in a reliable manner. In this work, we present RadPhi-3, a S…
4kLanguage ModelingLanguage ModellingSmall Language ModelPreempting Text Sanitization Utility in Resource-Constrained Privacy-Preserving LLM Interactions
Interactions with online Large Language Models raise privacy issues where providers can gather sensitive information about users and their companies from the prompts. While textual prompts can be sanitized using Differen…
Language ModelingLanguage ModellingPrivacy PreservingSmall Language ModelSlimLM: An Efficient Small Language Model for On-Device Document Assistance
While small language models (SLMs) show promises for mobile deployment, their real-world performance and applications on smartphones remains underexplored. We present SlimLM, a series of SLMs optimized for document assis…
Language ModelingLanguage ModellingQuestion AnsweringSmall Language ModelSecEncoder: Logs are All You Need in Security
Large and Small Language Models (LMs) are typically pretrained using extensive volumes of text, which are sourced from publicly accessible platforms such as Wikipedia, Book Corpus, or through web scraping. These models, …
AllLanguage ModellingSmall Language ModelTowards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model
We present a novel 4.5B parameter small language model that can handle multiple input and output modalities, including text, images, videos, and audio. Despite its small size, the model achieves near state-of-the-art per…
Language ModelingLanguage ModellingMulti-Task LearningSmall Language ModelPhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training
The interest in developing small language models (SLM) for on-device deployment is fast growing. However, the existing SLM design hardly considers the device hardware characteristics. Instead, this work presents a simple…
Language ModelingLanguage ModellingSmall Language ModelWave Network: An Ultra-Small Language Model
We propose an innovative token representation and update method in a new ultra-small language model: the Wave network. Specifically, we use a complex vector to represent each token, encoding both global and local semanti…
Language ModelingLanguage ModellingmodelSmall Language Model+2TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network
The telecommunications industry's rapid evolution demands intelligent systems capable of managing complex networks and adapting to emerging technologies. While large language models (LLMs) show promise in addressing thes…
ChunkingLanguage ModellingRAGRetrieval+2Improving In-Context Learning with Small Language Model Ensembles
Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can …
Domain LabellingIn-Context LearningLanguage ModelingLanguage Modelling+3Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors
The advent of large language models (LLMs) has revolutionized the field of text generation, producing outputs that closely mimic human-like writing. Although academic and industrial institutions have developed detectors …
Reinforcement Learning (RL)Small Language ModelText GenerationA Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such as next-token prediction) over a large …
Language ModelingLanguage ModellingLarge Language ModelSmall Language ModelMethods of improving LLM training stability
Training stability of large language models(LLMs) is an important research topic. Reproducing training instabilities can be costly, so we use a small language model with 830M parameters and experiment with higher learnin…
Small Language ModelSHAKTI: A 2.5 Billion Parameter Small Language Model Optimized for Edge AI and Low-Resource Environments
We introduce Shakti, a 2.5 billion parameter language model specifically optimized for resource-constrained environments such as edge devices, including smartphones, wearables, and IoT systems. Shakti combines high-perfo…
Language ModelingLanguage ModellingQuestion AnsweringSmall Language ModelBridging Large Language Models and Graph Structure Learning Models for Robust Representation Learning
Graph representation learning, involving both node features and graph structures, is crucial for real-world applications but often encounters pervasive noise. State-of-the-art methods typically address noise by focusing …
Graph Representation LearningGraph structure learningRepresentation LearningSmall Language ModelBilinear MLPs enable weight-based mechanistic interpretability
A mechanistic understanding of how MLPs do computation in deep neural networks remains elusive. Current interpretability work can extract features from hidden activations over an input dataset but generally cannot explai…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+1