xGen-small Technical Report

2025-05-10 · Erik Nijkamp, Bo Pang, Egor Pakhomov, Akash Gokul, Jin Qu, Silvio Savarese, Yingbo Zhou, Caiming Xiong

We introduce xGen-small, a family of 4B and 9B Transformer decoder models optimized for long-context applications. Our vertically integrated pipeline unites domain-balanced, frequency-aware data curation; multi-stage pre-training with quality annealing and length extension to 128k tokens; and targeted post-training via supervised fine-tuning, preference learning, and online reinforcement learning. xGen-small delivers strong performance across various tasks, especially in math and coding domains, while excelling at long context benchmarks.

📄 PDF Abstract BibTeX arXiv:2505.06496

Code (0)

등록된 구현이 없습니다.

Tasks

DecoderMath

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Position-Wise Feed-Forward Layer 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

A Generative Foundation Model for Chest Radiography

2025-09-04 · Yuanfeng Ji, Dan Lin, Xiyue Wang, Lu Zhang 외 arxiv

The scarcity of well-annotated diverse medical images is a major hurdle for developing reliable AI models in healthcare. Substantial technical advances have been made in generative foundation models for natural images. H…

Data Augmentation

XGen-7B Technical Report

2023-09-07 · Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang 외

Large Language Models (LLMs) have become ubiquitous across various domains, transforming the way we interact with information and conduct research. However, most high-performing LLMs remain confined behind proprietary wa…

2k 8k

LexGenie: Automated Generation of Structured Reports for European Court of Human Rights Case Law

2025-03-05 · T. Y. S. S Santosh, Mahmoud Aly, Oana Ichim, Matthias Grabmair

Analyzing large volumes of case law to uncover evolving legal principles, across multiple cases, on a given topic is a demanding task for legal professionals. Structured topical reports provide an effective solution by s…

xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

2024-08-16 · Le Xue, Manli Shu, Anas Awadalla, Jun Wang 외

This report introduces xGen-MM (also known as BLIP-3), a framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resul…

In-Context Learning

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues

2026-07-28 · Takyoung Kim, Kang-wook Kim, Sang Hoon Woo, Julia Hirschberg 외 hf

Turn-taking is a central component of full-duplex interaction. Which turn-taking behaviors are appropriate varies with the scenario, yet current models apply a single norm regardless of context. This limitation originate…