Papers Long-Context Understanding
“Long-Context Understanding” 태그가 달린 논문 134편 · 필터 해제
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
Post-training attention sparsification reduces the quadratic cumulative attention cost of pretrained Transformers by selecting a small set of context units (tokens or blocks) for each query. Existing trainable methods us…
Long-Context UnderstandingSafin-1: Safety from Within through Memory-Native State Evolution
Long-horizon complex tasks require foundation models to accumulate information, maintain internal states, and adapt over extended interactions. Safety should be an intrinsic property of the model itself, rather than a be…
Long-Context UnderstandingTest-time AdaptationOmniAlign: A Unified Multilingual Aligner for Word and Sentence Alignment
Cross-lingual sequence alignment is fundamental for building and exploiting parallel corpora, spanning mappings from documents and sentences down to words and subwords. Existing tools, however, typically specialize in a …
Long-Context UnderstandingSelf-Supervised LearningWord AlignmentMotif 3: Technical Report
We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per to…
Long-Context UnderstandingReinforcement LearningMathematical ReasoningInstruction FollowingK-EXAONE 2.0 Technical Report
This technical report presents K-EXAONE 2.0, an open-weight multilingual foundation model developed by LG AI Research as a step in our effort toward global frontier-scale foundation models. Rather than training from scra…
Long-Context UnderstandingLongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task covera…
Long-Context UnderstandingHyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding
CLIP (Contrastive Language-Image Pre-training) has become a de facto paradigm for image-text alignment, but it struggles with long-context descriptions (>77 tokens) due to absolute positional encoding and pretraining on …
Long-Context UnderstandingCross-Modal RetrievalOpenRCA 2.0: From Outcome Labels to Causal Process Supervision
Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label on…
Long-Context UnderstandingSWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments. Yet current agent benchmarks largely evaluate short-form ta…
Long-Context UnderstandingPersonal AI Agent for Camera Roll VQA
We study the personal camera roll visual question answering setting. In this setting, a conversational AI assistant can access a user's personal camera roll and retrieve relevant photos to answer queries, ranging from si…
Long-Context UnderstandingVisual Question AnsweringGive it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
Positional encoding (PE) underpins how permutation-invariant Transformers represent sequence order, yet how positional information is processed and stored remains poorly understood. Modern PE methods such as RoPE still s…
Long-Context UnderstandingMiA-Signature: Approximating Global Activation for Long-Context Understanding
A growing body of work in cognitive science suggests that reportable conscious access is associated with \emph{global ignition} over distributed memory systems, while such activation is only partially accessible as indiv…
Long-Context UnderstandingOn Stable Long-Form Generation: Benchmarking and Mitigating Length Volatility
Large Language Models (LLMs) excel at long-context understanding but exhibit significant limitations in long-form generation. Existing studies primarily focus on single-generation quality, generally overlooking the volat…
Long-Context UnderstandingFG$^2$-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Linear attention mechanisms have emerged as promising alternatives to softmax attention, offering linear-time complexity during inference. Recent advances such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) have …
Long-Context UnderstandingComputational EfficiencyShuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding, such as retrieval-augmented generation and multi-document reasoning. A common strategy is to fine-tune pretr…
Long-Context UnderstandingMi:dm K 2.5 Pro
The evolving LLM landscape requires capabilities beyond simple text generation, prioritizing multi-step reasoning, long-context understanding, and agentic workflows. This shift challenges existing models in enterprise en…
Long-Context UnderstandingReinforcement LearningText GenerationLookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearl…
Long-Context UnderstandingSabiá-4 Technical Report
This technical report presents Sabiá-4 and Sabiazinho-4, a new generation of Portuguese language models with a focus on Brazilian Portuguese language. The models were developed through a four-stage training pipeline: con…
Long-Context UnderstandingInstruction FollowingAutoFigure-Edit: Generating Editable Scientific Illustration
High-quality scientific illustrations are essential for communicating complex scientific and technical concepts, yet existing automated systems remain limited in editability, stylistic controllability, and efficiency. We…
Long-Context UnderstandingMemory Caching: RNNs with Growing Memory
Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context length. While plausible for retrieval t…
Long-Context Understanding