The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts

2026-01-21 · Warren Johnson arxiv

In "Compress or Route?" (Johnson, 2026), we found that code generation tolerates aggressive prompt compression (r >= 0.6) while chain-of-thought reasoning degrades gradually. That study was limited to HumanEval (164 problems), left the "perplexity paradox" mechanism unvalidated, and provided no adaptive algorithm. This paper addresses all three gaps. First, we validate across six code benchmarks (HumanEval, MBPP, HumanEval+, MultiPL-E) and four reasoning benchmarks (GSM8K, MATH, ARC-Challenge, MMLU-STEM), confirming the compression threshold generalizes across languages and difficulties. Second, we conduct the first per-token perplexity analysis (n=723 tokens), revealing a "perplexity paradox": code syntax tokens are preserved (high perplexity) while numerical values in math problems are pruned despite being task-critical (low perplexity). Signature injection recovers +34 percentage points in pass rate (5.3% to 39.3%; Cohen's h=0.890). Third, we propose TAAC (Task-Aware Adaptive Compression), achieving 22% cost reduction with 96% quality preservation, outperforming fixed-ratio compression by 7%. MBPP validation (n=1,800 trials) confirms systematic variation: 3.6% at r=0.3 to 54.6% at r=1.0.

📄 PDF Abstract BibTeX arXiv:2602.15843

Code (0)

등록된 구현이 없습니다.

Tasks

Code Generation

Persona-aware Generative Model for Code-mixed Language

2023-09-06 · Ayan Sengupta, Md Shad Akhtar, Tanmoy Chakraborty

Code-mixing and script-mixing are prevalent across online social networks and multilingual societies. However, a user's preference toward code-mixing depends on the socioeconomic status, demographics of the user, and the…

Decoder model valid

Measuring Translationese across Levels of Expertise: Are Professionals more Surprising than Students?

2021-05-01 · NoDaLiDa 2021 5 · Yuri Bizzoni, Ekaterina Lapshinova-Koltunski

The present paper deals with a computational analysis of translationese in professional and student English-to-German translations belonging to different registers. Building upon an information-theoretical approach, we t…

Language Modeling Language Modelling Machine Translation POS+1

Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

2026-01-16 · Lecheng Yan, Ruizhe Li, Guanhua Chen, Qing Li 외 arxiv

Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We …

Reinforcement Learning

AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

2026-04-04 · Archit Thorat arxiv

We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical information, with an NTK-based importance score of…

Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation Models

2026-03-10 · Anurag Mishra arxiv

Time series foundation models (TSFMs) are increasingly deployed in high-stakes domains, yet their internal representations remain opaque. We present the first application of sparse autoencoders (SAEs) to a TSFM, training…