Advancing Sequential Numerical Prediction in Autoregressive Models
Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces Numerical Token Integrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover's Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL.
Code (1)
Tasks
PredictionSimilar Papers 제목 키워드 기반
Visual Self-Refinement for Autoregressive Models
Autoregressive models excel in sequential modeling and have proven to be effective for vision-language data. However, the spatial nature of visual signals conflicts with the sequential dependencies of next-token predicti…
Comparative Study of Neural Surrogate Architectures for Autoregressive Prediction of Internal Battery States
The Doyle-Fuller-Newman (DFN) model resolves internal electrochemical states in lithium-ion batteries with high fidelity. However, the numerical solution of its governing equations is computationally prohibitive for real…
Autoregressive Generation Strategies for Top-K Sequential Recommendations
The goal of modern sequential recommender systems is often formulated in terms of next-item prediction. In this paper, we explore the applicability of generative transformer-based models for the Top-K sequential recommen…
Recommendation SystemsSequential RecommendationImplicit Stacked Autoregressive Model for Video Prediction
Future frame prediction has been approached through two primary methods: autoregressive and non-autoregressive. Autoregressive methods rely on the Markov assumption and can achieve high accuracy in the early stages of pr…
modelPredictionVideo PredictionWeather ForecastingOn the Efficient Marginalization of Probabilistic Sequence Models
Real-world data often exhibits sequential dependence, across diverse domains such as human behavior, medicine, finance, and climate modeling. Probabilistic methods capture the inherent uncertainty associated with predict…
Point Processes