paper-with-me

홈 › Papers

Test Harder than You Train: Probing with Extrapolation Splits

2021-11-01 · EMNLP (BlackboxNLP) 2021 11 · Jenny Kunz, Marco Kuhlmann

Previous work on probing word representations for linguistic knowledge has focused on interpolation tasks. In this paper, we instead analyse probes in an extrapolation setting, where the inputs at test time are deliberately chosen to be ‘harder’ than the training examples. We argue that such an analysis can shed further light on the open question whether probes actually decode linguistic knowledge, or merely learn the diagnostic task from shallow features. To quantify the hardness of an example, we consider scoring functions based on linguistic, statistical, and learning-related criteria, all of which are applicable to a broad range of NLP tasks. We discuss the relative merits of these criteria in the context of two syntactic probing tasks, part-of-speech tagging and syntactic dependency labelling. From our theoretical and experimental analysis, we conclude that distance-based and hard statistical criteria show the clearest differences between interpolation and extrapolation settings, while at the same time being transparent, intuitive, and easy to control.

📄 PDF Abstract BibTeX

Code (1)

jekunz/extrapolation 공식 구현 pytorch

Tasks

DiagnosticOpen-Ended Question AnsweringPart-Of-Speech Tagging

Similar Papers 제목 키워드 기반

An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization

2021-02-25 · Elan Rosenfeld, Pradeep Ravikumar, Andrej Risteski

A popular assumption for out-of-distribution generalization is that the training data comprises sub-datasets, each drawn from a distinct distribution; the goal is then to "interpolate" these distributions and "extrapolat…

Domain GeneralizationOut-of-Distribution Generalization

Simulated Human Learning in a Dynamic, Partially-Observed, Time-Series Environment

2025-11-19 · Jeffrey Jiang, Kevin Hong, Emily Kuczynski, Gregory Pottie arxiv

While intelligent tutoring systems (ITSs) can use information from past students to personalize instruction, each new student is unique. Moreover, the education problem is inherently difficult because the learning proces…

Reinforcement Learning

Pareto Probing: Trading Off Accuracy for Complexity

2020-10-05 · EMNLP 2020 11 · Tiago Pimentel, Naomi Saphra, Adina Williams, Ryan Cotterell

The question of how to probe contextual word representations for linguistic structure in a way that is both principled and useful has seen significant attention recently in the NLP literature. In our contribution to this…

ARCDependency Parsing

Disentangled Latent Dynamics Manifold Fusion for Solving Parameterized PDEs

2026-03-13 · Zhangyong Liang, Huanhuan Gao arxiv

Generalizing neural surrogate models across different PDE parameters remains difficult because changes in PDE coefficients often make learning harder and optimization less stable. The problem becomes even more severe whe…

Scaling Scaling Laws with Board Games

2021-04-07 · Andy L. Jones

The largest experiments in machine learning now require resources far beyond the budget of all but a few institutions. Fortunately, it has recently been shown that the results of these huge experiments can often be extra…

Board Games