Test Harder than You Train: Probing with Extrapolation Splits
Previous work on probing word representations for linguistic knowledge has focused on interpolation tasks. In this paper, we instead analyse probes in an extrapolation setting, where the inputs at test time are deliberately chosen to be ‘harder’ than the training examples. We argue that such an analysis can shed further light on the open question whether probes actually decode linguistic knowledge, or merely learn the diagnostic task from shallow features. To quantify the hardness of an example, we consider scoring functions based on linguistic, statistical, and learning-related criteria, all of which are applicable to a broad range of NLP tasks. We discuss the relative merits of these criteria in the context of two syntactic probing tasks, part-of-speech tagging and syntactic dependency labelling. From our theoretical and experimental analysis, we conclude that distance-based and hard statistical criteria show the clearest differences between interpolation and extrapolation settings, while at the same time being transparent, intuitive, and easy to control.
Code (1)
Tasks
DiagnosticOpen-Ended Question AnsweringPart-Of-Speech TaggingSimilar Papers 제목 키워드 기반
An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization
A popular assumption for out-of-distribution generalization is that the training data comprises sub-datasets, each drawn from a distinct distribution; the goal is then to "interpolate" these distributions and "extrapolat…
Domain GeneralizationOut-of-Distribution GeneralizationSimulated Human Learning in a Dynamic, Partially-Observed, Time-Series Environment
While intelligent tutoring systems (ITSs) can use information from past students to personalize instruction, each new student is unique. Moreover, the education problem is inherently difficult because the learning proces…
Reinforcement LearningPareto Probing: Trading Off Accuracy for Complexity
The question of how to probe contextual word representations for linguistic structure in a way that is both principled and useful has seen significant attention recently in the NLP literature. In our contribution to this…
ARCDependency ParsingDisentangled Latent Dynamics Manifold Fusion for Solving Parameterized PDEs
Generalizing neural surrogate models across different PDE parameters remains difficult because changes in PDE coefficients often make learning harder and optimization less stable. The problem becomes even more severe whe…
Scaling Scaling Laws with Board Games
The largest experiments in machine learning now require resources far beyond the budget of all but a few institutions. Fortunately, it has recently been shown that the results of these huge experiments can often be extra…
Board Games