Calibrated Structured Prediction
In user-facing applications, displaying calibrated confidence measures---probabilities that correspond to true frequency---can be as important as obtaining high accuracy. We are interested in calibration for structured prediction problems such as speech recognition, optical character recognition, and medical diagnosis. Structured prediction presents new challenges for calibration: the output space is large, and users may issue many types of probability queries (e.g., marginals) on the structured output. We extend the notion of calibration so as to handle various subtleties pertaining to the structured setting, and then provide a simple recalibration method that trains a binary classifier to predict probabilities of interest. We explore a range of features appropriate for structured recalibration, and demonstrate their efficacy on three real-world datasets.
Code (1)
Tasks
Medical DiagnosisOptical Character RecognitionOptical Character Recognition (OCR)Predictionspeech-recognitionSpeech RecognitionStructured PredictionSimilar Papers 제목 키워드 기반
Ensemble Distillation for Structured Prediction: Calibrated, Accurate, Fast-Choose Three
Modern neural networks do not always produce well-calibrated predictions, even when trained with a proper scoring function such as cross-entropy. In classification settings, simple methods such as isotonic regression or …
Machine Translationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+2A Pedestrian-Vehicle Interaction Benchmark and Annotation Framework for Unstructured Scenes via Uncalibrated Cameras
Predicting the interaction between pedestrian and vehicle is essential for autonomous driving safety in unstructured and semi-structured scenarios; however, this task is severely hindered by the scarcity of public datase…
Trajectory PredictionAutonomous DrivingOn Structured Prediction Theory with Calibrated Convex Surrogate Losses
We provide novel theoretical insights on structured prediction in the context of efficient convex surrogate loss minimization with consistency guarantees. For any task loss, we construct a convex surrogate that can be op…
PredictionStructured PredictionAudio Question Answering with GRPO-Based Fine-Tuning and Calibrated Segment-Level Predictions
In this report, we describe our submission to Track 5 of the DCASE 2025 Challenge for the task of Audio Question Answering(AQA). Our system leverages the SSL backbone BEATs to extract frame-level audio features, which ar…
Question AnsweringMulti-hypothesis 3D human pose estimation metrics favor miscalibrated distributions
Due to depth ambiguities and occlusions, lifting 2D poses to 3D is a highly ill-posed problem. Well-calibrated distributions of possible poses can make these ambiguities explicit and preserve the resulting uncertainty fo…
3D Human Pose EstimationDensity EstimationMulti-Hypotheses 3D Human Pose EstimationPose Estimation