GRIT: General Robust Image Task Benchmark
Computer vision models excel at making predictions when the test distribution closely resembles the training distribution. Such models have yet to match the ability of biological vision to learn from multiple sources and generalize to new data sources and tasks. To facilitate the development and evaluation of more general vision systems, we introduce the General Robust Image Task (GRIT) benchmark. GRIT evaluates the performance, robustness, and calibration of a vision system across a variety of image prediction tasks, concepts, and data sources. The seven tasks in GRIT are selected to cover a range of visual skills: object categorization, object localization, referring expression grounding, visual question answering, segmentation, human keypoint detection, and surface normal estimation. GRIT is carefully designed to enable the evaluation of robustness under image perturbations, image source distribution shift, and concept distribution shift. By providing a unified platform for thorough assessment of skills and concepts learned by a vision model, we hope GRIT catalyzes the development of performant and robust general purpose vision systems.
Code (1)
Tasks
Instance SegmentationKeypoint DetectionObject CategorizationObject LocalizationQuestion AnsweringReferring ExpressionSurface Normal EstimationSurface Normals EstimationVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Generalized Radius and Integrated Codebook Transforms for Differentiable Vector Quantization
Vector quantization (VQ) underpins modern generative and representation models by turning continuous latents into discrete tokens. Yet hard nearest-neighbor assignments are non-differentiable and are typically optimized …
Image ReconstructionImage GenerationNeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles
Outcome-only evaluation under-specifies whether an evaluated agent profile preserves the commitments required to solve a multi-turn task coherently. NeuroState-Bench is a human-calibrated benchmark that operationalizes c…
GRITv2: Efficient and Light-weight Social Relation Recognition
Our research focuses on the analysis and improvement of the Graph-based Relation Inference Transformer (GRIT), which serves as an important benchmark in the field. We conduct a comprehensive ablation study using the PISC…
Visual Social Relationship RecognitionData Pruning via Separability, Integrity, and Model Uncertainty-Aware Importance Sampling
This paper improves upon existing data pruning methods for image classification by introducing a novel pruning metric and pruning procedure based on importance sampling. The proposed pruning metric explicitly accounts fo…
Classificationimage-classificationImage ClassificationGRIT: Faster and Better Image captioning Transformer Using Dual Visual Features
Current state-of-the-art methods for image captioning employ region-based features, as they provide object-level information that is essential to describe the content of images; they are usually extracted by an object de…
Image Captioning