At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models
Vision-Language multimodal Models (VLMs) offer the possibility for zero-shot classification in astronomy: i.e. classification via natural language prompts, with no training. We investigate two models, GPT-4o and LLaVA-NeXT, for zero-shot classification of low-surface brightness galaxies and artifacts, as well as morphological classification of galaxies. We show that with natural language prompts these models achieved significant accuracy (above 80 percent typically) without additional training/fine tuning. We discuss areas that require improvement, especially for LLaVA-NeXT, which is an open source model. Our findings aim to motivate the astronomical community to consider VLMs as a powerful tool for both research and pedagogy, with the prospect that future custom-built or fine-tuned models could perform better.
Code (0)
등록된 구현이 없습니다.
Tasks
AstronomyClassificationzero-shot-classificationZero-Shot LearningSimilar Papers 제목 키워드 기반
CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical Imaging
Existing vision-text contrastive learning models enhance representation transferability and support zero-shot prediction by matching paired image and caption embeddings while pushing unrelated pairs apart. However, astro…
Contrastive LearningImage-text RetrievalRetrievalText Retrieval+2Simulation-Based Pretraining and Domain Adaptation for Astronomical Time Series with Minimal Labeled Data
Astronomical time-series analysis faces a critical limitation: the scarcity of labeled observational data. We present a pre-training approach that leverages simulations, significantly reducing the need for labeled exampl…
Anomaly DetectionDomain AdaptationZero-Shot Robustification of Zero-Shot Models
Zero-shot inference is a powerful paradigm that enables the use of large pretrained models for downstream classification tasks without further training. However, these models are vulnerable to inherited biases that can i…
Domain-Informed Multi-View Self-Distillation for Astronomical Light-Curve Representation Learning with JEPA
Light curves describe temporal variations in the brightness of celestial objects. Learning robust representations of light curves is essential for large-scale automatic discovery in the dynamic universe, but existing tim…
Representation LearningDomain AdaptationRethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications
Trained on large datasets, deep learning (DL) can accurately classify videos into hundreds of diverse classes. However, video data is expensive to annotate. Zero-shot learning (ZSL) proposes one solution to this problem.…
BenchmarkingGeneral ClassificationVideo ClassificationZero-Shot Action Recognition+1