Exploring Boundary of GPT-4V on Marine Analysis: A Preliminary Case Study

2024-01-04 · Ziqiang Zheng, YiWei Chen, Jipeng Zhang, Tuan-Anh Vu, Huimin Zeng, Yue Him Wong Tim, Sai-Kit Yeung

Large language models (LLMs) have demonstrated a powerful ability to answer various queries as a general-purpose assistant. The continuous multi-modal large language models (MLLM) empower LLMs with the ability to perceive visual signals. The launch of GPT-4 (Generative Pre-trained Transformers) has generated significant interest in the research communities. GPT-4V(ison) has demonstrated significant power in both academia and industry fields, as a focal point in a new artificial intelligence generation. Though significant success was achieved by GPT-4V, exploring MLLMs in domain-specific analysis (e.g., marine analysis) that required domain-specific knowledge and expertise has gained less attention. In this study, we carry out the preliminary and comprehensive case study of utilizing GPT-4V for marine analysis. This report conducts a systematic evaluation of existing GPT-4V, assessing the performance of GPT-4V on marine research and also setting a new standard for future developments in MLLMs. The experimental results of GPT-4V show that the responses generated by GPT-4V are still far away from satisfying the domain-specific requirements of the marine professions. All images and prompts used in this study will be available at https://github.com/hkust-vgd/Marine_GPT-4V_Eval

📄 PDF Abstract BibTeX arXiv:2401.02147

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Marine Variable Linker: Exploring Relations between Changing Variables in Marine Science Literature

2017-04-01 · EACL 2017 4 · Erwin Marsi, Pinar Pinar {\O}zturk, Murat V. Ardelan

We report on a demonstration system for text mining of literature in marine science and related disciplines. It automatically extracts variables ({``}CO2{''}) involved in events of change/increase/decrease ({``}increasin…

Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation

2024-12-03 · Sepand Dyanatkar, Angran Li, Alexander Dungate

Climate change's destruction of marine biodiversity is threatening communities and economies around the world which rely on healthy oceans for their livelihoods. The challenge of applying computer vision to niche, real-w…

RAG Retrieval Retrieval-augmented Generation

MarineGPT: Unlocking Secrets of Ocean to the Public

2023-10-20 · Ziqiang Zheng, Jipeng Zhang, Tuan-Anh Vu, Shizhe Diao 외

Large language models (LLMs), such as ChatGPT/GPT-4, have proven to be powerful tools in promoting the user experience as an AI assistant. The continuous works are proposing multi-modal large language models (MLLM), empo…

Language Modelling

MarineEval: Assessing the Marine Intelligence of Vision-Language Models

2025-12-24 · YuK-Kwan Wong, Tuan-An To, Jipeng Zhang, Ziqiang Zheng 외 arxiv

We have witnessed promising progress led by large language models (LLMs) and further vision language models (VLMs) in handling various queries as a general-purpose assistant. VLMs, as a bridge to connect the visual world…

Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge

2025-05-12 · Chao-Han Huck Yang, Sreyan Ghosh, Qing Wang, Jaeyeon Kim 외

We present Task 5 of the DCASE 2025 Challenge: an Audio Question Answering (AQA) benchmark spanning multiple domains of sound understanding. This task defines three QA subsets (Bioacoustics, Temporal Soundscapes, and Com…

Audio Question Answering Question Answering