iSafetyBench: A video-language benchmark for safety in industrial environment

2025-08-01 · Raiyaan Abdullah, Yogesh Singh Rawat, Shruti Vyas arxiv

Recent advances in vision-language models (VLMs) have enabled impressive generalization across diverse video understanding tasks under zero-shot settings. However, their capabilities in high-stakes industrial domains-where recognizing both routine operations and safety-critical anomalies is essential-remain largely underexplored. To address this gap, we introduce iSafetyBench, a new video-language benchmark specifically designed to evaluate model performance in industrial environments across both normal and hazardous scenarios. iSafetyBench comprises 1,100 video clips sourced from real-world industrial settings, annotated with open-vocabulary, multi-label action tags spanning 98 routine and 67 hazardous action categories. Each clip is paired with multiple-choice questions for both single-label and multi-label evaluation, enabling fine-grained assessment of VLMs in both standard and safety-critical contexts. We evaluate eight state-of-the-art video-language models under zero-shot conditions. Despite their strong performance on existing video benchmarks, these models struggle with iSafetyBench-particularly in recognizing hazardous activities and in multi-label scenarios. Our results reveal significant performance gaps, underscoring the need for more robust, safety-aware multimodal models for industrial applications. iSafetyBench provides a first-of-its-kind testbed to drive progress in this direction. The dataset is available at: https://github.com/iSafetyBench/data.

📄 PDF Abstract BibTeX arXiv:2508.00399

Code (0)

등록된 구현이 없습니다.

ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

2026-03-05 · Trapoom Ukarapol, Nut Chukamphaeng, Kunat Pipatanakul, Pakhapoom Sarapat arxiv

The safety evaluation of large language models (LLMs) remains largely centered on English, leaving non-English languages and culturally grounded risks underexplored. In this work, we investigate LLM safety in the context…

CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Models

2024-06-14 · Wenjing Zhang, Xuejiao Lei, Zhaoxiang Liu, Meijuan An 외

With the profound development of large language models(LLMs), their safety concerns have garnered increasing attention. However, there is a scarcity of Chinese safety benchmarks for LLMs, and the existing safety taxonomi…

Multiple-choice Question Answering

AISafetyBenchExplorer: A Metric-Aware Catalogue of AI Safety Benchmarks Reveals Fragmented Measurement and Weak Benchmark Governance

2026-04-14 · Abiodun A. Solanke arxiv

The rapid expansion of large language model (LLM) safety evaluation has produced a substantial benchmark ecosystem, but not a correspondingly coherent measurement ecosystem. We present AISafetyBenchExplorer, a structured…

Evaluating Implicit Regulatory Compliance in LLM Tool Invocation via Logic-Guided Synthesis

2026-01-13 · Da Song, Yuheng Huang, Boqi Chen, Tianshuo Cong 외 arxiv

The integration of large language models (LLMs) into autonomous agents has enabled complex tool use, yet in high-stakes domains, these systems must strictly adhere to regulatory standards beyond simple functional correct…

SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments

2026-07-06 · Suryanarayana Reddy Yarrabothula, Manisha Chawla, Kunal Sinha, Gagan Raj Gupta 외 arxiv

Existing video benchmarks evaluate action recognition on consumer videos, egocentric recordings, or simulated industrial environments. They do not test vision-language models under the visual and procedural conditions of…

Activity Recognition Action Recognition