paper-with-me

홈 › Papers

Benchmark AUC Is Not Deployable Reliability: A Cross-Dataset Audit of Off-the-Shelf Features for Surveillance Video Anomaly Detection

2026-06-28 · Mohammadreza Rashidi arxiv

Automated "suspicious behavior" flagging is a headline promise of AI surveillance, and the field reports high frame-level ROC-AUC on standard video anomaly detection benchmarks. Those numbers are measured by training and testing on the same camera and scene. We audit what happens when that assumption is dropped. We build an unsupervised normality model from the all-normal training frames of one dataset, using frozen off-the-shelf embeddings (CLIP, DINOv2, ResNet-50, EfficientNet-B0) and a nearest-neighbour distance, and score the test frames of the same and of other datasets. Across 4 real datasets (UCSD Ped1, UCSD Ped2, CUHK Avenue, ShanghaiTech) and 4 backbones, same-dataset AUC averages 0.704 but cross-dataset AUC averages 0.499, which is chance: a detector calibrated on one scene is no better than a coin flip on another, and in several pairs it is below chance. The strongest backbone makes this worse, not better: DINOv2 has the best same-dataset AUC (up to 0.901 on Ped2) and the largest cross-dataset drop. The collapse is not an artefact of the scoring rule: replacing the nearest-neighbour detector with a PaDiM-style Mahalanobis detector reproduces it almost exactly (cross-dataset gap 0.202 versus 0.208). Even at a favourable operating point the false-alarm rate is on the order of 31,931 per hour. We conclude that the benchmark numbers quoted for surveillance anomaly detection describe a calibrated laboratory setting and overstate deployable reliability by a wide margin, and we release the code that reproduces every number.

📄 PDF Abstract BibTeX arXiv:2606.29506

Code (0)

등록된 구현이 없습니다.

Tasks

Video Anomaly Detection

Results from the Paper

RankTaskDatasetModelMetrics
#4 Video Anomaly Detection Ped2 Benchmark AUC Is Not Deployable Reliabil AUC: 0.901

Similar Papers 제목 키워드 기반

MICA: Multi-Agent Industrial Coordination Assistant

2025-09-17 · Di Wen, Kunyu Peng, Junwei Zheng, Yufan Chen 외 arxiv

Industrial workflows demand adaptive and trustworthy assistance that can operate under limited computing, connectivity, and strict privacy constraints. In this work, we present MICA (Multi-Agent Industrial Coordination A…

Stopping and Routing LLM Judge Panels

2026-08-20 · Bin Zhu, Yi Xie, Yanghui Rao arxiv

LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and task-specific verifiers. The deployment question is not only which ju…

SCALE-LoRA: Auditing Post-Retrieval LoRA Composition with Residual Merging and View Reliability

2026-05-02 · Shuaipeng Zhou, Yu Zhang arxiv

Libraries of Low-Rank Adaptation (LoRA) adapters are becoming a practical by-product of parameter-efficient adaptation. Once such adapters accumulate, a natural question is no longer how to train one adapter for one task…

Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit

2026-06-30 · Mert Onur Cakiroglu, Zhihe Lu, Mehmet Dalkilic, Hasan Kurban arxiv

AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards, yet most evaluation protocols leave uncontrolled confounds that can inflate reported generalization. As an existen…

The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

2026-06-02 · Wojciech Zarzecki, Jan Dubiński, Sebastian Cygert arxiv

Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment. Statistical tools for detecting training-data membership exist, but have been validated almo…