Metrics vs Surveys: An Analysis for Human-Aligned Benchmarking in Social Robot Navigation
Social, also called human-aware, navigation is a key challenge for integrating mobile robots into human environments. The evaluation of such systems is complex, as factors such as comfort, safety, and legibility must be considered. Human-centered assessments, typically conducted through surveys, provide reliable insights but are costly, resource-intensive, and difficult to reproduce or compare across systems. Alternatively, numerical social navigation metrics are easy to compute and facilitate comparisons, yet the community lacks consensus on a standard set of metrics. This work explores the relationship between numerical metrics and human-centered evaluations to identify potential correlations. If specific quantitative measures align with human perceptions, they could serve as preliminary benchmarking tools, providing a human-aligned assessment when large-scale surveys are not feasible. Our results indicate that while current metrics capture some aspects of robot navigation behavior, important subjective factors remain insufficiently represented, necessitating new metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
Robot NavigationSimilar Papers 제목 키워드 기반
A short methodological review on social robot navigation benchmarking
Social Robot Navigation is the skill that allows robots to move efficiently in human-populated environments while ensuring safety, comfort, and trust. Unlike other areas of research, the scientific community has not yet …
Robot NavigationBenchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics
This paper surveys benchmarking principles, machine learning devices including GPUs, FPGAs, and ASICs, and deep learning software frameworks. It also reviews these technologies with respect to benchmarking from the persp…
BenchmarkingBIG-bench Machine LearningDeep LearningVizSeq: A Visual Analysis Toolkit for Text Generation Tasks
Automatic evaluation of text generation tasks (e.g. machine translation, text summarization, image captioning and video description) usually relies heavily on task-specific metrics, such as BLEU and ROUGE. They, however,…
BenchmarkingImage CaptioningMachine TranslationText Generation+3Enhancing Trust in LLMs: Algorithms for Comparing and Interpreting LLMs
This paper surveys evaluation techniques to enhance the trustworthiness and understanding of Large Language Models (LLMs). As reliance on LLMs grows, ensuring their reliability, fairness, and transparency is crucial. We …
BenchmarkingFairnessFew-Shot LearningHallucination+1SVGauge: Towards Human-Aligned Evaluation for SVG Generation
Generated Scalable Vector Graphics (SVG) images demand evaluation criteria tuned to their symbolic and vectorial nature: criteria that existing metrics such as FID, LPIPS, or CLIPScore fail to satisfy. In this paper, we …