EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpreting
Low-latency simultaneous speech-to-speech translation must keep pace with ongoing speech while preserving key information. To meet these demands, systems use segmentation, reformulation and condensation to reorganize and rephrase information. However, metrics developed for text translation, including BLEU and COMET, may not consistently distinguish faithful adaptations from semantic errors. We propose EviSI, a large language model evaluation agent combining Multidimensional Quality Metrics (MQM) with criteria developed with professional interpreters. Shared source evidence guides assessment across four dimensions: Anchor, Event, Logic and Fluency. Verified errors are deduplicated before deterministic scoring. On human-rated English to Chinese and Chinese to English data, EviSI recovers the aggregate English to Chinese human system ranking. Mean within-dataset Kendall correlations for system rankings reach 0.707 and 0.467, respectively, exceeding evaluated BLEU and COMET baselines. A multilingual extension to five directions without human ratings retains the dimensions and scoring rule, showing positive system ranking correlations with COMET throughout.
Code (0)
등록된 구현이 없습니다.
Tasks
Speech-to-Speech TranslationSimilar Papers 제목 키워드 기반
SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History
Agent skills extend language-model agents with task-specific procedures, scripts, and references, but the tasks and environments they target continually change. Existing methods improve skills in bounded runs and retain …
DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction
Large Language Model (LLM) agents offer a promising approach to attack chain reconstruction by retrieving and interpreting heterogeneous telemetry to infer ordered attacker actions. However, existing benchmarks mainly ev…
Self-Evolving Scientific Agent Designs Physically Reasoned White-Box Fluid Control
While neural networks excel in autonomous control, their black-box nature makes control decisions difficult to interpret and diagnose in dynamic fluids. Here, we show how self-evolving scientific agents can design explic…
Reinforcement LearningPhysical SimulationsCode GenerationInfini Memory: Maintainable Topic Documents for Long-Term LLM Agent Memory
Long-term LLM agents need persistent memory that can track changing facts and provide relevant evidence across sessions. Existing memory systems often store observations as isolated records, summaries, or indexed fragmen…
Beyond Local Surprise: Grounded Dialogue as Selective Belief Revision under Referential Uncertainty
When a speaker refers to a scene that the listener cannot directly see, the listener must decide whether to preserve its current understanding or revise it as new utterances arrive. Many language systems treat local mism…