MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities
Many organizations seek to ensure that machine learning (ML) and artificial intelligence (AI) systems work as intended in production but currently do not have a cohesive methodology in place to do so. To fill this gap, we propose MLTE (Machine Learning Test and Evaluation, colloquially referred to as "melt"), a framework and implementation to evaluate ML models and systems. The framework compiles state-of-the-art evaluation techniques into an organizational process for interdisciplinary teams, including model developers, software engineers, system owners, and other stakeholders. MLTE tooling supports this process by providing a domain-specific language that teams can use to express model requirements, an infrastructure to define, generate, and collect ML evaluation metrics, and the means to communicate results.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DINASTI: Dialogues with a Negotiating Appointment Setting Interface
This paper describes the DINASTI (DIalogues with a Negotiating Appointment SeTting Interface) corpus, which is composed of 1734 dialogues with the French spoken dialogue system NASTIA (Negotiating Appointment SeTting Int…
Dialogue ManagementManagementreinforcement-learningReinforcement Learning+3Artificial collectives of specialists and generalists excel at different tasks
Collective artificial intelligence, where multiple agents work on shared tasks, holds potential to solve expansive problems in fields from medicine to collective governance. But while prescriptive engineering solutions a…
Data Representativity for Machine Learning and AI Systems
Data representativity is crucial when drawing inference from data through machine learning models. Scholars have increased focus on unraveling the bias and fairness in models, also in relation to inherent biases in the i…
BIG-bench Machine LearningExperimental DesignFairnessChargingBoul: A Competitive Negotiating Agent with Novel Opponent Modeling
Automated negotiation has emerged as a critical area of research in multiagent systems, with applications spanning e-commerce, resource allocation, and autonomous decision-making. This paper presents ChargingBoul, a nego…
MOPaC: The Multiple Offers Protocol for Multilateral Negotiations with Partial Consensus
Existing protocols for multilateral negotiation require a full consensus among the negotiating parties. In contrast, we propose a protocol for multilateral negotiation that allows partial consensus, wherein only a subset…