MIRB: Mathematical Information Retrieval Benchmark
Mathematical Information Retrieval (MIR) is the task of retrieving information from mathematical documents and plays a key role in various applications, including theorem search in mathematical libraries, answer retrieval on math forums, and premise selection in automated theorem proving. However, a unified benchmark for evaluating these diverse retrieval tasks has been lacking. In this paper, we introduce MIRB (Mathematical Information Retrieval Benchmark) to assess the MIR capabilities of retrieval models. MIRB includes four tasks: semantic statement retrieval, question-answer retrieval, premise retrieval, and formula retrieval, spanning a total of 12 datasets. We evaluate 13 retrieval models on this benchmark and analyze the challenges inherent to MIR. We hope that MIRB provides a comprehensive framework for evaluating MIR systems and helps advance the development of more effective retrieval models tailored to the mathematical domain.
Code (1)
Tasks
Automated Theorem ProvingInformation RetrievalMathRetrievalSimilar Papers 제목 키워드 기반
AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels
Medical information retrieval (MIR) is essential for retrieving relevant medical knowledge from diverse sources, including electronic health records, scientific literature, and medical databases. However, achieving effec…
BenchmarkingInformation RetrievalRetrievalSelf-LearningMirBot: A collaborative object recognition system for smartphones using convolutional neural networks
MirBot is a collaborative application for smartphones that allows users to perform object recognition. This app can be used to take a photograph of an object, select the region of interest and obtain the most likely clas…
image-classificationImage ClassificationObjectObject Recognition+1Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
The advancement of large language models (LLMs) has significantly broadened the scope of applications in natural language processing, with multi-modal LLMs extending these capabilities to integrate and interpret visual d…
BenchmarkingWorld KnowledgeImplementing result-based agri-environmental payments by means of modelling
From a theoretical point of view, result-based agri-environmental payments are clearly preferable to action-based payments. However, they suffer from two major practical disadvantages: costs of measuring the results and …
ManagementSABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right re…
Information Retrieval