Towards a Benchmark for Scientific Understanding in Humans and Machines
Scientific understanding is a fundamental goal of science, allowing us to explain the world. There is currently no good way to measure the scientific understanding of agents, whether these be humans or Artificial Intelligence systems. Without a clear benchmark, it is challenging to evaluate and compare different levels of and approaches to scientific understanding. In this Roadmap, we propose a framework to create a benchmark for scientific understanding, utilizing tools from philosophy of science. We adopt a behavioral notion according to which genuine understanding should be recognized as an ability to perform certain tasks. We extend this notion by considering a set of questions that can gauge different levels of scientific understanding, covering information retrieval, the capability to arrange information to produce an explanation, and the ability to infer how things would be different under different circumstances. The Scientific Understanding Benchmark (SUB), which is formed by a set of these tests, allows for the evaluation and comparison of different approaches. Benchmarking plays a crucial role in establishing trust, ensuring quality control, and providing a basis for performance evaluation. By aligning machine and human scientific understanding we can improve their utility, ultimately advancing scientific understanding and helping to discover new insights within machines.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingInformation RetrievalPhilosophyRetrievalSimilar Papers 제목 키워드 기반
Human vs. supervised machine learning: Who learns patterns faster?
The capabilities of supervised machine learning (SML), especially compared to human abilities, are being discussed in scientific research and in the usage of SML. This study provides an answer to how learning performance…
BIG-bench Machine LearningMinds, Brains, AI
In the last year or so and going back many decades there has been extensive claims by major computational scientists, engineers, and others that AGI, artificial general intelligence, is five or ten years away, but withou…
Self-Driving CarsSmarnet: Teaching Machines to Read and Comprehend Like Human
Machine Comprehension (MC) is a challenging task in Natural Language Processing field, which aims to guide the machine to comprehend a passage and answer the given question. Many existing approaches on MC task are suffer…
Question AnsweringReading ComprehensionTriviaQAMEWL: Few-shot multimodal word learning with referential uncertainty
Without explicit feedback, humans can rapidly learn the meaning of words. Children can acquire a new word after just a few passive exposures, a process known as fast mapping. This word learning capability is believed to …
Do humans and machines have the same eyes? Human-machine perceptual differences on image classification
Trained computer vision models are assumed to solve vision tasks by imitating human behavior learned from training labels. Most efforts in recent vision research focus on measuring the model task performance using standa…
image-classificationImage Classification