The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?
In this work, we explore the use and reliability of Large Language Models (LLMs) in musicology. From a discussion with experts and students, we assess the current acceptance and concerns regarding this, nowadays ubiquitous, technology. We aim to go one step further, proposing a semi-automatic method to create an initial benchmark using retrieval-augmented generation models and multiple-choice question generation, validated by human experts. Our evaluation on 400 human-validated questions shows that current vanilla LLMs are less reliable than retrieval augmented generation from music dictionaries. This paper suggests that the potential of LLMs in musicology requires musicology driven research that can specialized LLMs by including accurate and reliable domain knowledge.
Code (0)
등록된 구현이 없습니다.
Tasks
Multiple-choiceQuestion GenerationQuestion-GenerationRetrievalRetrieval-augmented GenerationSimilar Papers 제목 키워드 기반
Machine listening intelligence
This manifesto paper will introduce machine listening intelligence, an integrated research framework for acoustic and musical signals modelling, based on signal processing, deep learning and computational musicology.
Deep LearningThe Cycle of Trust and Responsibility in Outsourced AI
Artificial Intelligence (AI) and Machine Learning (ML) are rapidly becoming must-have capabilities. According to a 2019 Forbes Insights Report, “seventy-nine percent [of executives] agree that AI is already having a tran…
EthicsTowards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models
Ensuring the trustworthiness of large language models (LLMs) is crucial. Most studies concentrate on fully pre-trained LLMs to better understand and improve LLMs' trustworthiness. In this paper, to reveal the untapped po…
FairnessMutual Information EstimationHow to Measure the Intelligence of Large Language Models?
With the release of ChatGPT and other large language models (LLMs) the discussion about the intelligence, possibilities, and risks, of current and future models have seen large attention. This discussion included much de…
ArticlesLove, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams
Romance-baiting scams have become a major source of financial and emotional harm worldwide. These operations are run by organized crime syndicates that traffic thousands of people into forced labor, requiring them to bui…