Analyzing Social Biases in Japanese Large Language Models
With the development of Large Language Models (LLMs), social biases in the LLMs have become a crucial issue. While various benchmarks for social biases have been provided across languages, the extent to which Japanese LLMs exhibit social biases has not been fully investigated. In this study, we construct the Japanese Bias Benchmark dataset for Question Answering (JBBQ) based on the English bias benchmark BBQ, and analyze social biases in Japanese LLMs. The results show that while current open Japanese LLMs improve their accuracies on JBBQ by setting larger parameters, their bias scores become larger. In addition, prompts with warnings about social biases and Chain-of-Thought prompting reduce the effect of biases in model outputs, but there is room for improvement in the consistency of reasoning.
Code (1)
Tasks
Question AnsweringSimilar Papers 제목 키워드 기반
Can Existing Methods Debias Languages Other than English? First Attempt to Analyze and Mitigate Japanese Word Embeddings
It is known that word embeddings exhibit biases inherited from the corpus, and those biases reflect social stereotypes. Recently, many studies have been conducted to analyze and mitigate biases in word embeddings. Unsupe…
Word EmbeddingsAnalyzing the Safety of Japanese Large Language Models in Stereotype-Triggering Prompts
In recent years, Large Language Models have attracted growing interest for their significant potential, though concerns have rapidly emerged regarding unsafe behaviors stemming from inherent stereotypes and biases. Most …
EthicsJUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs
Social biases reflected in language are inherently shaped by cultural norms, which vary significantly across regions and lead to diverse manifestations of stereotypes. Existing evaluations of social bias in large languag…
A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory
In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on …
Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset
Large language models (LLMs) exhibit social biases, prompting the development of various debiasing methods. However, debiasing methods may degrade the capabilities of LLMs. Previous research has evaluated the impact of b…