L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Context

2024-09-13 · Pritika Rohera, Chaitrali Ginimav, Akanksha Salunke, Gayatri Sawant, Raviraj Joshi

Large Language Models (LLMs) have made significant progress in incorporating Indic languages within multilingual models. However, it is crucial to quantitatively assess whether these languages perform comparably to globally dominant ones, such as English. Currently, there is a lack of benchmark datasets specifically designed to evaluate the regional knowledge of LLMs in various Indic languages. In this paper, we present the L3Cube-IndicQuest, a gold-standard factual question-answering benchmark dataset designed to evaluate how well multilingual LLMs capture regional knowledge across various Indic languages. The dataset contains 200 question-answer pairs, each for English and 19 Indic languages, covering five domains specific to the Indic region. We aim for this dataset to serve as a benchmark, providing ground truth for evaluating the performance of LLMs in understanding and representing knowledge relevant to the Indian context. The IndicQuest can be used for both reference-based evaluation and LLM-as-a-judge evaluation. The dataset is shared publicly at https://github.com/l3cube-pune/indic-nlp .

📄 PDF Abstract BibTeX arXiv:2409.08706

Code (1)

l3cube-pune/indic-nlp 공식 구현

Tasks

Question Answering

TaCube: Pre-computing Data Cubes for Answering Numerical-Reasoning Questions over Tabular Data

2022-05-25 · Fan Zhou, Mengkang Hu, Haoyu Dong, Zhoujun Cheng 외

Existing auto-regressive pre-trained language models (PLMs) like T5 and BART, have been well applied to table question answering by UNIFIEDSKG and TAPEX, respectively, and demonstrated state-of-the-art results on multipl…

Question Answering

Language-Image Models with 3D Understanding

2024-05-06 · Jang Hyun Cho, Boris Ivanovic, Yulong Cao, Edward Schmerling 외

Multi-modal large language models (MLLMs) have shown incredible capabilities in a variety of 2D vision and language tasks. We extend MLLMs' perceptual capabilities to ground and reason about images in 3-dimensional space…

Question Answering Visual Question Answering

Hypercube-RAG: Hypercube-Based Retrieval-Augmented Generation for In-domain Scientific Question-Answering

2025-05-25 · Jimeng Shi, Sizhe Zhou, Bowen Jin, Wei Hu 외

Large language models (LLMs) often need to incorporate external knowledge to solve theme-specific problems. Retrieval-augmented generation (RAG), which empowers LLMs to generate more qualified responses with retrieved ex…

Question Answering RAG Retrieval Retrieval-augmented Generation+2

Visual Question Answering on 360° Images

2020-01-10 · Shih-Han Chou, Wei-Lun Chao, Wei-Sheng Lai, Min Sun 외

In this work, we introduce VQA 360, a novel task of visual question answering on 360 images. Unlike a normal field-of-view image, a 360 image captures the entire visual content around the optical center of a camera, dema…

Question Answering Visual Question Answering Visual Question Answering (VQA)

MultiCube-RAG for Multi-hop Question Answering

2026-02-11 · Jimeng Shi, Wei Hu, Runchu Tian, Bowen Jin 외 arxiv

Multi-hop question answering (QA) necessitates multi-step reasoning and retrieval across interconnected subjects, attributes, and relations. Existing retrieval-augmented generation (RAG) methods struggle to capture these…

Multi-hop Question Answering