CitaLaw: Enhancing LLM with Citations in Legal Domain
In this paper, we propose CitaLaw, the first benchmark designed to evaluate LLMs' ability to produce legally sound responses with appropriate citations. CitaLaw features a diverse set of legal questions for both laypersons and practitioners, paired with a comprehensive corpus of law articles and precedent cases as a reference pool. This framework enables LLM-based systems to retrieve supporting citations from the reference corpus and align these citations with the corresponding sentences in their responses. Moreover, we introduce syllogism-inspired evaluation methods to assess the legal alignment between retrieved references and LLM-generated responses, as well as their consistency with user questions. Extensive experiments on 2 open-domain and 7 legal-specific LLMs demonstrate that integrating legal references substantially enhances response quality. Furthermore, our proposed syllogism-based evaluation method exhibits strong agreement with human judgments.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
Retrieval-Augmented Generation (RAG) systems are showing promising potential, and are becoming increasingly relevant in AI-powered legal applications. Existing benchmarks, such as LegalBench, assess the generative capabi…
RAGRetrievalRetrieval-augmented GenerationCan LLMs Time Travel? Enhancing Temporal Consistency in Legal Agentic Search through Reinforcement Learning
While large language models (LLMs) augmented with agentic search capabilities show promise for legal reasoning, they overlook a fundamental constraint that applicable law must match the temporal context of each case, as …
Reinforcement LearningDomain GeneralizationLegal ReasoningMaat: The Agentic Legal Research Assistant for Competition Protection
Competition law experts conducting legal research must review extensive volumes of cases, decisions, and judicial reports to identify precedents and assess key elements in competition and merger cases. Although general r…
CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generation
Legal professionals need to write analyses that rely on citations to relevant precedents, i.e., previous case decisions. Intelligent systems assisting legal professionals in writing such documents provide great benefits …
Information RetrievalRAGRetrievalRetrieval-augmented GenerationPoliLegalLM: A Technical Report on a Large Language Model for Political and Legal Affairs
Large language models (LLMs) have achieved remarkable success in general-domain tasks, yet their direct application to the legal domain remains challenging due to hallucinated legal citations, incomplete knowledge covera…
Reinforcement Learning