Is this Citation on Point?
In 2023, a New York judge sanctioned two attorneys in Mata v. Avianca for filing a brief with hallucinated citations generated by ChatGPT. Such failures are largely caught by database lookups; the harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered -- a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora, either replacing the cited case or changing only the pinpoint page within the same case. We evaluate fourteen model configurations on the resulting examples. Models catch 93-100% of wrong-case corruptions. They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Scale and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses 40% of pinpoint mismatches on court opinions and 18% on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Online Identification of Time-Varying Systems Using Excitation Sets and Change Point Detection
In this work, we first show that the problem of parameter identification is often ill-conditioned and lacks the persistence of excitation required for the convergence of online learning schemes. To tackle these challenge…
Change Point DetectionMuscle Excitation Estimation in Biomechanical Simulation Using NAF Reinforcement Learning
Motor control is a set of time-varying muscle excitations which generate desired motions for a biomechanical system. Muscle excitations cannot be directly measured from live subjects. An alternative approach is to estima…
Deep Reinforcement LearningPoint Trackingreinforcement-learningReinforcement Learning+1MinScIE: Citation-centered Open Information Extraction
Acknowledging the importance of citations in scientific literature, in this work we present MinScIE, an Open Information Extraction system which provides structured knowledge enriched with semantic information about ci…
Open Information ExtractionDEEP$^2$: Deep Learning Powered De-scattering with Excitation Patterning
Limited throughput is a key challenge in in-vivo deep-tissue imaging using nonlinear optical microscopy. Point scanning multiphoton microscopy, the current gold standard, is slow especially compared to the wide-field ima…
Deep LearningCiMaTe: Citation Count Prediction Effectively Leveraging the Main Text
Prediction of the future citation counts of papers is increasingly important to find interesting papers among an ever-growing number of papers. Although a paper's main text is an important factor for citation count predi…
Prediction