Certifying One-Phase Technology-Assisted Reviews
Technology-assisted review (TAR) workflows based on iterative active learning are widely used in document review applications. Most stopping rules for one-phase TAR workflows lack valid statistical guarantees, which has discouraged their use in some legal contexts. Drawing on the theory of quantile estimation, we provide the first broadly applicable and statistically valid sample-based stopping rules for one-phase TAR. We further show theoretically and empirically that overshooting a recall target, which has been treated as innocuous or desirable in past evaluations of stopping rules, is a major source of excess cost in one-phase TAR workflows. Counterintuitively, incurring a larger sampling cost to reduce excess recall leads to lower total cost in almost all scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningTARvalidSimilar Papers 제목 키워드 기반
Phase Shift Information Compression in IRS-aided Wireless Systems: Challenges and Opportunities
Intelligent reflecting surfaces (IRS) have emerged as a promising technology for future 6G wireless networks, offering programmable control of the wireless environment by adjusting the phase shifts of reflecting elements…
Meta-LearningReconfigurable Intelligent Surface Assisted Railway Communications: A survey
The number of train passengers and the demand for high data rates to handle new technologies such as video streaming and IoT technologies are continuously increasing. Therefore the exploration of millimeter waves (mmWave…
BlockingSurveyMulti-Modal Artificial Intelligence of Embryo Grading and Pregnancy Prediction in Assisted Reproductive Technology: A Review
As a global disease, infertility has always affected human beings. The development of assisted reproductive technology can effectively solve this disease. However, the traditional in vitro fertilization-embryo transfer t…
Stopping Methods for Technology Assisted Reviews based on Point Processes
Technology Assisted Review (TAR), which aims to reduce the effort required to screen collections of documents for relevance, is used to develop systematic reviews of medical evidence and identify documents that must be d…
Point ProcessesTARDo LLMs Favor LLMs? Quantifying Interaction Effects in Peer Review
There are increasing indications that LLMs are not only used for producing scientific papers, but also as part of the peer review process. In this work, we provide the first comprehensive analysis of LLM use across the p…