Randomized Controlled Trials for Security Copilot for IT Administrators
As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft's Security Copilot ("Copilot") on information technology administrators ("IT admins") through randomized controlled trials. Participants were divided into treatment and control groups, with the former granted access to Copilot within Microsoft's Entra and Intune admin centers. Across three IT admin scenarios - sign-in troubleshooting, device policy management, and device troubleshooting - Copilot users demonstrated substantial improvements in both accuracy and speed. Across all scenarios and tasks, Copilot subjects experienced a 34.53% improvement in overall accuracy and a 29.79% reduction in task completion time. We also find that the productivity benefits vary by task type, with more complex tasks showing greater improvement. In free response tasks, Copilot users identified 146.07% more relevant facts and reduced task completion time by 61.14%. Subject satisfaction with Copilot was high, with participants reporting reduced effort and a strong preference for using the tool in future tasks. These findings suggest that GAI tools like Copilot can significantly enhance the productivity and efficiency of IT admins, especially in scenarios requiring information synthesis and complex decision-making.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingSimilar Papers 제목 키워드 기반
Randomized Controlled Trials for Phishing Triage Agent
Security operations centers (SOCs) face a persistent challenge: efficiently triaging a high volume of user-reported phishing emails while maintaining robust protection against threats. This paper presents the first rando…
Randomized Controlled Trials for Conditional Access Optimization Agent
AI agents are increasingly deployed to automate complex enterprise workflows, yet evidence of their effectiveness in identity governance is limited. We report results from the first randomized controlled trial (RCT) eval…
Machine Learning Assisted Adjustment Boosts Efficiency of Exact Inference in Randomized Controlled Trials
In this work, we proposed a novel inferential procedure assisted by machine learning based adjustment for randomized control trials. The method was developed under the Rosenbaum's framework of exact tests in randomized e…
Comparison of Methods that Combine Multiple Randomized Trials to Estimate Heterogeneous Treatment Effects
Individualized treatment decisions can improve health outcomes, but using data to make these decisions in a reliable, precise, and generalizable way is challenging with a single dataset. Leveraging multiple randomized co…
Optimality of Matched-Pair Designs in Randomized Controlled Trials
In randomized controlled trials (RCTs), treatment is often assigned by stratified randomization. I show that among all stratified randomization schemes which treat all units with probability one half, a certain matched-p…