ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity
Large language models (LLMs) are rapidly acquiring capabilities relevant to biological research, from literature synthesis to interpretation of experimental data. Increasingly, LLM agents can also perform in silico biology tasks that previously required experienced human biologists. These emerging AI capabilities offer new opportunities for scientific discovery and biomedical advances, but they also shift the landscape of biosecurity risks. To address this, we introduce the Agentic Bio-Capabilities Benchmark (ABC-Bench), a suite of tasks to measure agentic biosecurity-relevant capabilities. ABC-Bench evaluates LLM agents on both benign and dual-use biology tasks: writing code to operate liquid handling robots, designing DNA fragments for in vitro assembly, and evading DNA synthesis screening. These tasks require a combination of biology and software expertise. All tested LLM agents outperformed the median expert human baseliner on all three tasks. Agents performed highly on tasks drawing on published knowledge and well-documented protocols, and more weakly on a task requiring novel bioinformatics reasoning. In three wet-lab validation experiments, we found that OpenAI's o4-mini-high produced scripts that, when run on an OpenTrons liquid handling robot, successfully assembled DNA with expected sequences.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Evaluating calibrated refusal and safe usefulness in dual-use biology settings
As AI agents are incorporated into life science workflows, the capabilities that speed discovery might also enable misuse. We present BioSecBench-Refusal, a benchmark for risk identification and refusal behavior for biol…
PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach
Recent advances in Large Language Models (LLMs) have sparked concerns over their potential to acquire and misuse dangerous or high-risk capabilities, posing frontier risks. Current safety evaluations primarily test for w…
Interpretable machine learning applied to on-farm biosecurity and porcine reproductive and respiratory syndrome virus
Effective biosecurity practices in swine production are key in preventing the introduction and dissemination of infectious pathogens. Ideally, biosecurity practices should be chosen by their impact on bio-containment and…
BenchmarkingBIG-bench Machine LearningInterpretable Machine LearningViroBench: Benchmarking Nucleotide Foundation Models on Viral Genomics Tasks
Nucleotide sequences constitute the fundamental genetic basis of biological systems, rendering viral genomic analysis critical for biomedical advancement. Despite progress in biological foundation models, specifically nu…
Biothreat Benchmark Generation Framework for Evaluating Frontier AI Models II: Benchmark Generation Process
The potential for rapidly-evolving frontier artificial intelligence (AI) models, especially large language models (LLMs), to facilitate bioterrorism or access to biological weapons has generated significant policy, acade…
Red Teaming