AI Chatbots Know How to Make Biological Weapons. Some Will Teach You.
AI companies play a cat-and-ouse game, trying to boost the capabilities of their creations while scrambling to block answers to dangerous queries
By Georgia Wells
After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.
The models, operating largely free of U.S. government restrictions, patiently answered, issuing step-by-step instructions described by company employees as simple enough for a high-school biology student to follow. Biology and terrorism experts later reviewed the exchanges for ChatGPT and judged some as deadly accurate, said people familiar with the matter. Most queries were about concocting poisons, according to OpenAI.
OpenAI banned these and other user accounts that asked about how to make poisons and biological weapons, but it didn’t alert law-enforcement officials. The U.S. has no federal laws requiring AI companies to either restrict or disclose queries about making weapons or formulating plans that pose a safety threat.
The absence of mandated safeguards coincides with a rise in queries from users asking AI models how to kill en masse—and chatbots responding with credible plans for mass-casualty attacks, according to current and former employees at the major AI labs, including OpenAI, as well as policy advisers and researchers who study biological weapons.
AI’s advancing proficiencies—especially for crafting nefarious biological and cyber-based plans—are alarming senior executives at AI companies and the White House. The world’s leading AI labs are grappling with how to prevent or mitigate damage by malevolent users without hampering well-intentioned research queries.
Senior White House and Defense Department officials have discussed the threat of AI becoming a how-to guide for biological weapons since the Biden era, according to Trump administration officials. The release of powerful new AI models has since prompted the administration to shift from a hands-off approach to increased oversight.
The Commerce Department recently restricted foreign use of two Anthropic models, prompting the company to shut down all access to them. The agency lifted its restrictions after the company said it had addressed workarounds that let users evade safeguards.
OpenAI, Anthropic and other AI companies say they are working closely with the administration on the release of models and have banned users or restricted accounts showing suspicious activity.
Yet no federal rules require AI companies to report users asking models for advice on how to injure or kill. Some lawmakers have proposed such legislation, and a few states have passed rules. But concerns about privacy and the potential impact on the industry’s growth have generally kept such notifications voluntary.
Federal laws bar chatbots from producing child sexual-abuse material, and AI companies must follow the same general rules for consumer protection and privacy as other businesses. Beyond that, it is generally up to AI executives to decide where to draw the line.
Chatbots have already sparked concerns over help to users plotting attacks with firearms. The dispensing of instructions for biological weapons gives savvy users the tools to kill people on a much larger scale.
OpenAI trains its models to refuse requests for instructions, tactics or planning that could harm people, an OpenAI spokeswoman said. The company runs safety evaluations for all models before release, she said, and can identify and disrupt attempts to use its models to obtain harmful biological information. ChatGPT receives some 2.5 billion queries a day.
When OpenAI believes a conversation indicates an imminent and credible risk of harm to others, it notifies authorities, according to the company.
Users seeking instructions for making biological weapons have also asked Anthropic’s Claude, Google’s Gemini and Elon Musk’s Grok, which was recently absorbed into SpaceX. It wasn’t clear if the queries were part of genuine efforts to make the weapons or exercises to probe the apps’ capabilities, according to people familiar with the exchanges.
A spokesman for Google said the company has a safety team that includes scientists who evaluate biological-weapons risks. Google also runs exercises to test the safety of the company’s AI, he said.