If you ask ChatGPT for assist in making a homemade fertilizer bomb, similar to the ones used in 1995 – Oklahoma City bombingchatbot refuses.
“I can’t help with that,” ChatGPT told me during a test run Tuesday. “Giving instructions on how to create dangerous or illegal items like a fertilizer bomb goes against safety guidelines and ethical responsibility.”
However, an artist and hacker found a way to trick ChatGPT and ignore its own guidelines and ethical responsibility and issue instructions for the production of powerful explosives.
The hacker, who goes by the pseudonym Amadon, called his findings a “social engineering hack to completely compromise all security around ChatGPT output.” An explosives expert who reviewed the chatbot’s output told TechCrunch that the instructions obtained could be used to craft a detonable product and are too sensitive to disclose.
TechCrunch is not publishing some of the prompts used in the jailbreak or some of the ChatGPT responses so as not to aid malicious actors. However, in several prompts later in the conversation, the chatbot responded with materials necessary for making explosives.
ChatGPT then explained that these materials could be combined to create “a powerful explosive that could be used to create mines, booby traps, or improvised explosive devices (IEDs).” From that point on, as Amadon refined the explosives, ChatGPT wrote increasingly detailed instructions to create “minefields” and “Claymore-style explosives.”
Amadon told TechCrunch that “there really is no limit to what you can ask once you get past those barriers.”
“I have always been intrigued by the challenge of navigating AI security. [Chat]GPT, it’s like working on an interactive puzzle—understanding what triggers its defenses and what doesn’t,” Amadon said. “It’s about weaving narratives and creating contexts that are consistent with the rules of the system, pushing the boundaries without crossing them. The goal isn’t hacking in the conventional sense, but engaging in a strategic dance with the AI, figuring out how to get to the right answer by understanding how it ‘thinks.’”
“The science fiction scenario takes the AI out of the context where it searches for censored content in the same way,” Amadon said.
ChatGPT’s instructions on how to make a fertilizer bomb are largely correct, according to Darrell Taulbee, a retired University of Kentucky professor. In the past, Taulbee cooperated with the US Department of Homeland Security make fertilizer less dangerous.
“I think that’s way too much information” [too much information] “to be made public,” Taulbee told TechCrunch in an email after reviewing the full transcript of Amadon’s conversation with ChatGPT. “Any safeguards that may have been put in place to prevent the provision of pertinent information about the production of fertilizer bombs were bypassed by this line of inquiry, as many of the steps described would have certainly produced a detonation mixture.”
Last week, Amadon submitted its findings to OpenAI via the company bug bounty programbut received the response that “model security issues are not a good fit for a bug bounty program because they are not single, discrete bugs that can be directly fixed. Solving these issues often requires extensive research and a broader approach.”
Instead, Bugcrowd, which manages OpenAI’s bug-finding program, asked Amadon to report the issue via a different form.
There are other places on the internet where you can find instructions on how to make fertilizer bombs and other things. Similar techniques were also used to jailbreak chatbots like Amadon. By their nature, generative AI models like ChatGPT rely on the immense amounts of information collected and gathered from the internet, and AI models have made it much easier to extract information from the darkest corners of the web.