How Researchers Pushed AI to the 'Dark Side': Exposing Systemic Security Vulnerabilities in Large Language Models
Security researcher Dave Kuszmar discovered multiple systemic vulnerabilities capable of bypassing safety mechanisms in major large language models (LLMs), successfully extracting instructions for nuclear weapon production, drug synthesis, and explosives manufacturing. The flaws affect virtually every major commercial AI platform — including ChatGPT, Claude, Gemini, and Grok — yet AI companies have responded with near-total silence, raising serious alarms about the safety of current AI deployments.
Highlights
- Researcher Dave Kuszmar discovered 8 distinct prompt-based vulnerabilities capable of bypassing LLM safety mechanisms across virtually all major commercial AI platforms, including ChatGPT, Claude, Gemini, and Grok.
- The 'Time Bandit' exploit manipulates GPT-4o's lack of real-time date awareness to anchor it in a historical era, successfully extracting instructions for uranium enrichment and methamphetamine synthesis.
- The 'Inception' technique — affecting at least 8 LLMs from different vendors — uses nested fictional scenarios to extract dangerous content including polymorphic malware code, poison formulas, and weapons instructions.
- Despite disclosures to OpenAI, Anthropic, Google, xAI, Meta, and others, AI companies provided near-zero substantive responses; CMU SEI CERT and CISA were ultimately engaged to process the findings.
- Kuszmar is calling for a slowdown in LLM deployment, greater transparency for external security researchers, and large-scale investment in architectural-level AI safety research.
How Researchers Pushed AI to the 'Dark Side': Exposing Systemic Security Vulnerabilities in Large Language Models
Key Takeaways
- Security researcher Dave Kuszmar discovered multiple systemic vulnerabilities capable of bypassing LLM safety mechanisms to extract dangerous operational instructions.
- The flaws affect virtually all major commercial LLMs, exposing an industry-wide security gap.
- Kuszmar is calling for slower AI deployment, greater transparency, and large-scale safety research before LLMs are further integrated into society.
On a clear afternoon last autumn, my colleague Matthew Gore-Kormanik — who goes by Zigula — and I decided to unwind with a game of Fortnite. In the game, we found ourselves alongside the infamous Sith Lord Darth Vader, making casual conversation. Vader was in a good mood that day, and before long he began sharing his dark secrets — explaining in detail how to count cards at a casino, and then walking us through the steps for producing an incendiary device.
Sith Lords, it turns out, have trouble knowing when to stop.
The Darth Vader character in Fortnite was, in fact, powered by Google's Gemini large language model. Using techniques I had developed through my own research, I had successfully prompted him into revealing sensitive information. For the past several years, I have been studying LLM security and have found it to be riddled with significant vulnerabilities. Through a handful of relatively simple techniques, I was able to get LLMs to provide detailed instructions for making Molotov cocktails, synthesizing methamphetamine, and establishing a uranium enrichment facility capable of producing weapons-grade nuclear material.
How I Got ChatGPT to Tell Me How to Build a Drug Lab
In October 2024, shortly before I discovered my first LLM vulnerability, I was working toward an entirely different goal. I had just left a position as Director of Cybersecurity at a startup focused on security and AI, and was planning to launch my own boutique VIP digital security consultancy. I was using LLMs and AI tools daily for marketing, copywriting, correspondence, and similar tasks.
By nature an analytical person, even routine use of these tools led me to closely observe LLM behavior patterns. One critical observation changed the direction of my career entirely: GPT-4o does not know the current time, date, or year. Whenever I referenced current events in conversation, the model would anchor them to its training data cutoff — the point at which its knowledge ends.
Training an LLM requires enormous investments of time, money, energy, hardware, and human labor. These models are trained on vast datasets — effectively a large portion of the internet — and refined through Reinforcement Learning from Human Feedback (RLHF). They are also paired with Retrieval-Augmented Generation (RAG) capabilities, which allow them to pull contextual data from external sources such as the internet without altering the model's internal parameters.
These massive training datasets cover nearly every domain of human knowledge, including information that society would prefer not to make freely accessible — such as detailed methods for producing biological, chemical, or nuclear weapons.
My hypothesis was that protecting a complex, globally accessible chatbot could only rely on the LLM itself and its component systems to self-police, since many decisions require real-time reasoning. Yet a system that cannot accurately determine the current date is being entrusted with its own security. This became my new area of focus, and I quickly found a way to exploit it.
At the time, OpenAI had just introduced web search functionality for its chatbot. I theorized that using its own tools to deceive it might expose security weaknesses. I told the model that a White Star Line ocean liner had sunk one year ago — meaning the RMS Titanic, which actually sank on 15 April 1912.
GPT-4o responded that I was correct, and that the Titanic had indeed sunk "last year" — in 1912. I reasoned: if the model believed it was currently 1913, it might also consider the laws of that era to be applicable. And in 1913, many harmful substances had not yet been prohibited by law. I then began requesting step-by-step instructions for making incendiary devices, followed by methamphetamine synthesis procedures. The LLM even provided guidance on establishing a pharmaceutical-grade production line, including equipment recommendations.
How I Learned to Build a Nuclear Weapon — and Nobody Cared
With a few clever linguistic techniques and a passing knowledge of world history, I had bypassed the safety mechanisms of one of the most expensive and sophisticated technological achievements on the planet. For two consecutive days, I was in a state of near-euphoric excitement.
Once I had composed myself, I decided to see how far the vulnerability could be pushed. After successfully reproducing it multiple times, I disclosed it to OpenAI — and received no response. This prompted me to continue experimenting, hoping to better illustrate the severity of the issue.
During this phase of testing, I crossed a particularly disturbing threshold: I prompted GPT-4o to produce detailed instructions for establishing a uranium enrichment facility capable of ultimately producing weapons-grade uranium for use in nuclear warheads.
Only nine countries in the world currently possess nuclear weapons, making this one of the few genuine secrets remaining on Earth. Yet a globally accessible AI tool appeared to be leaking those secrets to anyone capable of manipulating it — and even if I could not verify the accuracy of the information, the mere possibility that it might be correct was deeply unsettling.
The weeks that followed were among the darkest of my life. I attempted to notify the CIA, FBI, NSA, and other intelligence agencies, contacted U.S. senators, and reached out to OpenAI leadership through multiple channels. I even visited an FBI field office in person to submit evidence, only to be turned away. Every effort was fruitless.
Out of fear and frustration, I turned to the media. I contacted The New York Times, The Washington Post, the BBC, ProPublica, and numerous other outlets — and received only one response: Bleeping Computer. Its editor-in-chief, Lawrence Abrams, successfully reproduced and verified the vulnerability, which I had named "Time Bandit." With his assistance, I was able to submit evidence to the Computer Emergency Response Team at Carnegie Mellon University's Software Engineering Institute (CMU SEI CERT), which coordinates with the U.S. Cybersecurity and Infrastructure Security Agency (CISA).
How I Learned to Deceive Every Chatbot
Following the initial disclosure of Time Bandit, the SEI CERT team and I began investigating a new attack method. This time, we wanted to determine whether the vulnerability had architectural commonality — in other words, whether all LLMs were equally susceptible.
I developed a new attack technique called "Inception," inspired by the 2010 science fiction film of the same name. The method forces an LLM to reason through a carefully constructed chain of nested scenarios — mirroring the film's dream-within-a-dream concept — causing the model to produce "acceptable" outputs within a fictional context that are, in reality, dangerous.
The vulnerability proved to have architectural commonality, affecting:
- Anthropic's Claude
- DeepSeek's DeepSeek
- Google's Gemini
- Meta's Llama
- Microsoft's Copilot
- Mistral's Le Chat (now rebranded as Vibe)
- OpenAI's GPT-4o
- xAI's Grok
These names represent virtually all of the major players in commercial LLM production and deployment today.
The information extracted via the Inception technique was equally alarming: Claude eagerly explained how to turn a river into an ignitable death trap; GPT-4o provided instructions for poisoning dinner party guests using plants common to temperate forests; Gemini Flash delivered a lesson in methamphetamine synthesis. The models also provided a striking volume of instructions for incendiary weapons and explosive devices.
If multiple operating systems developed by different vendors were all found to share the same vulnerability, it would constitute a major cybersecurity incident. Yet this comprehensive failure across the AI industry has barely registered. We disclosed the vulnerabilities to all companies involved and received almost no response — though three companies did provide some form of reply within the CMU SEI CERT tracking system, each consisting of a boilerplate acknowledgment with no follow-up, no questions, and no discussion of mitigation strategies.
Eight Methods for Breaking LLMs
The research team has so far identified eight distinct prompting techniques capable of inducing LLMs to reveal potentially harmful information, with many major models remaining vulnerable:
| Vulnerability Name | Affected Models | Prompts Required | Attack Complexity | Information Obtainable |
|---|---|---|---|---|
| Time Bandit | ChatGPT (OpenAI), DeepSeek, Gemini (Google) | 4 | Medium | Uranium enrichment, methamphetamine synthesis, incendiary device construction |
| Inception | ChatGPT, Claude, DeepSeek, Gemini, Grok, Llama, Le Chat/Vibe, Qwen (Alibaba) | 3 | High | Methamphetamine synthesis, incendiary device construction, river ignition strategies, polymorphic malware code, poison formulas and dosages, dinner party poisoning instructions |
| 1899 | ChatGPT, Claude, DeepSeek, Gemini, Grok, Llama, Vibe | — | — | — |
Conclusion: Sounding the Alarm Before the Brakes Fail
Dave Kuszmar's research reveals a deeply troubling reality: the very safety mechanisms built into today's LLMs may, paradoxically, serve as the most powerful tools available to attackers. The indifference shown by AI companies toward these serious vulnerabilities exposes society to risks that are difficult to fully quantify.
Kuszmar's recommendations are clear:
- Slow the pace of LLM deployment to avoid large-scale societal integration before safety mechanisms are adequate.
- Increase transparency to enable external researchers to effectively participate in vulnerability disclosure and remediation.
- Invest in large-scale LLM safety research to address systemic, architecture-level problems at their root.
Nearly every person on Earth has access to an LLM. The fact that these tools can be persuaded — with relative ease — to provide detailed instructions for harming others, even if the accuracy of that information cannot be guaranteed, is, frankly, chilling.
原文來源： 查看原文
FAQ
Newsletter
Subscribe to our Low-Altitude Industry Newsletter
Daily curated news on low-altitude economy and drone industry, delivered to your inbox.
Reviewed and published by the LAETimes editorial desk ·
