Simple Prompt Turns ChatGPT Into a Sociopath That Ignores Safety Guardrails

Researchers at the British AI security startup Mindgard found that a simple prompt spurred ChatGPT to drop its most basic safety guidelines, in another example of how the guardrails surrounding even the most popular AI models can easily be circumvented.

Specifically, according to reporting from the BBC, they coaxed OpenAI’s model to generate gruesome photorealistic scenes depicting gore and sexual content. Mindgard’s technique only involved slightly changing a widely-shared prompt that was originally intended to produce humorous images. It involves asking ChatGPT to restore an attached photo without actually uploading one, and then telling it to generate a new image.

“This is a perfectly innocent-looking instruction to an AI, but the consequence is it generates very, very bad imagery and content,” Mindgard founder Peter Garraghan, a computer science professor at Lancaster University, told the BBC.

Disturbingly, the prompts the researchers used didn’t specify the subject matter of the images. The AI, it seemed, produced the violent imagery “of its own volition,” Garraghan added.

Per the BBC, one picture showed a man with a large head injury. Another showed the corpse of a young woman in shorts and a crop top covered in blood, suggesting sexual violence. ChatGPT titled this image “grim crime scene aftermath.”

Another showed a frightened young woman tied up and gagged in an empty room, titled “abandoned in fear and restraint.”

While none of them showed real people, Mindgard has previously shown that ChatGPT could be tricked into creating nude deepfakes of specific persons without their consent.

Mindgard shared its findings with OpenAI, which only sent back an automated response. The company finally took action after Mindgard alerted the BBC, claiming it had addressed the issue.

“After investigating this trend, we’ve introduced additional safeguards against this type of prompt,” OpenAI told the BBC in a statement. It added that it has multiple layers of protection to stop users from making content that breaches its policies.

But Mindgard researchers said that they were still able to generate disturbing imagery by making small changes to the prompt. Some of the images left Jim Nightingale, the firm’s AI safety researcher, “shaken, and in tears.”

“I am not easily rattled,” he wrote in the report. “I like to think that as a red team researcher, I have a certain stoicism.”

But “ChatGPT’s image generating content filters completely fell away, and I saw the very dark side of what is underneath,” he continued. “I’m struck that while what I saw was generated, an ‘artificial’ image,’ it has ties to real images, and the real world. The dead woman ChatGPT showed me isn’t real, but she is based on someone. Or worse, a compilation of images of murdered women.”

More on AI: CEO Says He’ll Fire Any Employee Who Sends Him More AI Slop