TLDR: Artificial intelligence is being increasingly weaponized to create highly realistic and traumatizing online death threats, as highlighted by Australian activist Caitlin Roper. Perpetrators are using AI models to generate manipulated images and videos depicting victims in violent scenarios, often requiring only a single profile picture to create convincing fakes. This alarming trend extends beyond individual harassment to include broader issues like AI-fueled scams, non-consensual adult content creation, and ‘swatting’ incidents, posing significant challenges for online safety and mental well-being. Social media platforms are struggling to adequately address this new form of abuse.
In a disturbing development in online harassment, artificial intelligence is now being deployed to create terrifyingly realistic death threats and violent abuse, according to Australian activist Caitlin Roper. Roper, who has years of experience working online, described herself as traumatized by the recent surge of AI-fueled threats she and her colleagues at Collective Shout, an Australian activist group, have received.
Historically, generating convincing digital threats required extensive online presence from the victim. However, experts now indicate that advanced AI models can replicate real individuals with just a single profile picture, enabling the creation of any desired content. Roper recounted receiving manipulated images and videos depicting her in horrific situations, such as hanging from a noose or ablaze and screaming. A particularly unsettling detail was the inclusion of a blue floral dress she actually owns in many of these fabricated images, making the threats feel ‘more real and, somehow, a different kind of violation.’ Roper noted, ‘These things can go from fantasy to more than fantasy.’
The torrent of online abuse directed at Roper began this summer following her campaign against violent video games that glorify adult scenes and abuse. While some of the offending accounts and images have been removed, Roper claims that X (formerly Twitter) informed her that other posts depicting her violent death did not violate the platform’s terms of service. In a shocking revelation, Roper stated that X, at one point, even included one of her harassing accounts on a list of recommended people to follow. Furthermore, some harassers have boasted about using AI tools like Grok to research how to locate women at home and in public places.
When Roper posted examples of the gratuitous gore she was receiving, X temporarily locked her account, citing a breach of its safety policies against such content. This incident underscores the broader challenges posed by AI misuse.
Beyond individual harassment, there is a global campaign against the use of artificial intelligence for various illicit activities. Criminals are leveraging AI to mimic real people’s voices for scams, create non-consensual adult content, and execute ‘swatting’ incidents. Swatting involves making false emergency calls to provoke a large police response, with perpetrators using cloned voices and manipulated images to convince law enforcement of fabricated scenarios. The National Association of Attorneys General noted this summer that AI ‘has significantly intensified the scale, precision, and anonymity’ of such attacks. One documented case involved a serial offender using simulated gunfire via AI to trigger a 20-minute lockdown at a Washington state high school, prompting a response from police and federal agents.
Also Read:
- Beyond the Hype: Unpacking the Overlooked Aspects of AI Accountability and Governance
- LinkedIn to Begin Default AI Training with User Data on November 3, 2025: How to Opt Out
The increasing sophistication of AI in generating deceptive and harmful content presents a critical challenge for online safety, cybersecurity, and the mental well-being of victims, demanding more robust responses from platforms and authorities.
