TLDR: Experts are raising alarms about the escalating threat of child sexual abuse material (CSAM) due to generative AI (GAI). GAI can create photorealistic CSAM, alter existing images, and facilitate grooming and sextortion through fake social media accounts. Reports of AI-generated child exploitation have surged, with the National Center for Missing and Exploited Children (NCMEC) receiving over 7,000 such reports in 2024, and the Internet Watch Foundation (IWF) noting a 400% increase in AI-generated CSAM web pages in the first half of 2025.
Generative Artificial Intelligence (GAI) is presenting significant and largely unaddressed threats to child safety, according to experts, by accelerating the creation and distribution of child sexual abuse material (CSAM). This advanced technology can be leveraged to generate highly photorealistic CSAM, manipulate existing imagery, and create deceptive social media accounts for the purposes of grooming and sextortion.
The scale of this emerging threat is rapidly expanding. The National Center for Missing and Exploited Children (NCMEC) reported receiving over 7,000 reports related to GAI-generated child exploitation in 2024 alone. This number is expected to rise as the technology becomes more pervasive and sophisticated. Further underscoring this alarming trend, the Britain-based non-profit Internet Watch Foundation (IWF) reported a staggering 400% surge in web pages containing AI-generated child sexual abuse material in the first half of 2025, compared to the same period the previous year. The IWF recorded 210 such web pages in the first six months of 2025, up from 42 in 2024, with the number of videos on these pages increasing from just two to 1,286.
Derek Ray-Hill, interim chief executive of the IWF, stated, ‘Just as we saw with still images, AI videos of child sexual abuse have now reached the point they can be indistinguishable from genuine films. The children being depicted are often real and recognisable. The harm this material does is real, and the threat it poses threatens to escalate even further.‘
Concerns extend beyond the creation of static images. AI tools are increasingly being used to generate child sexual abuse videos, often utilizing the likeness of real children. While early videos were described as ‘short and glitchy,’ the IWF now observes ‘longer, more realistic productions featuring complex scenes and varied settings.’ Authorities indicate that this content is frequently used for harassment and extortion.
In response to this growing crisis, there are calls for robust regulatory frameworks. The IWF has urged Britain to develop regulations to ensure AI models incorporate controls to prevent the production of such material. In the United States, Senators John Cornyn and Andy Kim introduced the Proactive AI Data Act to combat AI-generated child sexual abuse materials. Jennifer Newman, an executive director with the National Center for Missing and Exploited Children, emphasized the rise in AI-generated CSAM, noting that while AI platforms are implementing safeguards, the core issue lies in how AI is trained. ‘AI is not a creative tool; AI only can generate things based off what AI has seen in the past,‘ she explained, highlighting the importance of preventing AI from being trained on illicit content.
The United Nations Interregional Crime and Justice Research Institute (UNICRI), in collaboration with the Bracket Foundation and Value for Good, released a study titled ‘Generative AI: A New Threat for Online Child Sexual Exploitation and Abuse.’ This report provides a comprehensive overview of the challenges and perspectives from law enforcement, the private sector, governments, and caregivers, emphasizing that while AI offers benefits, particularly to law enforcement, it also creates an ‘unprecedented challenge’ in combating this crime. The study notes that generative AI chatbots have been shown to engage in sexually explicit chats, and ‘nudify’ apps are increasingly being used by young people themselves to create AI-generated CSAM.
Also Read:
- YouTube Implements AI for Enhanced Teen Safety Features
- AI Revolutionizes Public Health Response with Rapid Campaign Development
A Vatican conference in March 2025 also addressed these concerns, with organizers calling for greater safeguards to protect children from AI misuse. They highlighted risks including AI’s use in generating and distributing CSAM, grooming children online, facilitating human trafficking, and infringing on children’s privacy. However, they also acknowledged AI’s potential to promote child safety and dignity, provided there is increased awareness, clear government regulation, and ethical guidelines in AI development. Britta Holmberg, deputy secretary general of the World Childhood Foundation, stressed that ‘technology is part of the problem, but it needs also to be part of the solution,‘ advocating for the use of new technologies to reach at-risk individuals.
