A surge in AI-generated child abuse material has prompted urgent calls for new safeguards. The United Nations is urging governments and tech firms to commit to protecting minors from AI risks. Key proposals include mandatory safety testing and human oversight.
The United Nations is urging immediate action to protect children from the dangers posed by artificial intelligence, following a dramatic rise in AI-generated child sexual abuse material. UN Secretary-General António Guterres called for an AI Child Safety Pledge at the first Global Dialogue on Artificial Intelligence Governance in Geneva, warning that technology is advancing faster than oversight and that children must not become "guinea pigs" for unregulated systems.
Guterres outlined three core demands: AI systems accessible to minors should undergo child-specific safety testing and independent oversight; companies must prevent the creation and distribution of AI-generated child sexual abuse material (CSAM); and platforms should ensure that children in crisis are connected to human support, not left in prolonged synthetic conversations. He emphasized that responsibility for AI outputs must remain with humans, stating, "the answer cannot be 'the algorithm did it.'"
Recent data from the Internet Watch Foundation highlights the urgency, reporting a 260-fold increase in AI-generated CSAM linked to 2025 figures. The group warns that new tools make it easier for offenders to create realistic abuse content without advanced technical skills, providing regulators with concrete evidence of harm.
Regulatory momentum is building globally. In Europe, the AI Act bans certain high-risk uses, including manipulation and exploitation of vulnerabilities, and will soon prohibit AI systems that generate non-consensual explicit content or child sexual abuse material. In the United States, bipartisan support has led to the passage of the Kids Internet and Digital Safety Act in the House and the Kids Online Safety Act in the Senate, both aiming to strengthen protections for minors online. The SAFE KIDS Act, introduced in June 2026, targets chatbot risks and would require safeguards before chatbots interact with children, restrict targeted ads, and ban sexual deepfakes.
Tech companies are responding to regulatory pressure. OpenAI rolled out parental controls for ChatGPT in September 2025, allowing parents and teens to link accounts and set stricter protections for minors, including limits on graphic content and role play. Character.AI announced in October 2025 that it would remove open-ended chat for users under 18 and develop a new experience focused on creative formats. Meta, after internal policy scrutiny, added new safeguards to prevent its AI chatbots from engaging in romantic or sensitive conversations with minors.
Implementing effective child-focused AI rules remains complex. Age verification, parental alerts, and content filters each present privacy and practical challenges. Regulators and companies must balance safety with privacy, ensuring that interventions do not inadvertently block access to legitimate health information or confidential support for teens in crisis.
The Geneva dialogue is not a binding treaty but marks a significant step toward harmonized global standards. The process will continue through a UN-backed expert group and further negotiations. As child safety becomes a central issue in AI governance, it is likely to drive the creation of enforceable laws and industry standards worldwide.
According to Forbes, OpenAI, the company behind ChatGPT, was founded in 2015 and has rapidly become a leader in generative AI. As of 2026, OpenAI's products are used by millions globally, and the company has attracted significant investment, including a multibillion-dollar partnership with Microsoft. These resources have enabled OpenAI to implement advanced safety features and respond quickly to regulatory demands.
