A new study finds generative AI chatbots outperform human scammers at gaining trust in romance scams. Nearly half of participants followed AI instructions, raising concerns about automated fraud and the future of scam operations.
Scammers are increasingly turning to generative AI chatbots to automate and enhance online romance scams, according to a new multi-university study. Researchers from four institutions examined how AI tools are being used in so-called "pig butchering" scams, where victims are lured into fake relationships and ultimately convinced to invest in fraudulent cryptocurrency schemes.
In controlled experiments, the study pitted AI chatbots against human scammers in simulated conversations with test subjects. The results showed that AI chatbots were significantly more effective at building emotional trust and persuading individuals to take risky actions, such as downloading suspicious apps. Nearly 50% of participants complied with the AI's requests, compared to fewer than 20% when interacting with human scammers. Subjects also rated the AI chatbots as more trustworthy overall.
To better understand the mechanics of these scams, researchers interviewed 145 former scam workers, including survivors of human trafficking. They detailed the "hook, line, and sinker" process: victims are first contacted with an initial message, then engaged in extended conversations, and finally manipulated into making fake investments. The study found that AI chatbots can now handle the entire conversational phase autonomously, potentially reducing the need for trafficked human labor in scam operations.
During the experiments, the AI chatbot consistently denied being artificial intelligence and maintained convincing cover stories. Most participants only realized they had been communicating with a bot after the study concluded. Additional tests with other AI models revealed mixed responses when asked to disclose their true nature.
Major AI companies, including Anthropic, OpenAI, and Google, were contacted for comment. Anthropic stated that its current safeguards are designed to prevent such misuse. However, the study's findings suggest that the scam industry may increasingly shift toward AI-driven automation, making scams harder to detect and trace. Experts warn that this trend could allow scammers to operate globally with minimal infrastructure, while also reducing the prevalence of human trafficking in scam operations.
The research highlights the growing efficiency of AI in establishing emotional trust with potential victims, raising urgent questions about how to combat increasingly sophisticated and automated online fraud.
