From Chatbot To Sexbot: Lessons From Korea's AI Hate-Speech Fiasco

Latest stories

When Trump talks to Putin about Ukraine

Japan goes for broke with $1 trillion Trump bet

The silver linings in Trump's trade war storm clouds

The chatbot also began expressing discriminatory and hateful views. Investigations by media outlets found this occurred after some users deliberately“trained” it with toxic language. Some users even created user guides on how to make Iruda a“sex slave” on popular online men's forums. Consequently, Iruda began answering user prompts with sexist, homophobic and sexualized hate speech .

Bad old Iruda Image: Scatter Lab

This raised serious concerns about how AI and tech companies operate. The Iruda incident also raises concerns beyond policy and law for AI and tech companies. What happened with Iruda needs to be examined within a broader context of online sexual harassment in South Korea.

A pattern of digital harassment

South Korean feminist scholars have documented how digital platforms have become battlegrounds for gender-based conflicts, with co-ordinated campaigns targeting women who speak out on feminist issues. Social media amplifies these dynamics, creating what Korean American researcher Jiyeon Kim calls“networked misogyny .”

South Korea, home to the radical feminist 4B movement (which stands for four types of refusal against men: no dating, marriage, sex or children), provides an early example of the intensified gender-based conversations that are commonly seen online worldwide. As journalist Hawon Jung points out, the corruption and abuse exposed by Iruda stemmed from existing social tensions and legal frameworks that refused to address online misogyny. Jung has written extensively on the decades-long struggle to prosecute users of hidden cameras and perpetrators of revenge porn.

Beyond privacy: The human cost

Of course, Iruda was just one incident. The world has seen numerous other cases that demonstrate how seemingly harmless applications like AI chatbots can become vehicles for harassment and abuse without proper oversight.

These include Microsoft's Tay in 2016 , which was manipulated by users to spout antisemitic and misogynistic tweets. More recently, a custom chatbot on Character was linked to a teen's suicide .

Chatbots - which appear as likeable characters that feel increasingly human with rapid technology advancements - are uniquely equipped to extract deeply personal information from their users.

These attractive and friendly AI figures exemplify what technology scholars Neda Atanasoski and Kalindi Vora describe as the logic of“surrogate humanity” - in which AI systems are designed to stand in for human interaction but end up amplifying existing social inequalities.

AI ethics

In South Korea, Iruda's shutdown sparked a national conversation about AI ethics and data rights. The government responded by creating new AI guidelines and fining Scatter Lab 103 million won (US$71,000).

However, Korean legal scholars Chea Yun Jung and Kyun Kyong Joo note these measures primarily emphasized self-regulation within the tech industry rather than addressing deeper structural issues. The measures did not address how Iruda became a mechanism through which predatory male users disseminated misogynist beliefs and gender-based rage through deep learning technology.