Malaysia and Indonesia have temporarily blocked Elon Musk's xAI artificial intelligence chatbot Grok after reports that the tool was being used to create non-consensual sexual content . The move comes as the government addresses concerns about Grok's ability to produce images, including child sexual abuse material (CSAM).
Malaysia’s regulator on Sunday ordered restrictions on access to Grok, accusing X Corp of “repeated failures” to address these risks. Indonesia followed suit the day before, summoning X officials to explain their use of the tool.
The ability to create images and the risks
A recent update to the Grok Imagine allowed users to easily create images from text messages in the chatbot (which is integrated into the X). The new feature gave the tool a wide reach, but at the same time increased the potential for malicious use to create non-consensual content.
See also: Grok: Has not set any restrictions on deepfake images
In an effort to curb misuse, xAI announced that image creation and editing will be limited to paying subscribersin order to reduce the potential for dangerous content to spread. Elon Musk told X that users who create illegal material through Grok will face the same consequences as those who post such content directly to the platform.
Regulatory reactions
Malaysia’s Communications and Multimedia Commission called X’s responses “inadequate,” noting that the company relied primarily on user-initiated reporting mechanisms that did not address the inherent risks of the tool’s design. According to the authority, the restriction is being implemented as a precautionary measure until adequate protection systems are in place , especially to prevent content targeting women and children.
Indonesia, through its Minister of Communications and Digital Affairs, Meutya Hafid, has described non-consensual sexual deepfakes as a serious human rights violation and a form of “digital violence.” The authorities stressed that the misuse of artificial intelligence to create fake pornography endangers the safety and dignity of citizens in the digital space.
See also: Grok spreads misinformation about Bondi Beach shooting incident
International reactions and investigations
The problem with Grok is not limited to Southeast Asia. Authorities in the European Union, the United Kingdom, Brazil, and India have called for investigations into Grok’s role in facilitating obscene deepfakes. The Internet Watch Foundation in the United Kingdom reported that its analysts had identified child abuse images of 11–13-year-olds that appeared to have been created using Grok.
In the US, some Democratic lawmakers have recommended that app stores temporarily suspend the tool until xAI implements significant security changes. The Department of Justice has confirmed that it takes any case of AI-generated CSAM seriously and has pledged to aggressively prosecute those responsible.
Challenges and prospects for AI
The incident highlights the growing challenges that AI brings to content moderation. The ability to create realistic images and deepfakes requires strict rules, proactive filters, and collaboration between companies, regulators, and judicial bodies.
See also: Grok: Hackers abuse X's AI assistant
The Grok case brings to the fore a critical dilemma: how can the innovative capabilities of artificial intelligence be used responsibly, without endangering the most vulnerable members of society? The actions of Malaysia and Indonesia highlight that the international community requires immediate action and stricter oversight, as the speed of technological development often outpaces existing legislation.
With authorities watching closely and users becoming increasingly proactive in reporting abuse, the future of AI chatbots like Grok depends on companies' ability to combine innovation and responsibility — a challenge that will define the next generation of artificial intelligence.
