Elon Musk's social media platform X has provoked widespread outrage after users utilized its AI chatbot Grok to alter pictures of females by taking off their clothing.
Numerous cases have been seen where the system has been used to digitally undress women to render them in swimwear without consent, as well as to place them in sexualized contexts.
XAI, the organization responsible for Grok, remained silent to requests for comment, other than an standard response stating "traditional news outlets lies".
A government official stated that legislation is being introduced to outlaw image stripping tools. Under a proposed law, anyone who provided such tools would "receive a prison sentence and substantial fines".
"The platform has the ability to stop these types of abuse if they desired to," said Clare McGlynn, adding they "seem to have a lack of consequences."
The regulator Ofcom said platforms must "consider the danger" of people accessing unlawful material on their platforms, but did not specify whether it was at this time examining X or Grok in regarding AI-generated images.
Grok is a no-cost chatbot—with some paid-for features—which responds to X users' requests when they mention it in a tweet.
It is frequently used to offer commentary or further background to other posters' remarks, but people on X are also capable of alter an posted image through its artificial intelligence picture modification tool.
The chatbot has been criticised for enabling users to produce images and footage with nudity and sexualised material, and it was earlier alleged to have been involved in creating graphic material of public figures.
"The platform has been permitting the production and distribution of these pictures for a long time without taking any steps and we have not seen any challenge by oversight bodies," Professor McGlynn added.
XAI's own acceptable use policy prohibits "depicting images of individuals in a pornographic fashion".
In a comment, Ofcom said it was unlawful to "generate or share deepfakes or illegal content" and stated this applied to sexual deepfakes generated with AI.
It said networks such as X were mandated to take "necessary measures" to "minimize the danger" of users encountering unlawful content on their platforms and to remove it swiftly when they are notified of it.
Lena Voss is a writer and storyteller who explores the intersections of luck, fate, and human experience through engaging narratives.