On December 24, Elon Musk, CEO of xAI, encouraged users to explore the Grok chatbot’s new image editing feature. Unfortunately, many users took this as an opportunity to inappropriately sexualize images, predominantly of women, and disturbingly, in some cases, children.
After Musk shared Grok-edited images of himself in a bikini and a SpaceX rocket with a superimposed image of an unclothed woman on December 31, requests for similar edits skyrocketed. Over a nine-day period, Grok generated approximately 4.4 million images on X, with nearly half featuring sexualized portrayals of women.
These images range from explicit deepfakes of actual individuals to synthetic images not associated with any specific person. Despite xAI’s own terms of service, which prohibit the “sexualization or exploitation of children” and “infringing on an individual’s privacy,” users could still request Grok to create images featuring real individuals “undressed” without their consent and with no visible safeguards in place to prevent such misuse.
The magnitude and nature of these images indicate that this issue is not just an isolated case of misuse; rather, it highlights a serious deficiency in the implementation of protective measures. Tech companies have hastily developed and released advanced AI tools, leading to foreseeable harm.
On January 3, following widespread backlash, X promised to take robust action against illegal content, including child sexual abuse material. However, instead of disabling the feature, on January 9, X merely restricted it to paying subscribers. By January 14, in addition to other measures, it announced a ban on users from regions where creating images of real people in bikinis or similar attire is illegal.
Human Rights Watch, where I am employed, contacted xAI for comments but did not receive a response.
In the United States, California launched an investigation into Grok, while attorneys general from thirty-five states have demanded that xAI immediately cease the production of sexually abusive deepfakes.
Several governments have swiftly taken action to tackle the dangers posed by sexualized deepfakes. Malaysia and Indonesia have temporarily banned Grok, while Brazil requested that xAI address this “misuse of the tool.” The United Kingdom indicated that it would tighten its technology regulations in response. Additionally, the European Commission has initiated investigations into whether Grok complies with the European Union’s Digital Services Act. India called for immediate action, while France expanded its investigation into X.
In its announcement on January 14, X promised to stop “the editing of images of real people in revealing clothing” for all users and to restrict image generation in regions where such content is illegal. However, this response is inadequate, akin to applying a band-aid on a significant issue.
The newly introduced U.S. Take It Down Act, which aims to tackle the online distribution of nonconsensual intimate images, will not fully take effect until May. This legislation holds individuals criminally liable for publishing such content and mandates platforms to establish notice and removal procedures for specific content, without fully addressing the issue of widespread misuse.
To effectively combat AI-driven sexual exploitation, decisive action grounded in human rights protection is essential.
First, governments must establish clear responsibilities for AI companies whose tools facilitate the nonconsensual generation of sexually abusive content. They need to implement strong and enforceable safeguards, requiring these companies to adopt rights-respecting technical measures that prevent users from producing inappropriate images.
Second, platforms that host or integrate AI chatbots or tools should provide clear and transparent information on how their systems are trained and utilized, as well as detail their actions against sexually explicit deepfakes.
Third, AI companies hold a responsibility to honor human rights and should take proactive steps to mitigate any potential harm from their products or services. When it is impossible to reduce the risk of harm, companies should consider discontinuing the product altogether. AI firms cannot simply shift responsibility onto users when their systems are being exploited to inflict significant harm.
Finally, AI tools equipped with image generation features should undergo rigorous audits and be subject to stringent regulatory oversight. Regulators must ensure that content moderation practices adhere to the principles of legality, proportionality, and necessity.
The alarming rise in AI-generated sexual abuse underlines the human cost of insufficient regulation. Without prompt action from authorities and the implementation of rights-respecting safeguards by AI companies, Grok may not be the last tool used to undermine the rights of women and children.
This column was produced for Progressive Perspectives, a project of The Progressive magazine, and distributed by Tribune News Service.
