The Grok app on an iPhone, set against a backdrop of search results displayed on the social media platform X on a laptop. Thursday, January 8, 2026. (Press Association via AP Images)
In late December 2025, the introduction of the Grok app’s picture-editing features led to a significant controversy. This situation escalated with the emergence of unauthorized sexualized deepfakes of women and girls shared on X. The scandal prompted urgent scrutiny of existing EU legislation aimed at combating non-consensual intimate imagery (NCII) and child sexual abuse material (CSAM).
As investigations into X launched almost immediately under the Digital Services Act (DSA) and the General Data Protection Regulation (GDPR), the call for enhanced protections reverberated throughout the European Union. This culminated in a remarkable consensus between the European Parliament and the European Council to propose a ban on such harmful practices within the framework of the AI Act.
A recent report by the Centre for Democracy and Technology Europe evaluated the strengths and weaknesses of current legislation addressing these concerns. This article will examine the existing safeguards and the challenges that a proposed ban under the AI Act must navigate.
Understanding the DSA’s Role
The Digital Services Act serves as the EU’s primary legal framework for regulating online platforms. It provides a mechanism to combat non-consensual deepfakes when they are circulated on these platforms. The DSA incorporates both proactive and reactive strategies to combat CSAM and NCII, whether generated by artificial intelligence or not.
Though the DSA does not impose general monitoring requirements, it mandates that all online platforms promptly remove illegal content once they are made aware of it. CSAM has long been classified as illegal throughout the EU, while the criminalization of NCII is a more recent development, already adopted by numerous member states.
In 2027, the EU Directive on Combating Violence against Women will further standardize the criminalization of various forms of technology-facilitated gender-based violence (TFGBV) across the Union. This encompasses “producing, manipulating, or altering and subsequently making publicly accessible […] material depicting sexually explicit activities or the intimate parts of a person.”
Additional provisions in the DSA obligate Very Large Online Platforms and Search Engines (VLOP/SEs) to identify and mitigate systemic risks associated with their platforms. This encompasses measures to lessen overall risks rather than focusing solely on individual pieces of content. VLOPs must publish annual risk assessment reports detailing their evaluations and mitigation strategies, which they present to the European Commission and the public.
We have previously argued that TFGBV represents an interconnected risk tied to illegal content and fundamental rights. The DSA explicitly mentions gender-based violence risks as systemic concerns. Beyond annual reports, VLOPs are also required to conduct on-the-spot risk assessments prior to launching new platform features. A comprehensive risk assessment should precede the rollout of AI-enabled features, such as those introduced by X last December, to prevent the misuse of NCII. If incidents arise after such a rollout, it suggests that an adequate risk assessment was likely neglected, violating the DSA’s provisions.
Two critical aspects are vital when analyzing the DSA’s tools for combating AI-enabled NCII. First, the DSA’s efficacy heavily relies on the speed and strength of its enforcement measures, especially concerning provisions that may not be straightforward, such as risk assessments. The EU Commission initiated an investigation in January regarding the X case, and successful outcomes are essential for the DSA to fulfill its intended purpose.
Secondly, given its limited scope, the DSA can only require online platforms or search engines to manage their services in accordance with the law. Consequently, while it can address the integration of AI tools within services—like in the Grok situation—its limitations mean it cannot regulate nudification apps or similar tools that may fall outside its jurisdiction. The scope of other EU legislation, including the AI Act, could provide complementary coverage to the DSA.
How a Ban Under the AI Act Could Enhance Protections
The only requirement regarding deepfakes within the AI Act is to label them, a measure that does not effectively prevent the generation of NCII or CSAM. The GPAI Code of Practice acknowledges NCII and CSAM as risks that general-purpose AI model providers should consider. However, these providers only need to assess certain risks, leaving it at their discretion whether or not to include NCII and CSAM in this evaluation.
In summary, the safeguards under the AI Act are limited and heavily depend on the providers’ discretion and the enforcement authorities’ commitment and capability to monitor compliance. These protections only apply to image-generation models classified as GPAI with associated systemic risks. This limitation, along with the DSA’s confined reach, has prompted calls for a comprehensive ban within the AI Act to address these concerns. Both the European Parliament and the Council have proposed amendments that would include the AI-assisted generation of NCII in the list of prohibited practices under the AI Act.
To be effective, any such ban must overcome several obstacles. Notably, no mitigation strategy can completely eliminate the creation of NCII and CSAM. Both the Council’s and Parliament’s proposals acknowledge that AI systems must only be banned if providers fail to put effective safeguards in place against the generation of such materials. While this is a promising first step, the proposals need to provide more details on what safeguards are considered sufficient.
Given that the “effectiveness” of these safeguards can be subjectively interpreted by providers, it is crucial for legislators to clarify when this threshold is achieved. Requirement for comprehensive explanations on safeguard mechanisms, along with projected effectiveness, would allow public interest technologists to evaluate and verify these claims.
Moreover, as both proposals only prohibit non-consensual intimate imagery, a method of consent verification will be necessary. This requirement may raise privacy concerns, particularly affecting vulnerable communities like sex workers. Any measures instituted must be crafted carefully to minimize dangers associated with identity verification and should consider the perspectives of those impacted while ensuring strong protections are in place.
Finally, a robust ban necessitates a careful balance between outlawing certain types of content generation while still permitting widely accepted outputs. Providers must find effective methods to navigate this delicate balance, avoiding the generation of prohibited content while allowing for legitimate outputs. Lawmakers must recognize the potential trade-offs between restricting technical capabilities and enabling lawful content creation.
Currently, the proposed bans would only extend to images explicitly depicting intimate areas or individuals engaging in sexually explicit acts. For example, many images produced by Grok on X—like those portraying women in bikinis—would not necessarily fall under such legislation yet still have the potential to cause significant harm to the individuals involved. This points to the inherent limitations of legal solutions and emphasizes the necessity for a broader, more comprehensive strategy to deter the creation of such content in the first place.
Closing Enforcement Gaps in the DSA and AI Act
The current EU legal frameworks create a fragmented landscape with varying effectiveness in addressing the critical issues highlighted by the Grok scandal, especially for victims seeking justice. Ensuring that GPAI model providers with systemic risks adhere to their obligations, along with encouraging large online platforms to conduct thorough risk assessments and mitigation efforts under the DSA, could significantly enhance protections for affected individuals.
The proposed ban on AI systems generating NCII marks an essential step in addressing existing legislative gaps. As trilogue negotiations concerning the AI omnibus move forward, co-legislators must resolve any lingering ambiguities and trade-offs within the current proposals to ensure that the ban is both effective and enforceable.
