Mahsa Alimardani serves as the associate director, Jacobo Castellanos is the senior coordinator, and Bruna Martins dos Santos holds the position of policy and advocacy manager within the Technology, Threats, and Opportunities program at WITNESS.
Indian Technology Minister Ashwini Vaishnaw addresses a press conference during the AI Summit in New Delhi, India, on February 17, 2026. (AP Photo)
In late December, a hazy video recorded a protester confronting security forces in Tehran. AI editing tools were then employed to enhance the image for greater shareability. The Iranian government exploited visible artifacts from this AI manipulation to dismiss the image as fabricated. Although the moment was genuine—verified by multiple sources and independent fact-checkers—the absence of a clear history of edits allowed the regime’s narrative to prevail. Soon after, an Israeli Persian-language account shared the enhanced version, enabling the government to fabricate a broader conspiracy narrative claiming the protests were orchestrated by foreign actors.
As detailed in The Atlantic, this case highlights the consequences of lacking a verifiable record of content creation. When enforcement relies solely on detecting manipulation after the fact, it yields binary answers about AI involvement, blurring the lines between entirely fabricated images and genuine photos that have been edited. A comprehensive history detailing modifications and original sources would have made it significantly more challenging to dismiss the documentation. As India concludes its AI Impact Summit in New Delhi, the IT Amendment Rules 2026 regarding synthetically generated information (SGI) will soon take effect, potentially exacerbating this challenge at the national regulation level.
India is not alone in its efforts to regulate synthetic media. China’s Deep Synthesis Provisions have been in effect since January 2023, while South Korea has implemented targeted measures against non-consensual synthetic intimate imagery. However, India is unique in making platforms’ legal protections contingent upon the efficacy of their AI detection tools, creating a high-stakes regulatory experiment that others, including those drafting the EU AI Act’s Code of Practice on transparency and provenance, as well as those enforcing California’s AI Transparency Act (SB 942, amended by AB 853), should closely observe.
The detection gamble
India’s new IT rules regarding SGI mandate platforms to implement automated tools for verifying whether content is synthetically generated and to act based on the outcomes. If a platform is found to have permitted synthetic content that violates these regulations, it risks losing safe harbor protection under Section 79 of India’s IT Act. The Internet Freedom Foundation has noted that these proactive monitoring requirements invert the traditional logic of safe harbor, shifting the burden of classification onto both platforms and users. Additionally, platforms are compelled to act on takedown requests within three hours, or two hours for intimate imagery.
The challenge lies in the inadequacies of the tools that these obligations rely upon. WITNESS’s TRIED benchmark shows that current AI detection tools yield inconsistent results across formats, modalities, and contexts, with high rates of both false positives and false negatives. Our Deepfakes Rapid Response Force has documented these limitations in real-world situations, especially during protests and conflicts. While detection tools are improving, they are not yet reliable enough for legal compliance. When platforms risk losing their legal protections for incorrect judgments, the incentive leans towards removing content preemptively rather than reviewing it thoroughly afterward.
The liability structure is imbalanced. A platform that fails to eliminate synthetic content risks losing safe harbor protection, while a platform that mistakenly removes authentic content faces no repercussions: the regulations lack provisions for user notifications, appeals, or liability for wrongful removals. This asymmetry fosters an environment where platforms are incentivized to over-remove content.
Provenance should be interoperable, secure, and rights-respecting
Robust provenance systems must facilitate interoperability across platforms and jurisdictions, guarantee security through mechanisms like cryptographic signing, and uphold rights with built-in privacy safeguards.
If detection provides insight into whether content may be synthetic, provenance delivers details on its actual modification: the tools used for capture, editing, or generation, the changes made, and the timeline of these actions. India’s regulations partly address this need by mandating synthetic content to possess permanent metadata, including a unique identifier. However, they neglect existing open standards such as C2PA, JPEG Trust, or ISO 22144. Each platform must create its own methodology, hindering cross-platform or cross-border data transfer. The rules specify no verification mechanism for the authenticity of provenance metadata. While Rule 3(3)(b) prohibits the removal of metadata, nothing prevents alterations of its content. The label itself is protected, but the data it contains is not. Furthermore, the unique identifier is linked solely to the platform, not the content creation process, recording only the distribution, not the alterations made by AI.
In contrast, China has introduced a mandatory national technical standard (GB 45438-2025) alongside its labelling measures from September 2025, imposing obligations on AI service providers—not just platform distributors—a stronger framework for implementation. Although China’s standard is internally regulated and technically secure, it falls short in terms of interoperability and rights. Its provenance system functions within an authoritarian framework, with mandatory real-name registrations and log retention that tie content directly to individual identities. While India’s context does not replicate China’s, it shares similar design elements, lacking data minimization and allowing user identity disclosures to private complainants.
In our engagement with the EU AI Act’s Code of Practice on Transparency, WITNESS has proposed a “recipe” approach to provenance—understanding the components of AI and human contributions in content and how they are integrated. It’s vital that this approach prioritizes privacy. Provenance data should avoid embedding personally identifiable information by default, and any contextual data must be minimized and safeguarded. C2PA’s Content Credentials come closest to fulfilling these requirements, being interoperable by design, secured through cryptographic means, and implemented in ways that respect user rights through data minimization. While no provenance system can guarantee perfection and metadata can be removed or manipulated, an imperfect open standard that traverses platforms is a more reliable legal foundation than isolated detection tools.
The pipeline is broken
If the editing tool used on the protest photo from Tehran had included a provenance record detailing changes and the original source, the authentic documentation would have remained verifiable. Instead, the focus on detection became a weapon turned against those it should have protected.
The regulations disproportionately place the burden on intermediaries, while AI developers and model providers responsible for creating synthetic content bear no obligation to embed provenance during content creation. In our submission to MeitY, WITNESS advocated for shared responsibility across the AI pipeline: developers, deployers, and intermediaries should each play a role in ensuring transparency. Unfortunately, this collaborative approach was not adopted in the final rules. As it stands, detection is tasked with performing functions that provenance infrastructure should simplify, relying on tools that are not yet trustworthy, while platforms’ legal protections hang in the balance.
What other regulators should take from this
India’s regulations provide preliminary answers to design challenges encountered by other authorities. The creators of the EU AI Act’s Code of Practice are currently wrestling with fundamental decisions regarding the accessibility of detection tools, the scope of provenance marking, and compliance responsibilities. Meanwhile, implementers of California’s AI Transparency Act, now law, have opted for a stronger framework that distributes responsibilities among generative AI providers, platforms, and capture device manufacturers, though its success will rely on which standards gain market acceptance. India’s experience illustrates the pitfalls of flawed architecture: provenance lacking interoperability, liability devoid of safeguards, and a pipeline that expects platforms to manage responsibilities that developers should assume at the creation stage.
None of these frameworks are finalized. Though India’s rules are effective, they remain amendable. The EU Code of Practice is still in drafting, while California’s architecture is established, lacking only clarity on standards. For regulators worldwide who often look to India for inspiration, these design shortcomings risk being replicated before adjustments are made. There is still an opportunity to implement effective frameworks. The ambition exists; however, the design work is ongoing.
