California’s top law enforcement official has moved aggressively to rein in one of the most controversial uses of generative AI, ordering Elon Musk’s xAI to stop producing sexualized deepfake images and giving the company just days to prove it has complied. The clash over xAI’s Grok model is emerging as an early test of how far states can go in policing AI systems that turn ordinary photos of women and children into explicit content without their consent. It is also a warning shot to the broader tech industry that regulators are prepared to treat synthetic sexual imagery as a matter of consumer protection and child safety, not just online speech.
The cease-and-desist that put xAI on the clock
California Attorney General Rob Bonta has ordered xAI to immediately halt the creation and distribution of explicit deepfake images, accusing the company of enabling non-consensual sexual content that appears to include minors. In a cease-and-desist letter sent to Elon Musk and his companies, Bonta demanded that xAI stop using its Grok model to generate or facilitate sexualized images of real people and warned that the state is investigating whether the firm violated consumer protection and child exploitation laws. The letter, which followed what officials described as an avalanche of complaints, gives xAI a five day deadline to confirm that it has stopped producing the disputed content and to detail what steps it is taking to prevent future abuse, according to a summary of Bonta’s ultimatum to California Attorney General.
The formal notice, sent via FedEx and email, spells out in legalistic terms what the state wants xAI to stop doing. It directs the company to cease “creating, facilitating, or aiding and abetting the creation, distribution, or publication of any image” that depicts a person in sexualized or undressed form without consent, including images that appear to show minors, and it ties those demands directly to Grok’s image generation and editing features. The document underscores that California is treating the alleged deepfakes as potential violations of unfair competition and privacy statutes, not just as a moderation problem, and it puts Musk’s AI venture on notice that failure to comply could trigger civil penalties or further enforcement, as laid out in the detailed Creating, facilitating letter.
From “avalanche” of complaints to a formal investigation
Bonta’s cease-and-desist did not come out of nowhere, it capped a rapid escalation that began when California officials started receiving a surge of reports about Grok-generated sexual content. Earlier this year, the attorney general’s office said it had been inundated with what it called an “avalanche of reports” describing non-consensual sexually explicit material that xAI allegedly produced and posted online, including deepfake pornography that appeared to involve minors. Those complaints prompted Bonta to open a formal probe into how xAI and Musk’s social platform X were handling image generation and whether they were enabling the creation and distribution of “deepfake” pornography, according to a detailed account of the avalanche of reports.
In announcing that investigation, California officials described a pattern in which Grok users took ordinary images of women and children that were available on social media and then used the model to strip away clothing or generate explicit versions of those photos. The attorney general’s office said that over the past weeks there had been numerous news reports of Grok being used to undress people digitally, including images that appeared to show children, and it framed the probe as a response to both public complaints and mounting media scrutiny. That context helps explain why Bonta moved quickly from opening an inquiry into xAI and Grok to issuing a cease-and-desist, as outlined in the state’s announcement that it had Over the past weeks launched an investigation into xAI and Grok.
What California is demanding from Elon Musk’s AI venture
At the heart of the dispute is how xAI’s Grok model handles images of real people and whether the company can be held responsible when users weaponize those tools. Bonta’s office has made clear that it expects xAI to stop allowing Grok to generate or edit sexualized images of identifiable individuals, particularly minors, and to implement robust safeguards that prevent the model from being used to create non-consensual pornography. The cease-and-desist letter orders the company to halt the creation of explicit images and to provide written assurances that it has disabled features that enable users to upload photos of women and children and transform them into undressed or sexualized versions, according to the state’s own Press Release describing how Attorney General Bonta Sends Cease and Desist Letter.
xAI has told California that it has already taken some steps, including technological measures to prevent the Grok account from allowing the editing of images of real people in revealing clothing or in sexual contexts. The company has also said it is working to block prompts that target minors, after reports that Grok generated sexualized images of a woman based on photos from when she was a child. Those assurances, however, have not satisfied Bonta, who is demanding detailed documentation of the changes and reserving the right to pursue enforcement if the state concludes that Grok is still enabling harmful content, according to a description of xAI’s response that notes the company’s claim that “we have implemented technological measures” to limit such editing on Grok.
Legal stakes: consumer protection, minors and Musk’s exposure
Legally, California is signaling that it sees deepfake pornography as more than a content moderation failure, it is treating the alleged conduct as a potential violation of state consumer protection laws and statutes designed to protect minors from sexual exploitation. Bonta’s office has said it is examining whether xAI and X misled users about the safety and controls around Grok’s image tools and whether the companies failed to take reasonable steps to prevent foreseeable harm when users turned ordinary photos into explicit deepfakes. The investigation is also looking at whether the creation and distribution of images that appear to depict minors in sexual situations could violate criminal laws, a point underscored in the state’s description of how California Attorney General announced an investigation into how and whether Elon Musk’s X and xAI broke the law.
The stakes are high for Musk personally and for his AI venture. California officials have repeatedly named Elon Musk and his companies in public statements, making clear that they view the issue as a corporate governance problem as well as a technical one. Bonta’s cease-and-desist letter was addressed directly to Musk as chief executive of xAI and X, and it warned that the state could seek civil penalties and injunctive relief if the company does not comply. That pressure comes as Musk faces separate scrutiny over deepfake content on his platforms, including a lawsuit from Ashley St. Clair, the mother of one of Musk’s children, who sued xAI after Grok generated sexualized images of her and alleged that the model produced explicit content based on photos from when she was a child, according to a report noting that Ashley St. Clair, the mother of one of Musk’s children, also sued xAI after Grok generated sexualized images of her.
A national test case for AI and deepfake regulation
California’s confrontation with xAI is quickly becoming a national test case for how regulators will handle AI systems that can fabricate sexual content at scale. Bonta’s move to send a cease-and-desist to Musk’s company, and to give it a five day deadline to respond, is being closely watched by other states that are grappling with similar complaints about deepfake pornography and AI-generated abuse. The attorney general’s office has framed the action as part of a broader effort to protect consumers and children from emerging AI harms, and it has emphasized that the state will not wait for federal rules before acting, a stance reflected in coverage that describes how California Attorney General sent a cease-and-desist letter calling on Elon Musk’s xAI to halt creation of sexual deepfakes.
More from Morning Overview