A recent investigation identified close to 400 apps and websites, many tied to Chinese-language developers, that presented themselves as everyday AI photo or video editing tools while quietly offering features to generate non-consensual and violent sexual imagery. The apps looked unremarkable on the surface — the kind of utility a small business might stumble across while searching for a quick photo editor or video tool.
What happened
The apps in question were marketed as general-purpose AI image and video tools, the sort available by the dozens in app stores and through web search. Investigators found that behind their ordinary branding, many contained hidden menus or unlockable features that let users create sexually explicit deepfakes, including content depicting real people without their consent.
These tools reportedly operated for some time before detection, distributed across app stores, ad networks, and direct web links. Some used coded language or disguised interfaces to avoid triggering content moderation or app store review systems. The scale — hundreds of apps — suggests this wasn't the work of a single bad actor but a pattern replicated across many small developer operations.
The underlying technology isn't new. Deepfake generation tools have existed in various forms for years, and AI image models capable of photorealistic output are now widely accessible, including open-source versions anyone can run locally. What's notable here is the packaging: abusive functionality embedded inside apps designed to look mundane, likely to slip past app store screening and avoid scrutiny from users, advertisers, and payment processors.
App stores and ad platforms have faced this problem before, usually responding after the fact by removing flagged apps once reported. There's no indication yet of a systemic fix that catches this category of app before it reaches users, rather than after journalists or researchers flag it.
Why it matters
This fits a broader pattern in AI tool proliferation: as generative AI capabilities become cheaper and easier to embed in any app, the line between a legitimate utility and a harmful one is increasingly just a toggle switch or hidden menu. App store review processes, largely built for an earlier era of software, are being tested by tools that can generate content rather than simply process it.
The discovery also adds to a growing list of concerns about deepfake technology intersecting with everyday consumer software, following similar controversies involving mainstream apps that added or removed face-swap and image-generation features under public pressure.
What this means for small businesses
Most small businesses don't build AI apps, but many use off-the-shelf AI tools for marketing, photo editing, or content creation — often downloaded quickly based on app store rankings or search results rather than vetted reviews. This story is a reminder that an app's listed purpose and its actual feature set aren't always the same thing.
Businesses that use AI image or video editing tools, especially ones involving customer or employee photos, should consider sourcing tools from established vendors with transparent policies rather than unfamiliar apps with vague developer information. The reputational and legal risk of a tool being linked to abusive content, even tangentially, is not trivial — particularly for businesses in regulated industries or those serving minors.
It's also worth checking any AI tools already in use for red flags: unclear developer identity, features that seem disconnected from the app's stated purpose, or functionality that only appears after an update or unlock.
What to watch
Watch for how app stores respond — whether this triggers broader proactive screening changes or just a wave of app removals. Also watch for regulatory movement, since several jurisdictions have introduced or proposed laws specifically targeting non-consensual deepfake content, which could eventually create compliance obligations for platforms distributing AI tools.
The bottom line
Before adopting any AI editing or content-generation app for business use, verify the developer's identity, read recent user reviews for mentions of unexpected features, and favor established vendors with clear content policies over unfamiliar apps ranked highly in app store search results.