In its latest security disclosures, OpenAI announced the suspension of multiple accounts tied to China that were using its language models like ChatGPT to monitor and extract data from social media. One of the suspended accounts reportedly assisted in drafting promotional content and business strategies for a listening tool that scraped platforms including X, Facebook, Instagram, Reddit, TikTok, and YouTube.

These accounts focused on tracking political, religious, or ethnic content, and it is believed that the operations were conducted on behalf of Chinese government-affiliated entities — though OpenAI has not independently verified the identities of those actors.

Other Malicious Usages Identified

Tracking Uyghur individuals: Accounts were discovered that aimed to build early-warning systems to track the movements of persons associated with the Uyghur minority, a group under scrutiny for alleged human rights abuses.

Developing malware: OpenAI detected accounts run by Russian-, Korean-, and Chinese-speaking actors using ChatGPT to generate malicious code or components.

Organized fraud networks: Clusters operating in Cambodia, Myanmar, and Nigeria were disabled, as they used AI-generated messages and campaigns intended to deceive individuals online.

Ongoing Efforts Against Influence Campaigns

This action forms part of OpenAI’s series of threat reports since February 2024. The company has repeatedly sht down AI-powered influence campaigns worldwide. Over the past summer, it intervened in multiple countries where AI-generated content was used to produce posts and comments meant to sow division and amplify engagement, domestically and internationally.

Despite these dangerous uses, OpenAI notes that ChatGPT is used three times more often to detect fraud than to facilitate it.