Today OpenAI released theirs first threat report, detailing how actors in Russia, Iran, China, and Israel have attempted to operate the technology for foreign influence operations around the world. The report lists five different networks that OpenAI has identified and shut down in 2023-2024. In the report, OpenAI reveals that established networks such as Russia’s Doppleganger and China’s Spamoflauge are experimenting with using generative AI to automate their operations. They’re not very good at it either.
And while there may be some relief that these actors have not mastered generative AI to become unstoppable forces of disinformation, it is clear that they are experimenting and that in itself should be cause for concern.
OpenAI’s report reveals that influence campaigns are pushing the limits of generative AI, which does not reliably produce good text or code. He’s fighting with idioms— which makes the language sound more reliable and personal — and sometimes with basic grammar (to the point that OpenAI dubbed one network “Bad Grammar”). The Bad Grammar network was so sloppy that it once revealed its true identity: “As an AI language model, I am here to help and provide the desired commentary,” it wrote.
One network used ChatGPT to debug code that would allow it to automate posts on Telegram, a chat app that has long been a favorite of extremists and influence networks. Sometimes this worked well, but other times it led to two separate characters being entered into the same account, giving away the game.
In other cases, ChatGPT has been used to create code and content for websites and social media. For example, Spamoflauge used ChatGPT to debug code to create a WordPress site that published articles attacking members of the Chinese diaspora who were critical of the country’s government.
According to the report, AI-generated content failed to make it from the influence networks themselves to the mainstream, even when shared on widely used platforms such as X, Facebook and Instagram. This was the case with campaigns run by an Israeli company that apparently worked on commission and published content ranging from anti-Qatar to anti-BJP, the Hindu nationalist party that now controls the Indian government.
Overall, the report paints a picture of several relatively ineffective campaigns with crude, seemingly placatory propaganda many experts fear demonstrated the potential of this up-to-date technology to spread misinformation and disinformation, especially in a crucial election year.
However, social media influence campaigns often innovate over time to avoid detection, learning about the platforms and their tools, sometimes better than the platform workers themselves. While these initial campaigns may be diminutive or ineffective, they appear to still be in the experimental phase, says Jessica Walton, a researcher at the CyberPeace Institute who has studied Doppleganger’s operate of generative artificial intelligence.
In its research, the network used apparently real Facebook profiles to publish articles, often on divisive political topics. “The actual articles are written by generative AI,” he says. “And mostly they’re trying to see what will fly, what the Meta algorithms will catch and what they won’t be able to.”
In other words, expect things to only get better from here.