.){kind=link}
With the growing popularity of generative AI, many open-source models are now available on the Internet for anyone to adapt to their tasks, such as generating product renderings in a specific artistic style.
However, these models also fall into the hands of nefarious actors who can optimize them to create illegal content such as hate speech or child sexual abuse material (CSAM). This is a growing problem – National Center for Missing and Exploited Children received over 1.5 million reports AI-generated CSAM in 2025, up from 67,000 in 2024.
Engineers typically test AI for malicious capabilities by monitoring the model and checking its results, but with CSAM this is impossible because in the US it is illegal to generate such content, regardless of intent.
To avoid this dilemma and improve AI security, a team of MIT researchers, led by graduate student Vinith Suriyakumar and associate professors Ashia Wilson and Marzyeh Ghassemi, teamed up with researchers from Thorn develop a recent audit approach that determines whether a model can generate CSAM without prompting it. Thorn is a nonprofit child safety organization with a mission to transform how children are protected from sexual exploitation and abuse in the digital age.
Their technique examines how the inner workings of the model have been adapted, but never produces results. By examining hidden representations, it can be reliably inferred whether a model has been specialized to create malicious images.
During testing, the audit procedure identified model variations that specialized in generating CSAM with 100% accuracy. The hosting platform can exploit this technique to flag unsafe models and remove them quickly or prevent them from being uploaded in the first place.
“This opens up new opportunities for platforms supporting open-source models, and for law enforcement to actually test whether a model can generate CSAM. Previously, we had no way to measure this. It was a huge blind spot that some took advantage of. Now we can solve an AI security problem that has serious negative consequences,” says Vinith Suriyakumar, an electrical engineering and computer science (EECS) graduate student at MIT and lead author of a paper on the technique.
Suriyakamura and Wilson, the Lister Borthers Career Development Professor at EECS and principal investigator at the Laboratory for Information and Decision Systems (LIDS), are joined on the paper by Lena Stempfle, a postdoc at MIT; Ghassemi, associate professor at EECS and member of the Institute of Medical Engineering Sciences (IMES) and LIDS; and others at Boston University and Thorn. The paper was presented in the spotlight at the Trustworthy AI for Good workshop at the International Machine Learning Conference.
Adaptation audit
Recent techniques have made it easier for users to specialize a generative AI model for their tasks through a process known as tuning.
Instead of retraining the entire model on a task-specific dataset, individuals can exploit an algorithm called low-rank adaptation (LoRA) to specialize the model in a more competent way.
This has led to a wave of recent variants of generative AI models for various purposes, such as creating watercolor paintings that mimic artistic movement. However, it also allowed malicious actors to create models that can generate high-quality CSAM and other malicious images.
To audit a model, engineers typically query it about malicious content and check the results, but this manual auditing procedure is not scalable. Furthermore, the repeated generation of disgusting images may have a negative psychological impact on evaluators.
This evaluation method quickly falls apart when testing CSAM, which is illegal to generate for any purpose in the US and many other international jurisdictions.
“We are in a very difficult situation where, by law alone, we cannot apply de facto assessment measures. We have had to throw away the entire toolkit and take a different approach,” says Suriyakumar.
After learning about this puzzle, researchers teamed up with Thorn to solve the problem.
Non-generative solution
Instead of focusing on the results, researchers focused on the modifications made by the LoRA algorithm during tuning.
Their technique examines these modifications, called LoRA adapters, to determine whether the model has been specialized for malicious features, without generating results.
Using a technique called Gaussian probing, researchers feed the model a set of random data points and analyze how it manipulates that data within its multi-layered internal structure.
“We never run the model all the way or prompt it, so we never generate images,” explains Suriyakumar.
Scientists capture these modifications at multiple time points of the model’s internal structure and average them to summarize how the LoRA adapter changed the model’s calculations. They found that these responses provided a sturdy signal of model specialization.
They tested their method on variations of three types of models, comparing the results with real data from LoRA adapters known to generate CSAM, other malicious images, and safe and sound content.
Their method was 100% exact in identifying models suited to generating CSAM.
“There are a lot of child safety concerns around AI, and these are real concerns that need to be addressed. Many children are harmed by AI fakes. We’ve shown that Gaussian probing can be a very useful tool, and we hope that the research community will really pay more attention to this issue,” Wilson says.
Importantly, their technique is scalable and would be relatively inexpensive to implement. With thousands of model variations published on the Internet every month, scalability is crucial because it helps auditors remove harmful adaptations before they are widely distributed.
Gaussian probing is also more stalwart than other auditing techniques because a nefarious actor would have to carefully alter the inner workings of the underlying model to avoid detection.
In the future, the researchers want to evaluate their technique on a larger set of model variations and investigate whether Gaussian probing can detect malicious features in base models before they are adapted.
“We now have a technological approach that can partially solve this problem. So much effort has gone into this collaboration, which has allowed us to tackle a really difficult problem that is harming so many children in this country and around the world. Hopefully, we can have a transformative impact in this area,” says Ghassemi.
This work was supported in part by a Bridgewater AIA Labs Research Fellowship.