Security researchers Protect AI have found tens of thousands of ‘malicious’ models on Hugging Face, the Github for artificial intelligence.
© 2023 Bloomberg Finance LP
Hugging Face, the leading online repository for generative AI, has hosted tens of thousands of models containing hidden code that can poison data and steal information, including the tokens used to pay AI and cloud operators, according to security researchers.
Researchers from security startups ProtectAI, Hiddenlayer and Wiz have been warning for months that hackers have been uploading such “malicious models” to the Hugging Face site, where more than a million models are now available for download.
“The old Trojan horse computer viruses that tried to sneak malicious code onto your system evolved for the AI age,” said Ian Swanson, CEO and founder of Protect AI. The Seattle, Washington-based startup discovered tens of thousands of malicious models when it started scanning Hugging Face earlier this year.
Some of these bad actors even set up fake Hugging Face profiles to pose as Meta or other tech companies to lure downloads from the unwary, Swanson said. A scan of Hugging Face revealed dozens of fake accounts posing as companies such as Facebook, Visa, SpaceX and Swedish telecom giant Ericsson.
One model, which falsely claimed to be from the genomics testing startup 23AndMe, was downloaded thousands of times before it was noticed, Swanson said. He warned that the malicious code hidden in the fake 23AndMe model, when installed, would silently search for AWS passwords, which could be used to steal cloud computing resources. Hugging Face removed the model after being made aware of the risk.
Hugging Face has now integrated ProtectAI’s tool, which scans for malicious code, into its platform and lets users see the results before downloading anything.
The company said Forbes it has verified the profiles of major companies like OpenAI and Nvidia as of 2022. In November 2021, it started scanning the files commonly used to train machine learning models on the platform. “We hope that our work and collaboration with Protect AI, and hopefully many more, will help build greater trust in machine learning artifacts to make sharing and adoption easier,” said Julien Chaumond, CTO of Hugging Face in a statement email to Forbes.
The risk from malicious models is high enough to warrant a joint warning in April from the US Cybersecurity and Infrastructure Security Agency and the Canadian and British security services. The NSA and its British and Canadian counterparts warned companies to scan pre-trained models for dangerous code, then only redirect them away from critical systems.
The hackers who target Hugging Face typically inject rogue instructions into the code that developers download from the site, and use them to hijack the model when it is executed by an unsuspecting target. “These are classic attacks, but they’re just hidden in models,” Swanson said. “No one would know that the model is doing these nefarious things and it would be incredibly difficult for them to trace back.”
Hugging Face was last valued at $4.5 billion when it raised $235 million in August 2023. The eight-year-old startup, founded by Clément Delangue, Julien Chaumond and Thomas Wolf, went from running a teen-targeted chatbot app to a machine learning platform in 2018. It has raised $400 million to date and is becoming the Github for AI researchers.
“For a long time, AI was a research area and security practices were quite basic,” says Chaumond. “As our popularity grows, so does the number of potentially bad actors who may want to target the AI community.”
