Cyberattacks threaten to make open-source AI a little less open, Hugging Face exec warns

The very qualities that make open AI platforms valuable, like open doors and rich data, might also make them vulnerable to more hacks, Irene Solaiman, chief policy officer and SVP of strategy at Hugging Face said on a recent episode of the Equity podcast.

The October attack on OpenCode, which targeted users’ third-party API keys and caused financial damage, is the latest example. It follows an incident this summer in which an OpenAI agent swarm breached Hugging Face in an apparent attempt to cover its tracks after accessing answers to cybersecurity evaluations.

The risks aren’t limited to platforms that invite open community contributions. On Monday, AI infrastructure startup Fireworks, which helps developers run and train open AI models, disclosed that attackers had accessed some customers’ model weights, datasets, and billing information. The company temporarily disabled certain self-service dataset functions as it investigates. While the attack isn’t necessarily due to Fireworks’ openness, it highlights how much valuable information is concentrated within AI infrastructure providers. 

“The things I worry about making an open community targeted are also what makes it special,” Solaiman said on Equity. “We are necessarily permissive. We want the average hobbyist in their basement who doesn’t have an institutional affiliation to be able to contribute.”

“You want people to be able to just submit pull requests and allow anybody without a valid affiliation to contribute…but then you have some losers who ruin it for the rest,” she added in a separate conversation.

That permissiveness isn’t the only thing drawing attackers. The wealth of information open platforms host also make them targets. Hugging Face became a temptation for OpenAI’s agent swarm because of the evaluation datasets it holds, whereas OpenCode’s target in this case came from its “juicy third-party API keys,” per Solaiman. 

“The combination of that makes us prime targets,” Solaiman told TechCrunch in a separate conversation. She added that there’s a “danger of open source really closing down at a time when we’ve gotten so much public support.”

Much of the industry has come out recently in support of open AI models, including heavy hitters like Nvidia’s Jensen Huang. And increasingly, governments, enterprises and startups are relying on open models to build. 

So what would a pullback actually look like? It could theoretically mean restrictions on who can upload models or datasets, more stringent identity verification, or additional vetting of developers before they can access certain features. While these moves could protect platforms from attackers, it could also make it more difficult for independent researchers and hobbyists to participate in communities they would have been welcomed in before. 

For now, Hugging Face is trying to avoid that route. Solaiman said the company has been beefing up monitoring tools to catch hacks, as well as taking steps to prevent them. But monitoring can be expensive to maintain, she noted, and not every platform will be able to afford it. 

“Administrative access tokens were part of what gave that agent swarm better access, so we just rotate them a lot more frequently than we used to,” Solaiman said. “We also now have stricter network controls.”

In other words, the credentials that give admins broad access to Hugging Face’s systems now get changed more often, which means a stolen token would be useful for less time. 

Other parts of the open source ecosystem are dealing with similar security-related tradeoffs. GitHub, the world’s largest host for open-source projects, has been fighting off attackers who inject malicious code into projects to quickly spread malware. The company began tightening security requirements for publishing packages to the npm registry, revoking older authentication tokens, and limiting how long new ones remain valid. 

Zach Steindler, an engineer at GitHub who handles software supply chain security, told TechCrunch via email that while security measures can disrupt established workflows, the platforms can become more secure without fundamentally restricting participation. 

“It’s less about limiting functionality and more about going and fixing things that weren’t secure in the first place,” Steindler said. He noted that npm install scripts have existed for years, but recent attacks highlighted how dangerous they could be. GitHub worked with OpenJS — a nonprofit that hosts critical open-source projects built with JavaScript — on a safer approach.

The same logic applies to where security teams put their effort, per Steindler: “Security is all about maximizing impact with the resources we have, and I would much rather invest in detecting suspicious behavior and containment capabilities: things that cover all accounts and access methods. Suspicious behavior is suspicious regardless of who is exhibiting it.” 

When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.

Rebecca Bellan is a senior reporter at TechCrunch where she covers the business, policy, and emerging trends shaping artificial intelligence. Her work has also appeared in Forbes, Bloomberg, The Atlantic, The Daily Beast, and other publications.

You can contact or verify outreach from Rebecca by emailing [email protected] or via encrypted message at rebeccabellan.491 on Signal.