safetensors with model weight hashing
Project description
safemodels
Cryptographically-secure proof-of-concept for verifying the provenance of ML models.
This library is a thought experiment into what securing the supply chain of ML models could look like. It's built on top of safetensors
. You should probably read the blog post for more context!
Installation
$ pip install safemodels
Usage
Hashing
from safemodels import safe_hash
from huggingface_hub import hf_hub_download as dl
st = dl("gpt2", filename="model.safetensors")
pt = dl("gpt2", filename="pytorch_model.bin")
assert safe_hash(st) == safe_hash(pt) == 11799646609665420805
Signing
from safemodels import SafeModel
from huggingface_hub import hf_hub_download as dl
st = dl("gpt2", filename="model.safetensors")
sm = SafeModel.from_safetensor(st)
# or
st, sm = SafeModel.from_hf("gpt2", version="main")
sm.sign_safetensor(st) # backwards-compatible rewrite of file
Verification
>>> from safemodels import init, Issuer
>>> from huggingface_hub import hf_hub_download
>>>
>>> init(Issuer(identity="EleutherAI", issuer="https://auth.huggingface.com")
>>>
>>> hf_hub_download("EleuterAI/gpt-j-6B", filename="model.safetensors")
Downloading model.safetensors: 100%|███| 548M/548M [00:14<00:00, 39.2MB/s]
211it [00:00, 4785.46it/s]
Error: none of the expected identities matched what was in the certificate,
got subjects [EleuterAI] with issuer https://auth.huggingface.com
Traceback (most recent call last):
...
safemodels.InvalidSignature: Loaded a safetensor with an invalid signature!
safetensor
Metadata
from safemodels.utils.safetensors import extract_metadata, update_meta
from huggingface_hub import hf_hub_download as dl
st = dl("gpt2", filename="model.safetensors")
print(extract_metadata(st))
# {'format': 'pt'}
update_meta(st, {"hello": "world"})
print(extract_metadata(st))
# {'format': 'pt', 'hello': 'world'}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
safemodels-0.1.0.tar.gz
(6.6 kB
view hashes)
Built Distribution
Close
Hashes for safemodels-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ff5b86438c16bc8a48be17623a20fd227d9acc1ecc9d97d8bb8b0b717451fbd |
|
MD5 | 0dc9b7adca273a902f4982d0c933a6b2 |
|
BLAKE2b-256 | 3423f027e6eb1c4171c3d002b9eaae9fce2635c68cfa9da5636b2416c30917b0 |