Skip to main content

OFS - provides plugin-orientated low-level blobstore.

Project description

OFS is a bucket/object storage library.

It provides a common API for doing storing bitstreams (+ a bit of related metadata) in ‘bucket/object’ stores such as:

  • Filesystem (+ pairtree)

  • S3-compatible: S3, Google Storage, Eucalytus, Archive.org

  • Riak (buggy)

  • ‘REST’ Store (see remote/reststore.py - implementation at http://bitbucket.org/pudo/repod/)

  • add a backend here

Why use the library:

  • Abstraction: write common code but use different storage backends

  • More than a filesystem, less than a database - support for metadata as well bitstreams

  • Extra features:

    • Sharding - automatic sharding of files to support distributed file-system like structure (planned)

Example Usage

(local version - depends on ‘pairtree’, and ‘simplejson’):

 >>> from ofs.local import OFS

 >>> o = OFS()
 (Equivalent to 'o = OFS(storage_dir = "data", uri_base="urn:uuid:", hashing_type="md5")')

 # Claim a bucket - this will add the bucket to the list of existing ones
 >>> uuid_id = o.claim_a_bucket()
 >>> uuid_id
 '4aaa43cdf5ba44e2ad25acdbd1cf2f70'

 # Choose a bucket name - if it exists, a new UUID one will be formed instead and returned
 >>> bucket_id = o.claim_a_bucket("foo")
 >>> bucket_id
 'foo'
 >>> bucket_id = o.claim_a_bucket("foo")
 >>> bucket_id
 '1bf93208521545879e79c13614cd12f0'

 # Store a file:
 >>> o.put_stream(bucket_id, "foo.txt", open("foo....))
 {'_label': 'foo.txt', '_content_length': 10, '_checksum': 'md5:10feda25f8da2e2ebfbe646eea351224', '_last_modified': '2010-08-02T11:37:21', '_creation_date': '2010-08-02T11:37:21'}

 # or:
 >>> o.put_stream(bucket_id, "foo.txt", "asidaisdiasjdiajsidjasidji")
 {'_label': 'foo.txt', '_content_length': 10, '_checksum': 'md5:10feda25f8da2e2ebfbe646eea351224', '_last_modified': '2010-08-02T11:37:21', '_creation_date': '2010-08-02T11:37:21'}

 # adding a file with some parameters:
 >>> o.put_stream(bucket_id, "foooo", "asidaisdiasjdiajsidjasidji", params={"original_uri":"http://...."})
 {'_label': 'foooo', 'original_uri': 'http://....', '_last_modified': '2010-08-02T11:39:11', '_checksum': 'md5:3d690d7e0f4479c5a7038b8a4572d0fe', '_creation_date': '2010-08-02T11:39:11', '_content_length': 26}

 # Get the underlying URL pointing to a resource
 >>> o.get_url(bucket_id, "foo")
   [typical local pairtree response:]
"file:///opt/ofs_store/pairtree_root/1b/f9/32/......./obj/foo"
   [typical remote response]
"http://..."
"ftp://..."

 # adding to existing metadata:
 >>> o.update_metadata(bucket_id, "foooo", {'foo':'bar'})
 {'_label': 'foooo', 'original_uri': 'http://....', '_last_modified': '2010-08-02T11:39:11', '_checksum': 'md5:3d690d7e0f4479c5a7038b8a4572d0fe', '_creation_date': '2010-08-02T11:39:11', '_content_length': 26, 'foo': 'bar'}

 # Remove keys
 >>> o.remove_metadata_keys(bucket_id, "foooo", ['foo'])
 {'_label': 'foooo', 'original_uri': 'http://....', '_last_modified': '2010-08-02T11:39:11', '_checksum': 'md5:3d690d7e0f4479c5a7038b8a4572d0fe', '_creation_date': '2010-08-02T11:39:11', '_content_length': 26}

 # Delete blob
 >>> o.exists(bucket_id, "foooo")
 True
 >>> o.del_stream(bucket_id, "foooo")
 >>> o.exists(bucket_id, "foooo")
 False

 # Iterate through ids for buckets held:
 >>> for item in o.list_buckets():
 ...   print item
 ...
 447536aa0f1b411089d12399738ede8e
 4a726b0a33974480a2a26d34fa0d494d
 4aaa43cdf5ba44e2ad25acdbd1cf2f70
 .... etc

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ofs-0.1.tar.gz (10.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page