skip to navigation
skip to content

Not Logged In

s3 0.0.2

Python module which connects to Amazon's S3 REST API

Package Documentation

Latest Version: 3.0.0


s3 is a connector to S3, Amazon’s Simple Storage System REST API.

Use it to upload, download, delete, copy, test files for existence in S3, or update their metadata.

S3 files may have metadata in addition to their content. Metadata is a set of key/value pairs. Metadata may be set when the file is uploaded or it can be updated subsequently.


From PyPi

$ pip install s3

From source

$ hg clone ssh://
$ pip install -e s3

The installation is successful if you can import s3. The following command must produce no errors:

$ python -c 'import s3'

API to remote storage

S3 Filenames

An S3 file name consists of a bucket and a key. This pair of strings uniquely identifies the file within S3.

The S3Name class is instantiated with a key and a bucket; the key is required and the bucket defaults to None.

The RemoteStore class methods take a remote_name argument which can be either a string which is the key, or an instance of the S3Name class. When no bucket is given (or the bucket is None) then the default_bucket established when the connection is instantiated is used. If no bucket is given (or the bucket is None) and there is no default bucket then a ValueError is raised.

In other words, the S3Name class provides a means of using a bucket other than the default_bucket.

Headers and Metadata

Additional http headers may be sent using the methods which write data. These methods accept an optional headers argument which is a python dict. The headers control various aspects of how the file may be handled. S3 supports a variety of headers. These are not discussed here. See the S3 documentation for more information on S3 headers. Those headers whose key begins with the special preifx: ‘x-amz-meta-‘ are considered to be metadata headers and are used to set the metadata attributes of the file.

The methods which read files also return metadata which consists of only those response headers which begin with ‘x-amz-meta-‘.


The arguments remote_source, remote_destination, and remote_name may be either a string, or an S3Name instance.

local_name is a string and is the name of the file on the local system. This string is passed directly to open().

headers is a python dict used to encode additional request headers.

ok = store.copy(remote_source, remote_destination, headers={})
Copy from remote source to remote destination. If there are no metadata headers in headers, the destination metadata is copied from the source metadata, otherwise it is the metadata in headers.
ok = store.delete(remote_name)
Delete file from remote store.
exists, metadata = store.exists(remote_name)
Test if file exists in remote store and retrieve its metadata if it does.
ok, metadata =, local_name)
Download a file from remote store and retrieve its metadata.
ok = store.update_metadata(remote_name, headers)
Update a remote file’s metadata.
ok = store.write(local_name, remote_name, headers={})
Upload a file to remote store (and possibly set its metadata).

When a request is sent, the requests module may raise an exception or may return status. The status could indicate success or failure. So there are 2 kinds of failure.

All methods except exists catch connection exceptions, log either type of failure, and return True for success, and False for failure.

The exists method does not catch connection exceptions and may also raise RemoteStoreError. It returns True or False as the remote_name exists in remote storage or not.

copy, update_metadata, and write accept a headers argument which is used to provide additional headers to the request.

exists returns a tuple whose first element indicates if the file exists or not, and whose second element is the file’s metadata when the file exists, and an empty dict otherwise.

read returns a tuple whose first element indicates success or failure and whose second element is the file’s metadata on success, and an empty dict otherwise.



First configure your yaml file.

  • access_key_id and secret_access_key are generated by the S3 account manager. They are effectively the username and password for the account.
  • bucket is the name of the default bucket to use when referencing S3 files. bucket names must be unique (on earth) so by convention we use a prefix on all our bucket names: com.prometheus.
  • endpoint is the Amazon server url to connect to. See for a list of the available endpoints.
  • tls True => use https://, False => use http://. Default is True.

Here is an example s3.yaml

    access_key_id: "XXXXX"
    secret_access_key: "YYYYYYY"
    default_bucket: "ZZZZZZZ"
    endpoint: ""

Next configure your S3 bucket permissions. Eventually, s3 will support bucket management. Until then use Amazon’s web interface:

  • Log onto your Amazon account.
  • Create a bucket or click on an existing bucket.
  • Click on Properties.
  • Click on Permissions.
  • Click on Edit Bucket Policy.

Here is a example policy with the required permissions:

        "Version": "2008-10-17",
        "Id": "Policyxxxxxxxxxxxxx",
        "Statement": [
                        "Sid": "Stmtxxxxxxxxxxxxx",
                        "Effect": "Allow",
                        "Principal": {
                                "AWS": "arn:aws:iam::xxxxxxxxxxxx:user/XXXXXXX"
                        "Action": [
                        "Resource": [


Once the yaml file is configured and the bucket policy is set, you can instantiate a S3Connection and you use that connection to instantiate a RemoteStore.

import s3
import yaml

with open('s3.yaml', 'r') as fi:
    config = yaml.load(fi)

connection = s3.S3Connection(**config['s3'])
store = s3.RemoteStore(connection)

Then you call methods on the RemoteStore instance.

The following code uploads a file named “example” from the local filesystem as “example-in-s3” in s3. It then checks that “example-in-s3” exists in storage, downloads the file as “example-from-s3”, compares the original with the downloaded copy to ensure they are the same, deletes “example-in-s3”, and finally checks that it is no longer in storage.

import subprocess
assert store.write("example", "example-in-s3")
exists, metadata = store.exists("example-in-s3")
assert exists
ok, metadata ="example-in-s3", "example-from-s3")
assert ok
assert 0 ==['diff', "example", "example-from-s3"])
assert store.delete("example-in-s3")
exists, metadata = store.exists("example-in-s3")
assert not exists

The following code again uploads “example” as “example-in-s3”. This time it uses the bucket “my_other_bucket” explicitly, and it sets some metadata and checks that the metadata is set correctly. Then it changes the metadata and checks that as well.

headers = {
    'x-amz-meta-state': 'unprocessed',
remote_name = s3.S3Name("example-in-s3", bucket="my_other_bucket")
assert store.write("example", remote_name, headers=headers)
exists, metadata = store.exists(remote_name)
assert exists
assert metadata == headers
headers['x-amz-meta-state'] = 'processed'
assert store.update_metadata(remote_name, headers)
ok, metadata =, "example-from-s3")
assert ok
assert metadata == headers
File Type Py Version Uploaded on Size
s3-0.0.2.tar.gz (md5) Source 2014-05-28 16KB
  • Downloads (All Versions):
  • 38 downloads in the last day
  • 290 downloads in the last week
  • 1261 downloads in the last month