Virtual filesystem for SQLite to read from and write to S3
Project description
sqlite-s3vfs 
Virtual filesystem for SQLite to read from and write to S3
Installation
sqlite-s3vfs depends on APSW, which is not officially available on PyPI, but can be installed directly from GitHub.
pip install sqlite-s3vfs
pip install https://github.com/rogerbinns/apsw/releases/download/3.36.0-r1/apsw-3.36.0-r1.zip --global-option=fetch --global-option=--version --global-option=3.36.0 --global-option=--all --global-option=build --global-option=--enable-all-extensions
Usage
sqlite-s3vfs is an APSW virtual filesystem that requires boto3 for its communication with S3.
import apsw
import boto3
import sqlite_s3vfs
# A boto3 bucket resource
bucket = boto3.Session().resource('s3').Bucket('my-bucket')
# An S3VFS for that bucket
s3vfs = sqlite_s3vfs.S3VFS(bucket=bucket)
# sqlite-s3vfs stores many objects under this prefix
# Note that it's not typical to start a key prefix with '/'
key_prefix = 'my/path/cool.sqlite'
# Connect, insert data, and query
with apsw.Connection(key_prefix, vfs=s3vfs.name) as db:
cursor = db.cursor()
cursor.execute(f'''
CREATE TABLE foo(x,y);
INSERT INTO foo VALUES(1,2);
''')
cursor.execute('SELECT * FROM foo;')
print(cursor.fetchall())
# Get the serialized form of the sqlite file, say to upload to S3 as a single object
for chunk in s3vfs.serialize(key_prefix=key_prefix):
print(chunk)
See the APSW documentation for more examples.
Tests
The tests require the dev dependencies and APSW to installed, and MinIO started
pip install -r requirements-dev.txt
pip install https://github.com/rogerbinns/apsw/releases/download/3.36.0-r1/apsw-3.36.0-r1.zip --global-option=fetch --global-option=--version --global-option=3.36.0 --global-option=--all --global-option=build --global-option=--enable-all-extensions
./start-minio.sh
can be run with pytest
pytest
and finally Minio stopped
./stop-minio.sh
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
sqlite-s3vfs-0.0.5.tar.gz
(3.6 kB
view hashes)
Built Distribution
Close
Hashes for sqlite_s3vfs-0.0.5-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 0a0e09500008885b01feaa42dd3a85e57c4bb09cbb48a51964302c5ede18f0ec |
|
| MD5 | c6423afb502fb135dd9c12dbe6c77cf6 |
|
| BLAKE2b-256 | cb82f9b1327678398783192e6413952cde5e679592717e81b1afbd208af72cdb |
