DLT is an open-source python-native scalable data loading framework that does not require any devops efforts to run.
Project description
Follow this quick guide to implement DLT in your project
Simple loading of one row:
Install DLT
DLT is available in PyPi and can be installed with pip install python-dlt. Support for target warehouses is provided in extra packages:
pip install python-dlt[redshift] for Redshift
pip install python-dlt[gcp] for BigQuery
Create a target credential
credential = {'type':'redshift',
'host': '123.456.789.101'
'port': '5439'
'user': 'loader'
'password': 'dolphins'
}
Initialise the loader with your credentials and load one json row
import dlt
loader = dlt(credential)
json_row = "{"name":"Gabe", "age":30}"
table_name = 'users'
loader.load(table_name, json_row)
Loading a nested json object
import dlt
loader = dlt(credential)
json_row = "{"name":"Gabe", "age":30, "id":456, "children":[{"name": "Bill", "id": 625},
{"name": "Cill", "id": 666},
{"name": "Dill", "id": 777}
]
}"
table_name = 'users'
#unpack the nested json. To be able to re-pack it, we create the parent - child join keys via row / parent row hashes.
rows = loader.utils.unpack(table_name, json_row)
# rows are a generator that outputs the parent or child table name and the data row such as:
#("users", "{"name":"Gabe", "age":30, "id":456, "row_hash":"parent_row_md5"}")
#("users__children", "{"name":"Bill", "id":625, "parent_row_hash":"parent_row_md5", "row_hash":"child1_row_md5"}")
#("users__children", "{"name":"Cill", "id":666, "parent_row_hash":"parent_row_md5", "row_hash":"child2_row_md5"}")
#("users__children", "{"name":"Dill", "id":777, "parent_row_hash":"parent_row_md5", "row_hash":"child3_row_md5"}")
#loading the tables users, and users__children
for table, row in rows:
loader.load(table_name, row)
#to recreate the original structure
select users.*, users__children.*
from users
left join users__children
on users.row_hash = users__children.parent_row_hash
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
python-dlt-0.1.0rc1.tar.gz
(90.2 kB
view hashes)
Built Distribution
python_dlt-0.1.0rc1-py3-none-any.whl
(121.7 kB
view hashes)
Close
Hashes for python_dlt-0.1.0rc1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | cecf0038ab7de1daa7498fd7c4acc23b38c9905b213180067bb177144038c71e |
|
| MD5 | cbdc3e4647eaf1f78a80629a2f129b07 |
|
| BLAKE2b-256 | c20fb5c52beceb3ffa5e495cfd7797e0511fa800e9afcbd30175a08076704857 |
