A collection of Python utility functions for ingesting data into SQLAlchemy-defined PostgreSQL tables, automatically migrating them as needed, and minimising locking
Project description
pg-bulk-ingest
A Python utility function for ingesting data into SQLAlchemy-defined PostgreSQL tables, automatically migrating them as needed, allowing concurrent reads as much as possible.
Allowing concurrent writes is not an aim of pg-bulk-ingest. It is designed for use in ETL pipelines where PostgreSQL is used as a data warehouse, and the only writes to the table are from pg-bulk-ingest. It is assumed that there is only one pg-bulk-ingest running against a given table at any one time.
Features
pg-bulk-ingest exposes a single function as its API that:
- Creates the tables if necessary
- Migrates any existing tables if necessary, minimising locking
- Ingests data in batches, where each batch is ingested in its own transaction
- Handles "high-watermarking" to carry on from where a previous ingest finished or errored
- Optionally performs an "upsert", matching rows on primary key
- Optionally deletes all existing rows before ingestion
- Optionally calls a callback just before each batch is visible to other database clients
Visit the pg-bulk-ingest documentation for usage instructions.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pg_bulk_ingest-0.0.51-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | d7848f463d7ead6e30a6704b0fb518e87a9f5f2863e811a8cd728e69555571fd |
|
| MD5 | 4c46bbe4d645830976a521ea856574cd |
|
| BLAKE2b-256 | 4e2df3d5c28d3786c6c16c620d352f771beb44a3f13f4fc0f1ec1b3c6fc3f9f7 |
