Arrow -> PostgreSQL encoder
Project description
pgpq
Convert PyArrow RecordBatches to Postgres' native binary format.
Usage
import psycopg
import pyarrow.dataset as ds
from pgpq import ArrowToPostgresBinaryEncoder
dataset = ds.dataset("tests/testdata/")
encoder = ArrowToPostgresBinaryEncoder(dataset.schema)
# get the expected postgres destination schema
# note that this is _not_ the same as the incoming arrow schema
# and not necessarily the schema of your permanent table
# instead it's the schema of the data that will be sent over the wire
# which for example does not have timezones on any timestamps
pg_schema = encoder.schema()
# assemble ddl for a temporary table
# it's often a good idea to bulk load into a temp table to:
# (1) Avoid indexes
# (2) Stay in-memory as long as possible
# (3) Be more flexible with types (you can't load a SMALLINT into a BIGINT column without casting)
cols = [f"\"{col['name']}\" {col['data_type']['ddl']}" for col in pg_schema["columns"]]
ddl = f"CREATE TEMP TABLE data ({','.join(cols)})"
with psycopg.connect("postgres://postgres:postgres@localhost:5432/postgres") as conn:
with conn.cursor() as cursor:
cursor.execute(ddl) # type: ignore
with cursor.copy("COPY data FROM STDIN WITH (FORMAT BINARY)") as copy:
copy.write(encoder.write_header())
for batch in dataset.to_batches():
copy.write(encoder.write_batch(batch))
copy.write(encoder.finish())
# load into your actual table, possibly doing type casts
# cursor.execute("INSERT INTO \"table\" SELECT * FROM data")
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pgpq-0.4.1.tar.gz
(38.1 MB
view hashes)
Built Distributions
pgpq-0.4.1-cp37-abi3-win_amd64.whl
(350.4 kB
view hashes)
pgpq-0.4.1-cp37-abi3-win32.whl
(325.5 kB
view hashes)
Close
Hashes for pgpq-0.4.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c1547eef318c08a019125b544f4e703de030f0872e017205b9d6aac6de59407 |
|
MD5 | 179626a8fcf3b47d2f171673d20aa935 |
|
BLAKE2b-256 | 02a8c9236906233b3f97cd685f365dc2b66afbba9ffd4bf2aa52a086e36bc52a |
Close
Hashes for pgpq-0.4.1-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7adbc63692e9a047accd9ff4a0bf8711e3ba077f32a447259e726c39ce4c335a |
|
MD5 | db676bb3e5777b659b2f1e71f1ad8c28 |
|
BLAKE2b-256 | c866830a03b807b68bce557dbc6962684efaee5f33631c5d43267f64f25f4e14 |
Close
Hashes for pgpq-0.4.1-pp39-pypy39_pp73-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 045bc0bcfc76e052128bc4b7af4c231e8878eac677add238cf7638e3fbd6e7b6 |
|
MD5 | ec741e116777866a3f77f0fa676fad28 |
|
BLAKE2b-256 | bc058aab976f5833ed6a6be26a6e764bd1057f4224cf838f57b9ad45a4aacacc |
Close
Hashes for pgpq-0.4.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1faf17e95ea84f83dc4c9cdfdf5a5a536b938d9308c1f6dc1392f7c7249eebcd |
|
MD5 | 4ac41c8c3446b99849cc9d713ce7d2a0 |
|
BLAKE2b-256 | 3b767bc91c57696cd219baaa56013ba0f62472a04121c4f594de24a6597c9185 |
Close
Hashes for pgpq-0.4.1-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 902689eac0440745c26a89524f561f14ab6cce2144e97d785e256aedcaa978ef |
|
MD5 | 6525847772a07f13972a107616baa469 |
|
BLAKE2b-256 | efbc4c69d5c382467d2a2af22ee1b1b8cad079b5a54f07624f829af9259556d8 |
Close
Hashes for pgpq-0.4.1-pp38-pypy38_pp73-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9bf21a682913b3344d8af72ccc0420cfa7ff98ed1287bedd1408e2c3067effc5 |
|
MD5 | d2daae9d1f22106b929583349660f04d |
|
BLAKE2b-256 | eae59b11386acc5970a4011de5107fee4dd6b91b95723ca666813d8547f070c6 |
Close
Hashes for pgpq-0.4.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d2697591c53a8995768b9de12757c595dbba455efb8bdfd14984212fa141872 |
|
MD5 | e21aa4bf1c6beff3834dcd1bbaffaed7 |
|
BLAKE2b-256 | 98ceb8b5c3f60d6ebb2018a3a5ef0436860ab96da5b7572fbce59dae32955f96 |
Close
Hashes for pgpq-0.4.1-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 502a2263be2b042c7e551de9a7ba56e3e05da5a161ea7b15dcd28d1ce7b33f30 |
|
MD5 | 68c0ac201dbdbe02262710063ddda134 |
|
BLAKE2b-256 | 6d3b1b33bb98e3eb38e1d062f3cf3389c7c644b8f6bd27d0a42dd6d9498f733b |
Close
Hashes for pgpq-0.4.1-pp37-pypy37_pp73-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f86c9f14c408494ca04e3a97085d309bb0cdfb3a0b0685ff4bb27e2309aae789 |
|
MD5 | 551539b97ff818a0d5d324279e656a58 |
|
BLAKE2b-256 | 9b62253e5c67909c31464636cfcbfd0d6441e4e2834a46946b9559d508f189c9 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d9331741360506485e882ced61d4ee61ae01adfbb52d15c3d7e6aab89fdb8c2 |
|
MD5 | 3114c7abaaa42be39a19a48eb0ab8f30 |
|
BLAKE2b-256 | c845f291972ae0e22b62cf402d29ec2a24cab695f019da52141e4a9bc61cf174 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e0340f545c64b567482259d4768de27cda73b75a75f26a2ce49201253f19633d |
|
MD5 | 66105e3995330c08a0a0bf8f252f50fe |
|
BLAKE2b-256 | c328f0e5daa1c4e7927077795743bb523b760dabc5fcd81b506b15429a917a25 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-musllinux_1_2_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52a247c477b555832edeb1f483a99b655c7bf00e554cd330d5ebf136ea0e2027 |
|
MD5 | 63a8d7ceba4ab07fc29c7e0f1e161892 |
|
BLAKE2b-256 | fb52385daa49e2cf03e4b6ef36e6666e92d58e273d41301c70f1f10de6adc1a9 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-musllinux_1_2_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 39d36b50673f365c55d1bf015841b0c095abb065328cab17ef06a5685d9b586d |
|
MD5 | aefc817dde10f0cc8390b97e90b13ef2 |
|
BLAKE2b-256 | c5f5444fc9a63ebc9f4591d8e9e7584034964e2eae39f886a998063f187cf931 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-musllinux_1_2_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1c7014b9739d55636fa816629fbc59587fd656cb35f2b60f0aa7246d87acd8fc |
|
MD5 | 7897e07ce347f16210584df712234444 |
|
BLAKE2b-256 | 4870b103ad163f5ea6eecd2791bb04f712e98dbdd9c2d710f9ba4a6193ceae26 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ee6a41dbdda0b70f983959e4f15a3a9c654438a675ad1733d989157a2fb3791 |
|
MD5 | 831572cbf8ea261ef588f7553a2dfea1 |
|
BLAKE2b-256 | 825368e251ece3052bae1e41f6ad74b2b4cf055a9255a62f725926b4dabae03e |
Close
Hashes for pgpq-0.4.1-cp37-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5b2b35b04dbccc2b5fd7ec5b59e9fbef0a931bdc1247ecb49c9084a67e28062 |
|
MD5 | 9d109b92e739fcc16219ee6612f024c0 |
|
BLAKE2b-256 | ac7c497dd6b7b93b84198b2d27bb314cec89718c9ec43f1f52e77319173ddd95 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f60fa05d382ce3cc601819666d803b8d59505f85a8adbea7104bd5d614b1812a |
|
MD5 | 38a4eed44d55d0644c0a779df54ee737 |
|
BLAKE2b-256 | 9d6e9634b934a59123fbe504fdbb0ed51e14bc5f8364df631a84118c5d952923 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-manylinux_2_17_ppc64.manylinux2014_ppc64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82e585f5c1591e18539ab1e3e53f90051826fe6afb332768289599e4d16a8d8f |
|
MD5 | a2fb5937e602237be706e1d18458b5aa |
|
BLAKE2b-256 | 9eb986a45e979714c97e48388a1503dd4b37fe1bbb9d7a6892969f86f94fecdb |
Close
Hashes for pgpq-0.4.1-cp37-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 968be9ac3a5139ae87d26bc562f76b3fc571d2afaaf19d7d0d0611a1feb14e1e |
|
MD5 | 3343dafc4e4ba64af0fd4d2ae5f0850c |
|
BLAKE2b-256 | 9a995b1f30f94b1c209f9d8bef78d77da0393a678937b86919b52e3e75fa66b8 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29930fad3ed5aeb363709cc8c00ba379743e42cd87934d5a5226d45cd231ba5f |
|
MD5 | fe31a9cfb4e7ee618610559ea442f6df |
|
BLAKE2b-256 | 4e9338b0b6d43c58befd13e2fedc3491abf108c512b4604919aa6d81e9984c38 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1f7a8296ec057d27786e0cbfc7ca14b84e75e9c205af08c757cf699fd969be8 |
|
MD5 | 889082589cc7ad8756c54ef87321cab7 |
|
BLAKE2b-256 | bce6fedd2e5dde614c9a62186a3d44350f8f0e987aed8d0548e2075e7596d962 |
Close
Hashes for pgpq-0.4.1-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61d38a62408fe0d228873f1ac327b6fdbde2f37303a4315f3216becacde0fada |
|
MD5 | 66ded4444f3f40de7bcca707e2bd79f7 |
|
BLAKE2b-256 | 50e9d5a638e3a246281b54f32f80de98eaea6d484835bffccf3b4ef0d302890d |
Close
Hashes for pgpq-0.4.1-cp37-abi3-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36a604a31d9dfade59384a67b3409886e168df803331cf5a3f47c683c8a910c3 |
|
MD5 | 981f5a7a30aa3c80c65d7612e0f9f818 |
|
BLAKE2b-256 | f3c122fd3e918337494b5656739988d6c56e3b2725bf4834314e6fbce817d011 |