Communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed
Project description
iterable-subprocess
Python context manager to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed.
Data is sent to a subprocess's standard input via an iterable, and extracted from its standard output via another iterable. This allows an external subprocess to be naturally placed in a chain of iterables for streaming processing.
Installation
pip install iterable-subprocess
Usage
A single context manager iterable_subprocess
is exposed. The first parameter is the args
argument passed to the Popen Constructor, and the second is an iterable whose items must be bytes
instances and are sent to the subprocess's standard input.
Returned from the function is an iterable whose items are bytes
instances of the process's standard output.
from iterable_subprocess import iterable_subprocess
# In a real case could be a generator function that reads from the filesystem or the network
iterable_of_bytes = (
b'first\n',
b'second\n'
b'third\n'
)
with iterable_subprocess(['cat'], iterable_of_bytes) as output:
for chunk in output:
print(chunk)
If the process exits with a non-zero error code, a SubprocessError
exception will be raised, with the contents of the processes's standard error as the message. At the time of writing if the process outputs a lot to its standard error, there may be deadlock.
Usage: unzip the first file of a ZIP archive while downloading
It's possible to download the bytes of a ZIP file in Python, and unzip by passing the bytes to funzip
, as in the following example.
import httpx
from iterable_subprocess import iterable_subprocess
with \
httpx.stream('GET', 'https://www.example.com/my.zip') as r, \
iterable_subprocess(['funzip'], r.iter_bytes()) as unzipped_chunks:
for chunk in unzipped_chunks:
print(chunk)
Note that it's also possible to stream unzip files without resorting to another process using stream-unzip.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for iterable_subprocess-0.0.9.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1186e2edaa30d7b35f2b158d6328fb42cf632be4582199f1da9d4d6b3bc83b1c |
|
MD5 | abe7f0965d9401dfacad278a8bd07a9b |
|
BLAKE2b-256 | 68796862123656678603ed58680ed087a0c33bc4ca9cf16b57f43f8482b4ce04 |
Hashes for iterable_subprocess-0.0.9-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e6e5a097a3dfd58bbb555bc7cc33d0be276d534b58ed0554fb68b29f3b0cacb2 |
|
MD5 | e70bb2e9eaf725950bf35503377459a6 |
|
BLAKE2b-256 | 611a5384566932aa8fbc36290c9bef1b41a3fc2273bbba523e1390d5f1a57a8a |