iterable-subprocess

Python utility function to communicate with a subprocess using iterables: for when data is too big to fit in memory and has to be streamed.

Data is sent to a subprocess's standard input via an iterable, and extracted from its standard output via another iterable. This allows an external subprocess to be naturally placed in a chain of iterables for streaming processing.

Installation

pip install iterable-subprocess

Usage

A single function iterable_subprocess is exposed. The first parameter is the args argument passed to the Popen Constructor, and the second is an iterable whose items must be bytes instances and are sent to the subprocess's standard input.

Returned from the function is an iterable whose items are bytes instances of the process's standard output.

from iterable_subprocess import iterable_subprocess

def yield_input():
    # In a real case could read from the filesystem or the network
    yield b'first\n'
    yield b'second\n'
    yield b'third\n'

output = iterable_subprocess(['cat'], yield_input())

for chunk in output:
    print(chunk)

Usage: unzip the first file of a ZIP archive while downloading

While its not typically possible to completely unzip an arbitrary ZIP file on-the-fly, it is possible to unzip the first file in a ZIP archive using funzip, as in the following example.

from iterable_subprocess import iterable_subprocess
import httpx

def zipped_chunks():
    with httpx.stream('GET', 'https://www.example.com/my.zip') as r:
        yield from r.iter_bytes()

unzipped_chunks = iterable_subprocess(['funzip'], zipped_chunks())

for chunk in unzipped_chunks:
    print(chunk)

Ideally Python's zipfile module would be able to do this without calling into funzip. However, at the time of writing this does not appear possible.