luigi monkey patch for large s3 file iteration
Project description
Luigi is a Python (2.7, 3.3, 3.4, 3.5) package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
Current Luigi has some issues in s3 file iteration in production. When the file is too large, Amazon s3 would tend to disconnect first. Any application should support retry based on current status to aquire the whole part. This project add this part for luigi s3 file iteration feature.
Getting Started
Run pip install luigi_monkey_patch to install the latest stable version from Pypi. Documentation and any issues are hosted here.
Modified Functions
luigi.s3.ReadableS3File.close
luigi.s3.ReadableS3File.__iter__
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for luigi_monkey_patch-0.0.1-py2-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 11a02cbfe5da836e1616c1c1aa54f6b18f7d87dc00f2a43f0dcc14ae7a7a6e68 |
|
| MD5 | fa8bb6384dbf9d96a0682fcfcadfaac8 |
|
| BLAKE2b-256 | ea392da8ef86c93666d3d21cf8862ba10d3739bfd9d904e16b505b3e45b3cf8f |
