Consistent iterable API for reading and writing to external datasets
Project description
wq.io is a collection of Python libraries for consuming (input) and generating (output) external data resources in various formats. It thereby facilitates interoperability between the wq framework and other systems and formats.
The basic idea behind wq.io is to avoid having to remember the unique usage of e.g. csv, xlrd, or lxml every time one needs to work with an external dataset. Instead, wq.io abstracts these libraries into a consistent interface that works as an iterable of namedtuples.
Example
from wq.io import load_file
data = load_file('example.xls')
for row in data:
print row.name, row.date
Extending wq.io
The actual process is broken into several steps (load, parse, and map) which are handed by various mixins. These are mixed with the BaseIO class to provide a usable class that can load and iterate over files.
Example
from wq.io import make_io
from wq.io.loaders import FileLoader
from wq.io.parsers import JsonParser
class MyJsonParser(JsonParser):
def parse(self):
# custom parsing code ...
MyJsonFileIO = make_io(FileLoader, MyJsonParser)
for record in MyJsonFileIO(filename='file.json'):
print record.id
loaders
Load an external resource from the local filesystem or from the web into a file-like object. On export, loaders prepare the file-like object for writing and perform any needed wrap-up operations.
parsers
Parse the file (usually using the standard python library for that file type) and convert the recordset into a simple list of dictionaries. On export, parsers coonvert the dictionary list back into the source format and write out to the file.
mappers
Rename field names and values if needed and optionally convert the dictionaries into other object types (such as a namedtuple). On export, mappers convert the mapped object back into a simple dictionary and map the field names and values back to the format expected by the file.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
