skip to navigation
skip to content 1.0.0

Consistent iterable API for reading and writing to external datasets is a Pythonic library for consuming (input), iterating over, and generating (output) external data resources in various formats. facilitates interoperability between the wq framework and other systems and formats. is designed to be customized, with a base class and modular mixin classes that handle loading, parsing, and mapping external data to a convenient API.

Somewhat coincidentally, is also the URL for the website describing the wq framework as a whole. The documentation for (the library) is available on (the website) at

Getting Started

# Recommended: create virtual environment
# python3 -m venv venv
# . venv/bin/activate

# Install entire wq suite (recommended)
pip install wq

# Install only
pip install

# To enable's GIS support
pip install geopandas # includes Shapely & Fiona

# To enable's Excel write support
pip install xlwt # xls support
pip install xlsxwriter # xlsx support
# (xls/xlsx read support is enabled by default)

See the wq documentation for more information.

Features provides a general purpose API for loading, iterating over, and writing tabular datasets. The basic idea is to avoid needing to remember the unique usage of e.g. csv, xlrd, or xml.etree every time one needs to work with external data. Instead, abstracts these libraries into a consistent interface that works as an iterable of namedtuples. Whenever possible, the field names for a dataset are automatically determined from the source file, e.g. the column headers in an Excel spreadsheet.

from import ExcelFileIO
data = ExcelFileIO(filename='example.xls')
for row in data:
    print, provides a number of built-in classes like the above, including a CsvFileIO, XmlFileIO, and JsonFileIO. There is also a convenience function, load_file(), that attempts to automatically determine which class to use for a given file.

from import load_file
data = load_file('example.csv')
for row in data:

All of the included *FileIO classes support both reading and writing to external files, though write support for Excel files requires additional libraries (xlwt and xlsxwriter) that aren’t listed as dependencies.

Network Client also provides network-capable equivalents of each of the above classes, to facilitate loading data from third party webservices.

from import JsonNetIO
class WebServiceIO(JsonNetIO):
    url = ""

data = WebServiceIO(params={'type': 'all'})
for row in data:
    print row.timestamp, row.value

The powerful requests library is used internally to load data over HTTP.

Pandas Analysis

When Pandas is installed, the as_dataframe() method on classes can be used to create a DataFrame, enabling more extensive analysis possibilities.

instance = WebServiceIO(params={'type': 'all'})
df = instance.as_dataframe()
print df.value.mean()

GIS Support

When Fiona and Shapely are installed, can also open and create shapefiles and other OGR-compatible geographic data formats.

from import ShapeIO
data = ShapeIO(filename='sites.shp')
for id, site in data.items():
    print id, site.geometry.wkt


Each IO class is composed of mixin classes (loaders, parsers, and mappers) that handle the various steps of the process. By extending these mixin or the pre-mixed classes above, it is straightforward to extend to support arbitrary formats. The climata library provides a number of examples of custom IO classes for loading climate and hydrology data.

File Type Py Version Uploaded on Size (md5) Source 2017-07-19 17KB