acumos 0.5.3

Installation

You will need a Python 3.4+ environment in order to install acumos. You can use Anaconda (preferred) or pyenv to install and manage Python environments.

If you’re new to Python and need an IDE to start developing, we recommend using Spyder which can easily be installed with Anaconda.

The acumos package can be installed with pip:

pip install acumos

conda install -c anaconda libprotobuf

Importing Acumos

First import the modeling and session packages:

from acumos.modeling import Model, List, Dict, create_namedtuple, create_dataframe
from acumos.session import AcumosSession, Requirements

Creating A Session

An AcumosSession allows you to export your models to Acumos. You can either dump a model to disk locally, so that you can upload it via the Acumos GUI, or push the model to Acumos directly.

If you’d like to push to Acumos, create a session with the push_api and auth_api arguments:

# replace these fake APIs with ones appropriate for your instance!
session = AcumosSession(push_api="https://my.acumos.instance.com/upload",
                        auth_api="https://my.acumos.instance.com/auth")

If you’re only interested in dumping a model to disk, the API arguments aren’t needed:

session = AcumosSession()

A Simple Model

Any Python function can be used to define an Acumos model using Python type hints.

Let’s first create a simple model that adds two integers together. Acumos needs to know what the inputs and outputs of your functions are. We can use the Python type annotation syntax to specify the function signature.

Below we define a function add_numbers with int type parameters x and y, and an int return type. We then build an Acumos model with an add method.

Note: Function docstrings are included with your model and used for documentation, so be sure to include one!

def add_numbers(x: int, y: int) -> int:
    '''Returns the sum of x and y'''
    return x + y

model = Model(add=add_numbers)

Exporting Models

We can now export our model using the AcumosSession object created earlier. The push and dump APIs are shown below.

Note: Pushing a model to Acumos will prompt you for your username and password. You can also set the ACUMOS_USERNAME and ACUMOS_PASSWORD environment variables to avoid being prompted.

session.push(model, 'my-model')
session.dump(model, 'my-model', '~/')  # creates ~/my-model

Defining Types

In this example, we make a model that can read binary images and output some metadata about them. This model makes use of a custom type ImageShape.

We first create a NamedTuple type called ImageShape, which is like an ordinary tuple but with field accessors. We can then use ImageShape as the return type of get_shape. Note how ImageShape can be instantiated as a new object.

import io
import PIL

ImageShape = create_namedtuple('ImageShape', [('width', int), ('height', int)])

def get_format(data: bytes) -> str:
    '''Returns the format of an image'''
    buffer = io.BytesIO(data)
    img = PIL.Image.open(buffer)
    return img.format

def get_shape(data: bytes) -> ImageShape:
    '''Returns the width and height of an image'''
    buffer = io.BytesIO(data)
    img = PIL.Image.open(buffer)
    shape = ImageShape(width=img.width, height=img.height)
    return shape

model = Model(get_format=get_format, get_shape=get_shape)

Note: Starting in Python 3.6, you can alternatively use this simpler syntax:

from acumos.modeling import NamedTuple

class ImageShape(NamedTuple):
    '''Type representing the shape of an image'''
    width: int
    height: int

Using DataFrames With scikit-learn

In this example, we train a RandomForestClassifier using scikit-learn and use it to create an Acumos model.

When making machine learning models, it’s common to use a dataframe data structure to represent data. To make things easier, acumos can create NamedTuple types directly from pandas.DataFrame objects.

NamedTuple types created from pandas.DataFrame objects store columns as named attributes and preserve column order. Because NamedTuple types are like ordinary tuple types, the resulting object can be iterated over. Thus, iterating over a NamedTuple dataframe object is the same as iterating over the columns of a pandas.DataFrame. As a consequence, note how np.column_stack can be used to create a numpy.ndarray from the input df.

Finally, the model returns a numpy.ndarray of int corresponding to predicted iris classes. The classify_iris function represents this as List[int] in the signature return.

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
X = iris.data
y = iris.target

clf = RandomForestClassifier(random_state=0)
clf.fit(X, y)

# here, an appropriate NamedTuple type is inferred from a pandas DataFrame
X_df = pd.DataFrame(X, columns=['sepal_length', 'sepal_width', 'petal_length', 'petal_width'])
IrisDataFrame = create_dataframe('IrisDataFrame', X_df)

# ==================================================================================
# # or equivalently:
#
# IrisDataFrame = create_namedtuple('IrisDataFrame', [('sepal_length', List[float]),
#                                                     ('sepal_width', List[float]),
#                                                     ('petal_length', List[float]),
#                                                     ('petal_width', List[float])])
# ==================================================================================

def classify_iris(df: IrisDataFrame) -> List[int]:
    '''Returns an array of iris classifications'''
    X = np.column_stack(df)
    return clf.predict(X)

model = Model(classify=classify_iris)

Check out the sklearn examples in the examples directory for full runnable scripts.

Declaring Requirements

my_pkg/
├── __init__.py
├── bar.py
└── foo.py

from my_pkg.bar import do_thing

def transform(x: int) -> int:
    '''Does the thing'''
    return do_thing(x)

model = Model(transform=transform)

reqs = Requirements(packages=['~/repos/my_pkg'])

# using the AcumosSession created earlier:
session.push(model, 'my-model', reqs)
session.dump(model, 'my-model', '~/', reqs)  # creates ~/my-model

reqs = Requirements(req_map={'PIL': 'pillow'})

TensorFlow

Check out the TensorFlow example in the examples directory.

Testing Models

The acumos.modeling.Model class wraps your custom functions and produces corresponding input and output types. This section shows how to access those types for the purpose of testing. For simplicity, we’ll create a model using the add_numbers function again:

def add_numbers(x: int, y: int) -> int:
    '''Returns the sum of x and y'''
    return x + y

model = Model(add=add_numbers)

The model object now has an add attribute, which acts as a wrapper around add_numbers. The add_numbers function can be invoked like so:

result = model.add.inner(1, 2)
print(result)  # 3

The model.add object also has a corresponding wrapped function that is generated by acumos.modeling.Model. The wrapped function is the primary way your model will be used within Acumos.

We can access the input_type and output_type attributes to test that the function works as expected:

AddIn = model.add.input_type
AddOut = model.add.output_type

add_in = AddIn(1, 2)
print(add_in)  # AddIn(x=1, y=2)

add_out = AddOut(3)
print(add_out)  # AddOut(value=3)

model.add.wrapped(add_in) == add_out  # True

More Examples

Below are some additional function examples. Note how numpy types can even be used in type hints, as shown in the numpy_sum function.

from collections import Counter
import numpy as np

def list_sum(x: List[int]) -> int:
    '''Computes the sum of a sequence of integers'''
    return sum(x)

def numpy_sum(x: List[np.int32]) -> np.int32:
    '''Uses numpy to compute a vectorized sum over x'''
    return np.sum(x)

def count_strings(x: List[str]) -> Dict[str, int]:
    '''Returns a count mapping from a sequence of strings'''
    return Counter(x)

v0.5

• Modeling

  • Python 3.6 NamedTuple syntax support now tested

  • User documentation includes example of new NamedTuple syntax

• Model wrapper

  • Model wrapper now has APIs for consuming and producing Python dicts and JSON strings

• Protobuf and protoc

  • An explicit check for protoc is now made, which raises a more informative error message

  • User documentation is more clear about dependence on protoc, and provides an easier way to install protoc via Anaconda

• Keras

  • The active keras backend is now included as a tracked module

  • keras_contrib layers are now supported

v0.4

• Replaced library-specific onboarding functions with “new-style” models

  • Support for arbitrary Python functions using type hints

  • Support for custom user-defined types

  • Support for TensorFlow models

  • Improved dependency introspection

  • Improved object serialization mechanisms