What is NumS?

NumS is a Numerical computing library for Python that Scales your workload to the cloud. It is an array abstraction layer on top of distributed memory systems that implements the NumPy API, extending NumPy to scale horizontally, as well as provide inter-operation parallelism (e.g. automatic parallelization of Python loops). NumS differentiates itself from related solutions by implementing the NumPy API, and providing tighter integration with the Python programming language by supporting loop parallelism and branching. Currently, NumS implements a Ray system interface, S3 and distributed filesystems for storage, and NumPy as a backend for CPU-based array operations.

Installation

Currently, NumS only supports CPU-based workloads; we are working on providing GPU support.

pip installation

To install NumS on Ray with CPU support, simply run the following command.

pip install nums

conda installation

We are working on providing support for conda installations, but in the meantime, run the following with your conda environment activated.

pip install nums
# Run below to have NumPy use MKL.
conda install -fy mkl
conda install -fy numpy scipy

S3 Configuration

To run NumS with S3, configure credentials for access by following instructions here: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

Contributing

To contribute to NumS on Ray, we recommend cloning the repository and installing the project in developer mode using the following set of commands:

conda create --name nums python=3.6 -y
source activate nums
pip install ray==0.8.7
python setup.py develop

Contributing NumPy Functionality

To make basic contributions to the NumPy API, follow these steps:

1. Replicate the function signature in nums.numpy.api. If it's a np.ndarray method, add the function signature to nums.core.array.blockarray.BlockArray.
2. If possible, implement the function using existing methods in nums.core.array.application.ArrayApplication or nums.core.array.blockarray.BlockArray.
3. Write a new implementation ArrayApplication or BlockArray if it's not possible to implement using existing methods, or the implementation's execution speed can be improved beyond what is achievable using existing methods.
4. Add kernel interfaces to nums.core.systems.interfaces.ComputeInterface, and implement the interface methods for all existing compute implementations. Currently, the only compute interface is nums.core.systems.numpy_compute.
5. Write tests covering all branches of your implementation in the corresponding test module in the project's tests/ directory.
6. Do your best to implement the API in its entirety. It's generally better to have a partial implementation than no implementation, so if for whatever reason certain arguments are difficult to support, follow the convention we use to raise errors for unsupported arguments in functions like nums.numpy.api.min.
7. If you run into any issues and need help with your implementation, open an issue describing the issue you're experiencing.

We encourage you to follow the nums.numpy.api.arange implementation as a reference.