skip to navigation
skip to content

Not Logged In

EasyCluster 0.01

EasyCluster: a remote execution/clustering module for Python

Package Documentation

Latest Version: 0.21

easycluster
=========

EasyCluster is a remote execution / clustering module for Python.

Possible uses include:
* computation (e.g. NumPy, PyOpenCL)
* coordinated automation for testing networks / SANs
* access to specific hardware in multiple systems (e.g. GPUs, video capture/encoding boards)

Requirements
------------
* CPython 2.6+ or 3.2+

Features
--------
* Transparent calling of functions and methods
* Transparent handling of exceptions
* Convenience functions for calling one function in parallel on
  multiple remote systems
* Automatic support for threading
* Requests and responses protected with shared HMAC key
* Cross-platform compatible; Master scripts running on Linux/OSX can connect
  to servers running on Windows and vice/versa.

How it works
------------

EasyCluster works by having a single master script connect to one or more
servers running the cluster service. The master can then call Python functions
on the remote service or send code to execute:

 >>> from easycluster import *
 >>> define_common('''
 ... def addvals(a, b):
 ...     return a + b
 ... def subvals(a, b):
 ...     return a - b
 ... ''')
 >>> key = read_key_file('secret.key')
 >>> rmt = Client(key, 'localhost')
 >>> rmt.addvals(3, 4)
 7
 >>> rmt.subvals(15, 4)
 11
 >>> rmt.subprocess.call(['/bin/echo', 'hello'])
 >>>

Any "complex" objects that the server returns are converted into proxy objects
on the master which maintain a reference to the object on the server. Calling a
method on this proxy calls the corresponding method on the server. These proxy
objects can also be passed as arguments to other functions on the same
connection, and will be unserialized as the original object on the
server. "Simple" objects like strings, numbers, lists, tuples, etc. are copied
by value.

 >>> define_common('''
 ... class TestObject1(ServerObject):
 ...    def __init__(self, val):
 ...        self.val = val
 ...    def getval(self):
 ...        return self.val
 ...    def newobj(self):
 ...        return TestObject1(self.val + 1)
 ... ''')
 >>> rmt.update_definitions()
 >>> obj1 = rmt.TestObject1(100)
 >>> obj1
 <RemoteProxy for oid 1 on localhost:11999>
 >>> obj1.getval()
 100
 >>> obj2 = obj1.newobj()
 >>> obj2
 <RemoteProxy for oid 2 on localhost:11999>
 >>> obj2.getval()
 101
 >>>

Classes can indicate that they should be proxied rather than copied by
inheriting from ServerObject. Existing classes which are unaware of EasyCluster
can be registered on the server by calling make_server_class.

There are three ways classes can specify which methods and attributes to export:
* Using the default proxy
  Classes which inherit from ServerObject but do not specify a proxy class or
  exported methods and attributes will use a "default" proxy which treats all
  attributes as methods. This is the method shown above.
* Specifying export_attrs and export_methods
  Classes which specify export_methods or export_attrs will have an anonymous
  proxy class created on the client which contains wrappers for only the
  specified methods and attributes:

 >>> define_common('''
 ... class TestObject2(TestObject1):
 ...     export_methods = ('getval',)
 ...     export_attrs = ('val',)
 ... ''')
 >>> rmt.update_definitions()
 >>> obj2 = rmt.TestObject2(200)
 >>> obj2.val
 200
 >>> obj2.getval()
 200
 >>> obj2.non_existant_method()
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 AttributeError: 'dynamic_proxy_getval_val' object has no attribute 'non_existant_method'
 >>>

* Defining a proxy class directly
  This is the most flexible way of exporting methods and attributes. This allows
  you to not only define proxy methods and attributes, but allows you to:
** Subclass a proxy class for another object which your object inherits from
   without worrying about updating your export_methods and export_attrs when the
   base class changes.
** Implement simple methods on the client. For example, most iterators simply
   return 'self' from __iter__. In fact, easycluster provides a proxy class you
   can inherit from called SelfIterProxy which does this.
** Implement caching of attributes which you know will not change.

 >>> define_common('''
 ... class TestObject3Proxy(RemoteProxy):
 ...     proxy_methods = ('getval',)
 ...     proxy_attrs = ('val',)
 ... class TestObject3(TestObject1):
 ...     proxy_class = TestObject3Proxy
 ... ''')
 >>> rmt.update_definitions()
 >>> obj3 = rmt.TestObject3(300)
 >>> type(obj3)
 <class 'easycluster_code.TestObject3Proxy'>
 >>> obj3.val
 300

The master script can connect to the same server multiple times. Each connection
creates a separate process with a clean environment. The master can also create
a "local" instance, which starts a new server process without having to run a
separate server.

See easycluster_demo.py for an example of how to use most of the features.

EasyCluster is intended to run on a local area network to coordinate specific
tasks. It is not intended as a generic "cloud computing" service. Although data
is signed with a private HMAC key, it is not encrypted, so it should not be used
openly on the Internet with sensitive data. If you want to use EasyCluster to
coordinate systems in remote geographic areas, consider using a VPN or SSH
tunnel. The EasyCluster service operates over a single TCP port, so most
tunneling solutions will work.

Installing
----------

EasyCluster uses setuptools for installation. To install, run:

    python setup.py build
    sudo python setup.py install

If you do not have setuptools installed, it will be downloaded for you.
 
File Type Py Version Uploaded on Size
EasyCluster-0.01-py2.7.egg (md5) Python Egg 2.7 2013-02-07 42KB
EasyCluster-0.01.tar.gz (md5) Source 2013-02-07 26KB
  • Downloads (All Versions):
  • 295 downloads in the last day
  • 762 downloads in the last week
  • 3837 downloads in the last month