skip to navigation
skip to content

Not Logged In

CrowdProcess 0.1.5

Latest Version: 0.2.0

This is a client for CrowdProcess’s REST API.

It works in python 2.7 and 3.4+.

Installing

pip install crowdprocess

or

easy_install crowdprocess

Usage example

>>> from crowdprocess import CrowdProcess
>>> crp = CrowdProcess('username', 'password')

>>> x2 = crp.job('function Run (d) { return d*;2r }')
>>> results = x2(range(5))
>>> list(results)
[0, 2, 4, 6, 8, 10] # comes in a random order

More detailed use

Importing and instanciating

>>> from crowdprocess import CrowdProcess
>>> crp = CrowdProcess('username@email.com', 'password')

To get those credentials you must register with CrowdProcess.

You can also instanciate it with a token instead of a username and password:

>>> crp = CrowdProcess(token='3c46d593-5435-47c5-92aa-1613ade978c2')

Jobs

Creating a job

With the CrowdProcess class instanciated above,

>>> program='function Run (d) { return d }'
>>> job = crp.job(program)
>>> job.id
'3c46d593-5435-47c5-92aa-1613ade978c2'

Invoking crp.job with the program parameter automatically creates a job in CrowdProcess and returns an instanciated Job.

After you get a job.id, you can use it to get a Job again, without creating it:

>>> job = crp.job(id='3c46d593-5435-47c5-92aa-1613ade978c2')

Listing jobs

>>> crp.list_jobs()
[{u'status': u'active', u'failed': 0, u'bid': 1, u'created': u'2014-05-14T10:07:52.747503Z', u'modified': u'2014-05-14T10:07:53.716147Z', u'browserHours': 137, u'finished': 1000, u'lastResult': u'2014-05-14T10:07:59.06Z', u'total': 1000, u'id': u'3c46d593-5435-47c5-92aa-1613ade978c2'}]

Prettier:

>>> jobs = crp.list_jobs()
>>> print json.dumps(jobs, sort_keys=True, indent=2)
[
  {
    "bid": 1,
    "browserHours": 137,
    "created": "2014-05-14T10:07:52.747503Z",
    "failed": 0,
    "finished": 1000,
    "id": "3c46d593-5435-47c5-92aa-1613ade978c2",
    "lastResult": "2014-05-14T10:07:59.06Z",
    "modified": "2014-05-14T10:07:53.716147Z",
    "status": "active",
    "total": 1000
  }
]

Deleting a job

>>> job = crp.job(id='3c46d593-5435-47c5-92aa-1613ade978c2')
>>> job.delete()

Deleting all jobs

>>> crp.list_jobs()

Tasks and Results

After creating a job, you’re all set to send it tasks and get back results.

tasks can be any iterable object, results will be a generator:

>>> job = crp.job('function Run (d) { return Math.pow(d, 2); }')
>>> tasks = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> results = job(tasks)
>>> list(results)
[49, 64, 16, 25, 9, 36, 4, 81, 0, 1]

which would be the same as,

>>> job = crp.job('function Run (d) { return Math.pow(d, 2); }')
>>> list(job(range(10)))
[49, 64, 16, 25, 9, 36, 4, 81, 0, 1]

which would also be the same as,

>>> job = crp.job('function Run (d) { return Math.pow(d, 2); }')
>>> def tasks():
...     for i in range(10):
...             yield i
...
>>> list(job(tasks))
[25, 64, 49, 16, 36, 9, 0, 81, 1, 4]

Notice that the results never come in order.

Pro tip: you can use the results of one job as tasks of another job

>>> multiply = crp.job('function Run (d) { return d*2 }')
>>> divide = crp.job('function Run (d) { return d/2 }')
>>> numbers = range(10)
>>> multiplied = multiply(numbers)
>>> divided = divide(multiplied)
>>> list(divided)
[7, 2, 6, 1, 5, 9, 8, 4, 3, 0]

Tasks and Results, lower level

Creating tasks

Once again, tasks may be any iterable:

>>> multiply = crp.job('function Run (d) { return d*2 }')
>>> multiply.create_tasks(range(10))

Getting results

>>> results = multiply.get_results()
>>> list(results)
[18, 8, 10, 4, 6, 16, 14, 0, 2, 12]

This delivers all the job’s computed results at the moment, but you should in fact get every result as soon as it’s computed, in a stream:

Streaming results

>>> results = multiply.get_results_stream()
>>> list(results)
[18, 8, 10, 4, 6, 16, 14, 0, 2, 12]

In the code block above, list(results) will block until all results are computed and delivered.

You can also iterate through every result as soon as it comes in:

>>> results = multiply.get_results_stream()
>>> for result in results:
...     print(result)

And that loop will finish when the last result is delivered.

To use this properly you should start listening for streaming results before sending tasks, probably a separate thread:

>>> import threading
>>> job = crp.job("function Run(d) { return d; }")
>>> def get_results():
...     for result in job.get_results_stream():
...             print result
...
>>> t = threading.Thread(target=get_results)
>>> t.start()
>>> job.create_tasks(range(10))
>>> 7
9
6
2
3
8
1
4
0
5

Errors and streaming errors

Sometimes your tasks throw uncaught exceptions, and you should get them:

>>> program = """
... function Run (d) {
...     if (d === 4) {
...             throw new Error("oops, it's "+d);
...     } else {
...             return d;
...     }
... }
... """
>>> job = crp.job(program)
>>> job.create_tasks(range(10))
>>> list(job.get_results())
[1, 6, 9, 8, 5, 7, 2, 3, 0] # oh no, 4 is missing...
>>> list(job.get_errors())
[{u'message': u"oops, it's 4", u'type': u'program', u'name': u'Error', u'stack': u'Run@blob:9a4029f7-fff7-4da8-b552-92507e341749:5\n[2]</</self.onmessage@blob:9a4029f7-fff7-4da8-b552-92507e341749:9\n'}]
>>> print json.dumps(list(job.get_errors()), sort_keys=True, indent=2) # prettier
[
  {
    "message": "oops, it's 4",
    "name": "Error",
    "stack": "Run@blob:9a4029f7-fff7-4da8-b552-92507e341749:5\n[2]</</self.onmessage@blob:9a4029f7-fff7-4da8-b552-92507e341749:9\n",
    "type": "program"
  }
]

The same way you get streaming results, you can (and should) get streaming errors:

>>> errors = multiply.get_errors_stream()
>>> for error in errors:
...     print(error)
 
File Type Py Version Uploaded on Size
CrowdProcess-0.1.5.tar.gz (md5) Source 2014-05-16 4KB
  • Downloads (All Versions):
  • 12 downloads in the last day
  • 206 downloads in the last week
  • 770 downloads in the last month