skip to navigation
skip to content

Not Logged In

spotman_client 0.1

Client for talking to Qubole's spotman service

Latest Version: 0.3.0

spotman
=======
This project helps you in making better bids for Amazon EC2 spot instances.

EC2 spot instances allow you to name your own price for Amazon EC2 computing
capacity. You simply bid on spare Amazon EC2 instances and run them whenever
your bid exceeds the current spot price. Spot prices are often significantly
less than on-demand prices. They are set by Amazon and fluctuate in real-time
according to supply and demand. However, the exact criteria for setting the
spot instance prices hasn't been publicly disclosed.

We have observed a few peculiar characteristics of spot prices:

1. Often, the spot price of a particular instance varies considerably between
different availability zones of the same region.

2. Many times even if you bid higher than the current spot price you don't get
any spot instance.

3. Sometimes a bigger instance is available at a lesser spot price than a
smaller instance.

This project protects you against these quirks and helps you take advantage of
them. (Yay, marketing line!)


How does this work?
-------------------
Amazon provides the spot price history for all instance types across all
availability zones. We analyze this information and tell you which
(availability zone, instance type) pair is best for you.


Normalization of availability zones
-----------------------------------
EC2 availability zone names in different accounts do not refer to the same
underlying physical infrastructure. So, the zone which is referred to as
`us-east-1a` in one account may be referred to as `us-east-1b` in another
account and so on. There is no official way to find out which availability zone
in one account correspond to which availability zone in another account.
However, there are some places where this information gets leaked. For example,
each physical availability zone has a unique spot price history. We use this
fact to normalize availability zones across different accounts.


Usage
-----
* Get an API Token by going to http://localhost:5000/register

* Find out the normalized availability zones for your AWS account.
```
$ export SPOT_API_TOKEN=YOUR_API_TOKEN
$ export SPOT_API_URL=http://localhost:5000
$ spotman-client normalize-az YOUR_AWS_ACCESS_KEY_ID YOUR_AWS_SECRET_ACCESS_KEY us-east-1
Getting history from provider...
Getting customer history from AWS...
Fetching history complete.
{
"us-east-1a": "us-east-1-2",
"us-east-1b": "us-east-1-3",
"us-east-1c": "us-east-1-1",
"us-east-1d": "us-east-1-4"
}
```

The output shows that your `us-east-1a` refers to normalized availability zone
`us-east-1-2` and so on.

* Now let's say you want to run some job and you are okay with running them
either in your `us-east-1b` or `us-east-1c` availability zones. Further, you
want to use either `m1.large` or `m1.xlarge` instances and you are willing to
pay $0.12 and $0.20 for those instances respectively. So, you would do the
following:

Issue the following request using `curl`:
```
$ curl -H "X-Api-Token:your_api_token" "http://localhost:5000/get_analysis?normalized_az=us-east-1-1&normalized_az=us-east-1-3&instance_type=m1.large&price=0.10&instance_type=m1.xlarge&price=0.16"
```

or using the CLI:
```
$ spotman-client get-analysis --normalized-az us-east-1-1 us-east-1-3 --instance-type m1.large m1.xlarge --price 0.10 0.16
```

The result will be something like this:
```
{
"results": [
{
"availability": "0.984280753968",
"average_price": "0.0283093275076",
"current_price": "0.0275",
"given_price": "0.10",
"instance_type": "m1.large",
"maximum_price": "0.319999992847",
"normalized_az": "us-east-1-1"
},
{
"availability": "0.982306547619",
"average_price": "0.0540314569575",
"current_price": "0.0523",
"given_price": "0.16",
"instance_type": "m1.xlarge",
"maximum_price": "0.800000011921",
"normalized_az": "us-east-1-3"
},
{
"availability": "0.96083994709",
"average_price": "0.0530308997584",
"current_price": "0.0522",
"given_price": "0.16",
"instance_type": "m1.xlarge",
"maximum_price": "3.0",
"normalized_az": "us-east-1-1"
},
{
"availability": "0.848913690476",
"average_price": "0.0282319755835",
"current_price": "0.0265",
"given_price": "0.10",
"instance_type": "m1.large",
"maximum_price": "2.0",
"normalized_az": "us-east-1-3"
}
]
}
```

The results show that it is best to bid for m1.large instances in us-east-1-1
(= your us-east-1c) zone. Based on the last week's history, they were
available 98.43% of the time at a price which was lower than your bid price
($0.10). The average of the spot price while it was lower than your bid price
was $0.0283 and the current price is $0.0275. During the last week, the
maximum price for this instance in this availability zone hit $0.32.

Further, it shows that for m1.xlarge instance type in us-east-1-3 has a better
availability but slightly higher average price than us-east-1-1. This means
that during the time the m1.xlarge instance was available below your bid price
($0.16), it would have cost lower in us-east-1-1 but there would be more
interruptions in that zone. You can use take advantage of this if your
workload is highly tolerant to instance termination.

Finally, it shows that it may not be a good idea to bid for m1.large instances
in us-east-1-3 (= your us-east-1b) zone as there would be frequent
interruptions.


Get instance type availability
------------------------------
AWS hasn't made some instance types available in some availability zones. You
can use this service to find out which instance types are available in which
zones. We use the spot history to determine this. If an (instance_type,
normalized_az) pair is not found in the spot history then it means that that
instance type is not available in that availability zone.

Issue the following request using `curl`:
```
$ curl -H "X-Api-Token:your_api_token" "http://localhost:5000/get_instance_type_availability?regions=us-east-1®ions=sa-east-1"
```

or using the CLI:
```
$ spotman-client get-instance-type-availability --regions us-east-1 sa-east-1
```

The result would be something like this:
```
{
"c1.medium": [
"us-east-1-1",
"us-east-1-2",
"us-east-1-3",
"us-east-1-4",
"sa-east-1-1",
"sa-east-1-2"
],
"c3.2xlarge": [
"us-east-1-1",
"us-east-1-3",
"us-east-1-4"
],
.
.
.
}
```

The result (truncated for readability here) shows that some instance types are
available only in some AZs of a region and are entirely absent from
some regions.


Installation (for client)
-------------------------
* Get the package from PyPI. Using virtualenv is optional (but recommended.)
```
cd spotman/
virtualenv venv
source venv/bin/activate
pip install spotman_client
```


Feedback
--------
We have noticed that some (availability zone, instance type) frequently have
capacity issues. You won't be able to procure them even if you bid higher than
the current spot price. Right now, we don't have too much data to figure out
which (availability zone, instance type) pairs experience these issues. You can
help us here by providing information about your spot request
successes/failures.

Whenever you make a spot request, tell us the result of that request.
Specifically, the following things:

* `normalized_az`: the normalized availability zone where you made that request

* `instance_type`: the instance type you bid for

* `bid_time`: the time when you made the request

* `response_delay`: the delay in seconds before amazon responded with
success/failure

* `num_instances_requested`: the number of instances requested by you

* `bid_price`: the bid price you submitted

* `status`: whether the request was a success or a failure

* `reason`: if the request was a failure, what was the reason

Here's how to do that:

Issue the following request using `curl`:
```
$ curl -H "X-Api-Token:your_api_token" -d "normalized_az=us-east-1-1&instance_type=m1.xlarge&bid_time=2014-03-04T10:11:12Z&response_delay=300&num_instances_requested=20&bid_price=0.16&status=failure&reason=REASON_RETURNED_BY_AMAZON" http://localhost:5000/add_feedback

$ curl -H "X-Api-Token:your_api_token" -d "normalized_az=us-east-1-1&instance_type=m1.large&bid_time=2014-03-04T15:16:00Z&response_delay=200&num_instances_requested=5&bid_price=0.10&status=success" http://localhost:5000/add_feedback
```

or using the CLI:
```
$ spotman-client add-feedback --normalized-az us-east-1-1 --instance-type m1.xlarge --bid-time 2014-03-04T10:11:12Z --response-delay 300 --num-instances-requested 20 --bid-price 0.16 --status failure --reason REASON_RETURNED_BY_AMAZON

$ spotman-client add-feedback --normalized-az us-east-1-1 --instance-type m1.large --bid-time 2014-03-04T15:16:00Z --response-delay 200 --num-instances-requested 5 --bid-price 0.10 --status success
```

Once we have a good amount of this information, we would start to factor that
in our analysis for finding the best (availability zone, instance type) pair.  
File Type Py Version Uploaded on Size
spotman_client-0.1.tar.gz (md5) Source 2014-05-07 7KB
  • Downloads (All Versions):
  • 43 downloads in the last day
  • 172 downloads in the last week
  • 1531 downloads in the last month