skip to navigation
skip to content

Not Logged In

mr.awsome 0.13

A script allowing to setup Amazon EC2 instances through configuration files.

Latest Version: 1.0rc1

Overview

mr.awsome is a commandline-tool (aws) to manage and control Amazon Webservice's EC2 instances. Once configured with your AWS key, you can create, delete, monitor and ssh into instances, as well as perform scripted tasks on them (via fabfiles). Examples are adding additional, pre-configured webservers to a cluster (including updating the load balancer), performing automated software deployments and creating backups - each with just one call from the commandline. Aw(e)some, indeed, if we may say so...

Installation

mr.awsome is best installed with easy_install, pip or with zc.recipe.egg in a buildout. It installs two scripts, aws and assh.

With zc.recipe.egg you can set a custom configfile location like this:

[aws]
recipe = zc.recipe.egg
eggs = mr.awsome
arguments = configpath="${buildout:directory}/etc/servers.cfg"

The pycrypto package is throwing some deprecation warnings, you might want to disable them by adding an initialization option to the aws part like this:

initialization =
    import warnings
    warnings.filterwarnings("ignore", ".*", DeprecationWarning, "Crypto\.Hash\.MD5", 6)
    warnings.filterwarnings("ignore", ".*", DeprecationWarning, "Crypto\.Hash\.SHA", 6)
    warnings.filterwarnings("ignore", ".*", DeprecationWarning, "Crypto\.Util\.randpool", 40)

Configuration

Support for backends is implemented by plugins. Two plugins are included with mr.awsome. To use the plugins, you have to configure one or more masters, like this:

[ec2-master:default]
module = mr.awsome.ec2

[plain-master:default]
module = mr.awsome.plain

The ec2-master is for the Amazon cloud. The plain-master is for servers reachable by ssh which you want to included in the config for Fabric integration and easy ssh access with a centralized config.

To authorize itself against AWS, mr.awsome uses the following two environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

You can find their values at http://aws.amazon.com under 'Your Account'-'Security Credentials'.

You can also put them into files and point to them in the [ec2-master:default] section with the access-key-id and secret-access-key options. It's best to put them in ~/.aws/ and make sure only your user can read them.

All other information about server instances is located in aws.conf, which by default is looked up in etc/aws.conf.

Before you can create a server instance with the create command described below, you first have to declare a security group in your aws.conf like this:

[ec2-securitygroup:demo-server]
description = Our Demo-Server
connections =
  tcp 22 22 0.0.0.0/0
  tcp 80 80 0.0.0.0/0

The security group is used for both the firewall settings, as documented in the AWS docs, and to find the server instance associated with it.

Then you can add the info about the server instance itself like this:

[ec2-instance:demo-server]
keypair = default
securitygroups = demo-server
region = eu-west-1
placement = eu-west-1a
# we use images from `http://alestic.com/`_
# Ubuntu 9.10 Karmic server 32-bit Europe
image = ami-a62a01d2
startup_script = startup-demo-server
fabfile = `fabfile.py`_

Ephemeral disks

You can setup the use of more than one ephemeral disk like this:

[ec2-instance:demo-server]
...
instance_type = m1.large
device_map = /dev/sdb:ephemeral0 /dev/sdc:ephemeral1

Consult the AWS documentation to see how many ephemeral disks each instance type has.

Startup scripts

The startup_script option above allows you to write a script which is run right after instance creation to setup your server. This feature is supported by many AMI images and was made popular by http://alestic.com/ (See http://alestic.com/2009/06/ec2-user-data-scripts).

Most of the time these are bash scripts like this (for Ubuntu in this case):

#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
apt-get update && apt-get upgrade -y

The set -e -x is for debugging. You can see the commands which ran and their output in /var/log/syslog once you are logged into the server.

The startup scripts have a maximum size of 16kb. You can check the size with the debug command of the aws script.

The startup script is basically a template for the Python string format method (See http://docs.python.org/library/string.html#formatstrings). So anything inside curly brackets is expanded. To get normal curly brackets, when you write bash functions etc, just double them like this:

function LOG() {{ echo "$*"; }}

If you want to include any files for something like ssh authorized_keys, you do something the following:

authorized_keys: file,escape_eol ssh-authorized_keys

#!/bin/bash
...
/bin/bash -c "echo -e \"{authorized_keys}\" >> /root/.ssh/authorized_keys"

So the startup script basically has rfc822 syntax (internally the e-mail parser is used). The file,escape_eol tells the script that the ssh- authorized_keys string should be used as a filename for a file which is then read and the \n characters are escaped so the resulting string can be used in the echo -e command.

You have the following possibilities (brain dump, needs fleshing out):
  • file
  • base64
  • format
  • template
  • gzip
  • escape_eol

In addition to that, you have access to some more variables. For example full access to the server config in the aws.conf. With servers[demo- server].instance.dns_name for example, you can get the current DNS name of the server (this only works with other servers already started, not the one for which the startup script is for, since the DNS isn't set at the time the script is created).

You can modify the options for the startup script by declaring a hook like this in your config:

hooks = mymodule.Hooks

Where Hooks is a class with a startup_script_options method. Here is an example which adds an addresses option containing the IP address of available EC2 instances:

class _IPProxy(object):
    def __init__(self, servers):
        self.servers = servers

    def __getitem__(self, value):
        result = self.servers[value]
        instance = result.instance
        if instance is None:
            # return a dummy address
            return u'192.168.0.1'
        return result.instance.private_ip_address


class Hooks(object):
    def startup_script_options(self, options):
        addresses = options.get('addresses')
        if addresses is None:
            options['addresses'] = _IPProxy(options['servers'])

You can add a gzip: prefix before the filename to let the script be self extracting. The code used looks like this:

#!/bin/bash
tail -n+4 $0 | gunzip -c | bash
exit $?

Directly after that follows the binary data of the gzipped startup script.

Controlling instances

  • start
  • stop
  • status

Snapshots

(Needs description of volumes in "Configuration")

SSH integration

mr.awsome provides an additional tool assh to easily perform SSH based operations against named EC2 instances. Particularly, it encapsulates the entire SSH fingerprint mechanism, as EC2 instances are often short-lived and normally trigger warnings, especially, if you are using elastic IPs.

Note:: it does so not by simply turning off these checks, but by transparently updating its own fingerprint list (it relies on the console output of the instance to provide the fingerprint via the AWS API, some images may not be configured to do so) when adding new instances.

The easiest scenario is simply to create an SSH session with an instance. You can either use the ssh subcommand of the aws tool like so:

aws ssh SERVERNAME

Alternatively you can use the assh command direct, like so:

assh SERVERNAME

The latter has been provided to support scp and rsync. Here are some examples, you get the idea:

scp -S `pwd`/bin/assh some.file demo-server:/some/path/
rsync -e "bin/assh" some/path fschulze@demo-server:/some/path

Fabric integration

Since Fabric <http://fabfile.org/`_>`_ basically works through ssh, all the bits necessary for ssh integration are also needed for Fabric. To make it easy to run fabfiles, you specifiy them with the "fabfile" option in your aws.conf and use the ``do command to run them.

Take the following fabfile.py as an example:

from fabric.api import env, run

env.reject_unknown_hosts = True
env.disable_known_hosts = True

def get_syslog():
  run("echo /var/log/syslog")

If you have that fabfile for the demo-server above, you can then run the command with "bin/aws demo-server do get_syslog".

For more info about fabfiles, read the docs at http://fabfile.org/.

Macro expansion

In the aws.conf you can use macro expansion for cleaner configuration files. This looks like this:

[ec2-instance:demo-server2]
<= demo-server
securitygroups = demo-server2

[ec2-securitygroup:demo-server2]
<= demo-server

All the options from the specified macro are copied with some important exceptions:

  • For instances the ip and volumes options aren't copied.

If you want to copy data from some other kind of options, you can add a colon in the macro name. This is useful if you want to have a base for instances like this:

[macro:base-instance]
keypair = default
region = eu-west-1
placement = eu-west-1a

[ec2-instance:server]
<= macro:base-instance
...

Changelog

0.13 - 2013-09-20

  • Use os.path.expanduser on all paths, so that one can use ~ in config values like the aws keys. [fschulze]

0.12 - 2013-09-11

  • There is no need to add the AWS account id to security group names anymore. [fschulze]
  • Rules are removed from security groups if they aren't defined in the config. [fschulze]
  • Allow adding of custom config massagers from inside the config. [fschulze]
  • Support block device maps to enable use of more than one ephemeral disk. [fschulze]
  • Added do method on ec2 and plain instances which allows to call fabric commands. [fschulze]
  • Use PathMassager for access-key-id and secret-access-key in the ec2-master section. This might break existing relative paths for these options. [fschulze]
  • Added support for EBS boot instances. [fschulze]
  • Add option ssh-key-filename to point to a private ssh key for ec2 and plain instances. [fschulze]
  • Fix Fabric integration for newer versions of Fabric. [fschulze]
  • Support proxycommand option for plain instances. This also caused a change in the init_ssh_key API for plugins. [fschulze]
  • Support ProxyCommand from ~/.ssh/config for plain instances. Requires Fabric 1.5.0 and Paramiko 1.9.0 or newer. [fschulze]

0.11 - 2012-11-08

  • Support both the ssh and paramiko libraries depending on which Fabric version is used. [fschulze]

0.10 - 2012-06-04

  • Added ec2-connection which helps in writing Fabric scripts which don't connect to a server but need access to the config and AWS (like uploading something to S3). [fschulze]
  • Fix several problems with using a user name other than root for the do and ssh commands. [fschulze]
  • Require Fabric >= 1.3.0. [fschulze]
  • Require boto >= 2.0. [fschulze]
  • Added hook for startup script options. [fschulze]
  • Added possibility to configure hooks. [fschulze]
  • Refactored to enable plugins for different virtualization or cloud providers. [fschulze]
  • Added lots of tests. [fschulze]

0.9 - 2010-12-09

  • Overwrites now also affect server creation, not just the startup script. [fschulze]
  • Added list command which supports just listing snapshots for now. [fschulze]
  • Added delete-volumes-on-terminate option to delete volumes created from snapshots on instance termination. [fschulze]
  • Added support for creating volumes from snapshots on instance start. [natea, fschulze]
  • Added support for ~/.ssh/config. This is a bit limited, because the paramiko config parser isn't very good. [fschulze]
  • Added help command which provides some info for zsh autocompletion. [fschulze]

0.8 - 2010-04-21

  • For the do command the Fabric options reject_unknown_hosts and disable_known_hosts now default to true. [fschulze]
  • Allow adding normal servers to use with ssh and do commands. [fschulze]
  • Refactored ssh connection handling to only open network connections when needed. Any fabric option which doesn't need a connection runs right away now (like -h and -l). [fschulze]
  • Fix status output after start. [fschulze]

0.7 - 2010-03-22

  • Added snapshot method to Server class for easy access from fabfiles. [fschulze]

0.6 - 2010-03-18

  • It's now possible to specify files which contain the aws keys in the [aws] section with the access-key-id and secret-access-key options. [fschulze]
  • Added -c/--config option to specify the config file to use. [fschulze]
  • Added -v/--version option. [tomster, fschulze]
  • Comment lines in the startup script are now removed before any variables in it are expanded, not afterwards. [fschulze]
  • Use argparse library instead of optparse for more powerful command line parsing. [fschulze]

0.5 - 2010-03-11

  • Added gzipping of startup script by looking for gzip: prefix in the filename. [fschulze]
  • Added macro expansion similar to zc.buildout 1.4. [fschulze]

0.4 - 2010-02-18

  • Check console output in status and tell user about it. [fschulze]
  • Friendly message instead of traceback when trying to ssh into an unavailable server. [fschulze]
  • Remove comment lines from startup script if it's starting with #!/bin/sh or #!/bin/bash. [fschulze]
  • Removed -r option for start and debug commands and replaced it with more general -o option. [fschulze]
  • Made startup script optional (not all AMIs support it, especially Windows ones). [fschulze]
  • The stop command actually only stops an instance now (only works with instances booted from an EBS volume) and the new terminate command now does what stop did before. [fschulze]
  • Better error message when no console output is available for ssh finger print validation. [fschulze]
  • Fixed indentation in documentation. [natea, fschulze]

0.3 - 2010-02-08

  • Removed the [host_string] prefix of the do command output. [fschulze]

0.2 - 2010-02-02

  • Snapshots automatically get a description with date and volume id. [fschulze]
  • The ssh command can now be used with scp and rsync. [fschulze]

0.1 - 2010-01-21

  • Initial release [fschulze]
 
File Type Py Version Uploaded on Size
mr.awsome-0.13.zip (md5) Source 2013-09-20 52KB
  • Downloads (All Versions):
  • 45 downloads in the last day
  • 206 downloads in the last week
  • 1555 downloads in the last month