Skip to main content

AWS cloudformation-based deployment framework

Project description

humilis
==========

[![Build Status](https://travis-ci.org/humilis/humilis.svg?branch=master)](https://travis-ci.org/humilis/humilis)
[![PyPI](https://img.shields.io/pypi/v/humilis.svg?style=flat)](https://pypi.python.org/pypi/humilis)

Helps you deploy AWS infrastructure with [Cloudformation][cf].

[cf]: https://aws.amazon.com/cloudformation/

This project is originally based on the [cumulus][cumulus] project. See
[CUMULUS_LICENSE][cumulus_license] for license information.

[cumulus]: https://github.com/cotdsa/cumulus
[cumulus_license]: https://github.com/humilis/humilis/blob/master/CUMULUS_LICENSE


# Installation

Install the [AWS CLI][awscli]:

[awscli]: https://aws.amazon.com/cli/

```
pip install awscli
```

Configure the AWS CLI:

```
aws configure
```

`humilis` will use whatever credentials you introduced when configuring your
AWS CLI installation.

You can now install the latest "stable" version of `humilis`:

```
pip install humilis
```

or the development version if you prefer that:

````
pip install git+https://github.com/humilis/humilis
````

After installation you need to configure humilis. To configure globally for
your system:

```
humilis configure
```

The command above will store and read the configuration options from
`~/.humilis.ini`. You can also store the configuration in a `.humilis.ini` file
stored in your current working directory by using:

```
humilis configure --local
```

`humilis` will always read the configuration first from a `.humilis.ini` file
under your current work directory. If it is not found then it will read it from
your system global config file `~/.humilis`.


# Development environment

Assuming you have [virtualenv][venv] installed:

[venv]: https://virtualenv.readthedocs.org/en/latest/

```
make develop

. .env/bin/activate
```


# Testing

At the moment, most tests are integration tests with the AWS SDK. This means
that you will need to [set up your system][aws-setup] to access AWS resources
if you want to run the test suite.

[aws-setup]: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

```
py.test tests
```


# Quickstart

Define your infrastructure environment following the examples in the
[examples directory][examples-dir]. Then to create the environment:

[examples-dir]: https://github.com/humilis/humilis/tree/master/examples


````
humilis create example-environment.yml
````

To update the environment after it has been deployed:

````
humilis update example-environment.yml
````

And to delete it:

````
humilis delete example-environment.yml
````


# Humilis environments

A `humilis` environment is just a collection of cloudformation stacks that
are required for an application. Instead of having a monolytic CF template for
your complete application, `humilis` allows you to define infrastructure
_layers_ that are combined into an _environment_. Each `humilis` layer
translates exactly into one CF template (therefore into one CF stack after
the layer is deployed).

Breaking a complex infrastructure environment into smaller layers has at least
two obvious advantages:

* __Easier to maintain__. It's easier to maintain a simple layer that contains
just a bunch of [CF resources][cf-resource] than serve a well-defined
purpose.

* __Easier to reuse__. You should strive to define your infrastructure
layers in such a way that you can reuse them across various environments. For
instance, many projects may require a base layer that defines a VPC, a few
subnets, a gateway and some routing tables, and maybe a (managed) NAT. You
can define a humilis layer with those resources and have a set of layer
parameters (e.g. the VPC CIDR) that will allow you to easily reuse it across
environments.

[cf-resource]: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html


## Environment anatomy

An environment _definition file_ is a [yaml][yaml] document that specifies the
list of layers that form your enviroment. The file should be named as your
environment. That is, for environment `my-app-environment` the environment
description file should be called `my-app-environment.yaml`. The contents of
the environment definition should be organized as follows:

[yaml]: https://en.wikipedia.org/wiki/YAML

```
---
my-app-environment:
description:
A description of what this environment is for
layers:
# The layers that you environment requires. They will be deployed in the
# same order as you list them. Note that you can also pass parameters
# to a layer (more on that later).
- {layer: name_of_first_layer, layer_param: layer_value}
- {layer: name_of_second_layer}
- {layer: name_of_third_layer}
```

## Layer anatomy

Anything associated to a given layer must be stored in a directory with the
same name as the layer, within the same directory where the environment
_definition file_ is located. If we consider the `my-app-environment`
environment we used above then your directory tree should look like this:

```
.
├── my-app-environment.yaml
├── name_of_first_layer
│   ├── meta.yaml
│   └── resources.yaml
├── name_of_second_layer
│   ├── meta.json
│   └── meta.yaml
└── name_of_third_layer
├── resources.json.j2
└── resources.yaml.j2
```

A layer must contain at least two files:

* `meta.yaml`: Meta information about the layer such as a description,
dependencies with other layers, and layer parameters.
* `resources.yaml`: Basically a CF template with the resources that the layer
contains.

Those two files can also be in `.json` format (`meta.json` and
`resources.json`). Or you can add the extension `.j2` if you want the files to
be pre-processed with the [Jinja2][jinja2] template compiler.

[jinja2]: http://jinja.pocoo.org/

Below an example of how a layer `meta.yaml` may look like:

```
---
meta:
description:
Creates a VPC, that's it
parameters:
vpc_cidr:
description: The CIDR block of the VPC
value: 10.0.0.0/16
```

Above we declare only one layer parameter: `vpc_cidr`. `humilis` will make pass
that parameter to Jinja2 when compiling any template contained in the layer. So
the `resources.yaml.j2` for that same layer may look like this:

```
---
resources:
VPC:
Type: "AWS::EC2::VPC"
Properties:
CidrBlock: {{ vpc_cidr }}
```


# References

You can use references in your `meta.yaml` files to refer to thing other than
resources within the same layer (to refer to resources within a layer you can
simply use Cloudformation's [Ref][cf-ref] or [GetAtt][cf-getatt] functions).
Humilis references are used by setting the value of a layer parameter to a dict
that has a `ref` key. Below an a `meta.yaml` that refers to a resource (with
a logical name `VPC`) that is contained in another layer (called `vpc_layer`):

```
---
meta:
description:
Creates an EC2 instance in the vpc created by the vpc layer
dependencies:
- vpc
parameters:
vpc:
description: Physical ID of the VPC where the instance will be created
value:
ref:
parser: layer
parameters:
layer_name: vpc_layer
resource_name: VPC
```

Every reference must have a `parser` key that identifies the parser that
should be used to parse the reference. There are also two optional keys:

* `parameters`: allows you to pass parameters to the reference parser. You can
pass either named parameters (as a dict) or positional arguments (as a
list).

* `priority`: the parsing priority. Parameters with a lower value in `priority`
will be parsed before parameters with a higher value. This allows some
reference parsers to refer internally to other parameters within the same
layer. For example, the `lambda` parser, when parsing templated lambda code,
it uses previously parsed layer parameters as template parameters.

More information on the reference parsers that are bundled with humilis below.

[cf-ref]: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-ref.html
[cf-getatt]: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html


## Available reference parsers

### `layer_resource` references

`layer_resource` references allow you to refer to the physical ID of a resource
that is part of another layer.

__Parameters__:

* `layer_name`: The name of the layer you are referring to
* `resource_name`: The logical name of the layer resource


__Example__:

Consider the following environment definition:

```
---
my-environment:
description:
Creates a VPC with a NAT in the public subnet
layers:
- {layer: vpc}
- {layer: nat}
```

Obviously the `nat` layer that takes care of deploying the NAT in the public
subnet will need to know the physical ID of that subnet. You achieve this by
declaring a `layer_resource` reference in the `meta.yaml` for the `nat` layer:

```
---
meta:
description:
Creates a managed NAT in the public subnet of the NAT layer
parameters:
subnet_id:
description:
The physical ID of the subnet where the NAT will be placed
value:
ref:
parser: layer_resource
parameters:
layer_name: vpc
# The logical name of the subnet in the vpc layer
resource_name: PublicSubnet
```

When parsing `meta.yaml` humilis will replace this:

```
ref:
parser: layer_resource
parameters:
layer_name: vpc
# The logical name of the subnet in the vpc layer
resource_name: PublicSubnet
```

with the physical ID you need (something like `subnet-bafa90cd`). You can then
use this physical ID in the `resources.yaml.j2` section of the `nat` layer:

```
{# Pseudo-content of layers/nat/resources.yaml.j2 #}
resources:
{# An Elastic IP reservation that will be associated to the NAT #}
NatEip:
Type: 'AWS::EC2::EIP'
Properties: {}
{# Custom resource deploying the NAT #}
NatGateway:
Type: 'Custom::NatGateway',
Properties:
{# The ARN of the Lambda function backing the custom resource #}
ServiceToken: 'arn:aws:lambda:eu-west-1:XXXX:function:CreateNatGateway'
{# Here we use the subnet_id reference defined in meta.yaml #}
SubnetId: {{subnet_id}}
AllocationId:
Ref: NatEip
```

### `environment_resource` references

`environment_output` references allow you to refer to resources that belong
to other humilis environments.


__Parameters__:

* `environment_name`: The name of the environment you are referring to
* `layer_name`: The name of the layer you are referring to
* `resource_name`: The logical name of the layer resource


### `layer_output` references

`layer_output` references allow you to refer to outputs produced by another
layer.

__Parameters__:

* `layer_name`: The name of the layer you are referring to
* `output_name`: The logical name of the output parameter

In general you should prefer using `layer_output` references over
`layer_resource` references. The output parameters produced by a layer define
an informal _layer interface_ that is more likely to remain constant than the
logical names of resources within a layer.


### `boto3` references

`boto3` references define arbitrary calls to [boto3facade][boto3facade]. The
latter is just a simpler facade interface on top of [boto3][boto3].

[boto3]: https://github.com/boto/boto3
[boto3facade]: https://github.com/InnovativeTravel/boto3facade


__Parameters__:

* `service`: The AWS service, e.g. `ec2` or `cloudformation`. Note that only
only AWS services that have a facade in [boto3facade][boto3facade] are
supported.
* `call`: The corresponding facade method, e.g. `get_ami_by_name`. The value of
this parameter must be a dictionary with a `method` key (the name of the
facade method to invoke) and an optional `args` key (the parameters to pass to
the facade method). Best to look at the example below to understand how this
works.
* `output_attribute`: Optional. If provided the reference parser will return the
value of this attribute from the object returned by the facade method.

Below an example of a layer that uses a `boto3` reference:

```
---
meta:
description:
Creates an EC2 instance using a named AMI
# More stuff omitted for brevity
ami:
description: The AMI to use when launching the EC2 instance
value:
ref:
parser: boto3
parameters:
service: ec2
call:
method: get_ami_by_name
args:
- test-ami
output_attribute: id
```

`humilis` will parse the reference using this code:

```
# Import the Ec2 facade
from boto3facade.ec2 import Ec2

# Create a facade object
ec2_facade = Ec2()

# Make the call
ami = ec2_facade.get_ami_by_name('test-ami')

# Extract the requested attribute
ref_value = ami.id
```


### `file` references

`file` references allow you to refer to a local file. The file will be uploaded
to S3 and the reference will evaluate to the corresponding S3 path.

__Parameters__:

* `path`: The path to the file, relative to the layer root directory.


### `lambda` references

`lambda` references allow you to refer to some Python code in your local
machine. If your code follows some simple conventions `humilis` will take care
of building a [deployment package][aws-lambda-deploy] for you, uploading it
to S3, and the reference will evaluate to the S3 path of the deployment
package.

[aws-lambda-deploy]: http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

__Parameters__:

* `path`: Path to either a completely self-contained `.py` file, or to the root
directory of your lambda code. In the latter case your code needs to follow
some simple conventions for this to work. More information below.

* `dependencies`: A list of dependencies to be included in the Lambda
deployment package. Dependencies may be either pip installable packages, or
paths to local Python packages or modules, or paths to local
`requirements` files.


__Example__:

```
ref:
parser: lambda
parameters:
# Path to the root directory containing your lambda code
path: dummy_function
dependencies:
# The Lambda code requires Pypi's pyyaml
- pyyaml
# It also requires a local package in this path
- mycode/mypkgdir
# And this local module
- mycode/mymodule.py
```

which will evaluate to a S3 path such as:

```
s3://[bucket_name]/[environment_name]/[stage_name]/[func_name]-[commithash].zip
```


__Code conventions__:

Following the example above, the contents of the layer responsible of deploying
the `dummy_function` lambda may look like this:

```
.
├── dummy_function
│   ├── dummy_function.py
│   └── setup.py
├── meta.yaml
├── outputs.yaml.j2
└── resources.yaml.j2
```

Basically all your code needs to be included under directory `dummy_function`.
In this case there is only one file: `dummy_function.py`. External dependencies
need to be specified in your `setup.py`.


### `secret` references

`secret` references retrieve a secret using Python's [keyring][keyrig] module.

[keyring]: https://pypi.python.org/pypi/keyring


__Parameters__:

* `service`: The name of the service the secret is associated to.

* `key`: The key (e.g. the username) that identifies the secret.


__Example__:

```
ref:
parser: secret
parameters: {"service": "mysqldb", "key": "adminuser"}
```


## Custom Jinja2 filters

Humilis defines the following [custom Jinja2 filters][jinja2filters]:

[jinja2filters]: http://jinja.pocoo.org/docs/dev/templates/#filters

* `uuid`: A random UUID. Example: `{{''|uuid}}`.
* `password(length=8)`: A random password. Example: `{{10|password}}`.



## Secrets vault

If your environment includes a [secrets vault layer][secrets-vault] you can use
humilis to easily store secrets in the vault:

[secrets-vault]: https://github.com/humilis/humilis-secrets-vault

```
humilis set-secret --stage [STAGE] [ENVIRONMENT_FILE] [SECRET_KEY] [SECRET_VAL]
```

You can test that the secret was properly stored using the `get-secret`
command:

```
humilis set-secret --stage [STAGE] [ENVIRONMENT_FILE] [SECRET_KEY]
```

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

humilis-0.6.6.tar.gz (25.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page