lmdo 2.1.2

lmdo
====
A CD/CI tool for developing micro-services components using AWS Lambda function (python2.7) and managing other AWS resources.

Inspirations
------------
[Apex](https://github.com/apex/apex), [Serverless](https://github.com/serverless/serverless),
[Zappa](https://github.com/Miserlou/Zappa),
[sceptre](https://github.com/cloudreach/sceptre)

Why
---
Most of the open-source apps are very much opinionated and the model they employ doesn't always fit for the actual individual use case. In fact, there aren't a lot of flexibility provided. In addition, abstraction makes it hard to diagnose issues.

The tool I have in mind should allow raw inputs, be atomic, has important functionalities like the others and easy to use to a certain degree (You need to know what you are doing. E.g. I don't expect you to use it for creating stack if you don't know how to write a raw CloudFormation). Hence born lmdo.

Features
--------
- Initialize project via Github boiler plate
- Use of CloudFormation templates in either json or yaml format
- Use of CloudFormation paramter files in either json or yaml format
- Manage one or more CloudFormation stacks
- Use of swagger template for API Gateway
- Manage API Gateway resources like deployments and stages
- Automatically generate API Gateway for Lambda functions
- Manage life cycles of AWS Lambda functions
- Offer two type of managed Lambda functions: Django wsgi wrapper and CloudWatch Event scheduler dispatcher
- Maintain Lambda function heart beat
- CloudWatch log output on CL
- Upload any files to S3 bucket

Contents:
---------
1. [Installation](#installation)
2. [Project initiation](#project-initiation)
3. [Basic configuration](#basic-configuration)
4. [One step deployment](#one-step-deployment)
5. [CloudFormation](#cloudformation)
4. [Lambda function](#lambda-function)
5. [API Gateway](#api-gateway)
6. [CloudWatch events](#cloudwatch-events)
7. [CloudWatch logs](#cloudwatch-logs)
8. [S3 Upload](#s3-upload)

Installation
------------
Installing via pypi:

$ sudo pip install lmdo

Installing via code (Recommended, as lmdo is under active development at the moment):

$ git pull https://github.com/MerlinTechnology/lmdo.git
$ cd lmdo
$ sudo pip install -U ./

**Note**: All lmdo commands need to be run at the same directory of the `lmdo.yaml` file

Project Initiation
------------------
To initiate your project, run:

$ lmdo init <project_name>

This will create you named project folder and the sample lmdo configuration file `lmdo.yaml`.

If you already have an existing project, you can run:

$ lmdo init config

The configuration file `lmdo.yaml` will be copied to your current directory. If there is already one, the new configuration file will be renamed to `lmdo.yaml.copy`

To start a project by using a github boiler plate, run:

$ lmdo bp fetch <url>

The repo will then be copied from github to your current project folder without all the git folders or files


Basic Configuration
-------------------
1. AWS credentials

You can either use session (`Profile`) or configure AWS key and secret (`Region, AWSKey, AWSSecret`) in `lmdo.yaml`

When using session, you will need to create two files:

~/.aws/config and ~/.aws/credentials

Details please ref to [AWS CLI](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)

**Note**: If explicitly using config options `Region, AWSKey, AWSSecret`, it's recommended to define them in the environment. Using syntax like `$env|YOUR_ENV_VAR` lmdo will replace them with the actual environment value.

2. Other mandatory configuration Options

`Service`: The name of your service/project

`User`: The user that deploys the service/project

`Stage`: The deployment stage


One Step Deployment
-------------------
To deploy your entire service, run:

$ lmdo deploy

To delete, run:

$ lmdo destroy


CloudFormation
--------------
### Available reserved utility variables (`WARNING`, see Note)
They will be replaced with correct value during deployment

`$env|ENV_VAR_NAME`: Environment variables, can be used both in parameters and templates.

`$template|template-file-name`: Nested stack template to be used to construct proper S3 bucket url for stack resource `TemplateURL`, mostly used in templates.

`$stack|stack-name::output-key`: The value of an existing stack's output based on key name. Can be used both in parameters and templates.

**Note**:

- the stack referring to must exist before deployment.
- Recommand to avoid use those variable in the template so to keep the template complying to AWS. Instead, use those variables in the parameter file to pass in as parameters

### Configuration examples:

1. Single CloudFormation template without parameters

```
CloudFormation:
Stacks:
- Name: your-stack-name
TemplatePath: relative/path/to/template
```

2. Single CloudFormation template with parameters. You can either provide a single file or a directory that contains all the parameter files. If a directory is provided, lmdo will combine all files into one during deployment.

```
CloudFormation:
Stacks:
- Name: your-stack-name
TemplatePath: relative/path/to/template
ParamsPath: relative/path/to/params/file/or/directory
```

3. CloudFormation using S3 bucket

```
CloudFormation:
S3Bucket: your.bucket.url
Stacks:
- Name: your-stack-name
TemplatePath: relative/path/to/template
ParamsPath: relative/path/to/params/file/or/directory
```

4. Single CloudFormation template with nested stacks

```
CloudFormation:
S3Bucket: your.bucket.url
TemplateRepoPath: relative/path/to/nested/stack/template/directory
Stacks:
- Name: your-stack-name
TemplatePath: relative/path/to/template
ParamsPath: relative/path/to/params/file/or/directory
```

**Note**:

a. You must provide `S3Bucket` for nested stacks as it'll be used for uploading all the templates to.

b. All nested stack templates must reside in `TemplateRepoPath`. If not given, lmdo will look for nested stack template (see point **c** below) from the project folder by default.

c. Using syntax like `TemplateURL: $template|your-nested-stack-template-file-name` in your master template stack resource, lmdo will replace the syntax to appropriate S3 url.

d. You can use `DisablePrefix` option to create stack with exact name you give

5. Multiple CloudFormation Stacks

```
CloudFormation:
S3Bucket: your.bucket.url
TemplateRepoPath: relative/path/to/nested/stack/template/directory
Stacks:
- Name: your-stack-name-1
TemplatePath: relative/path/to/template-1
ParamsPath: relative/path/to/params/file-1/or/directory-1
- Name: your-stack-name-2
TemplatePath: relative/path/to/template-2
ParamsPath: relative/path/to/params/file-2/or/directory-2
```

### Parameter file
Parameter file can be in either `.json` or `.yaml` format.

For json file, you can use two types of syntax:

1. Standard AWS stack parameter format

```
[
{
"ParameterKey": "your-parameter-key-1",
"ParameterValue": "your-parameter-value-1"
},
{
"ParameterKey": "your-parameter-key-2",
"ParameterValue": "your-parameter-value-2"
}
]
```

2. lmdo json format

```
{
"your-parameter-key-1": "your-parameter-value-1",
"your-parameter-key-2": "your-parameter-value-2"
}
```

3. For yaml file, the format as follow:

```
your-parameter-key-1: your-parameter-value-1
your-parameter-key-2: your-parameter-value-2
```

4. Available reserved utility variables

They will be replaced with correct value during deployment

`$env|ENV_VAR_NAME`: Environment variables, can be used both in parameters and templates.

`$stack|stack-name::output-key`: The value of an existing stack's output based on key name. Can be used both in parameters and templates.

`$template|template-file-name`: Nested stack template to be used to construct proper S3 bucket url for stack resource `TemplateURL`, mostly used in templates.

**Note**:

The stack referring to must exist before deployment.

For `CommaDelimitedList` type, you can do `"$env|ENV_VAR_NAME1, $env|ENV_VAR_NAME2"`. Same to `$stack|*`.

### Commands

To create your CloudFormation, run:

$ lmdo cf create

To update or delete, run the similar command using `update` or `delete` keyword

To use change-set instead of directly update stack, use `-c` or `--change_set` option:

$ lmdo cf create -c

Stack event will be output by default, if you want to hide it, use `-he` or `--hide-event` option:

$ lmdo cf create -he

For only change one specific stack, use option `--stack=`


Lambda Function
---------------
lmdo facilitates packaging, uploading and managing your lambda function. Out of box, it also provides support for two types of lambda function wrapper: wsgi and event dispatcher apart from the standard Lambda function.

### Basic configuration structure


VirtualEnv: False
Lambda:
- function-1 configuration
- function-2 configuration
...

**Note**:
- If you are using virtualenv, please set `VirtualEnv` to `True`
- The actual deployed function name created by lmdo will be using `<user>-<stage>-<service-name>-<FunctionName>`

### Optional configurations and their default values available for all function types

`Type`: default `default`. Other availabe types are `wsgi` and `cron_dispatcher`

`MemorySize`: default `128`

`Runtime`: default `python2.7` (Note: lmdo only support python at the moment)

`Timeout`: default `180`

`HeatUp`: default `False`. Provide CPR for lambda function to keep container alive. Only avaiable for `wsgi` and `default` functions.

`HeatRate`: default `rate(4 minutes)` before container becomes inactive. Only avaiable for `wsgi` and `default` functions.

VPC configuration:

VpcConfig:
SecurityGroupIds:
- security-group-id-1
- security-group-id-2
SubnetIds:
- subnet-id-1
- subnet-id-2

Runtime environment variables

EnvironmentVariables:
MYSQL_HOST: host-url
MYSQL_PASSWORD: password
MYSQL_USERNAME: username
MYSQL_DATABASE: dbname

Role and policies:

RoleArn: your-role-arn

or

RolePolicy:
AssumeRoles:
- sns.amazonaws.com
PolicyDocument: file/path/to/your/policy
ManagedPolicyArns:
- your-managed-policy-arn

**Note**:
- `$region` and `$accountId` are available to use in your `PolicyDocument` so you don't need to hard-code them
- Only one of `RoleArn` and `RolePolicy` required.
- If both provided, `RolePolicy` takes over.
- If none provided, lmdo will create a default role with default policy
- the default role will assume role of apigateway, lambda, events, ec2.
- the default policy will allow `lambda:InvokeFunction`, `lambda:AddPermission`, `lambda:RemovePermission` on the lambda function, `log:CreateLogGroup`, `logs:CreateLogStream`, `logs:PutLogEvents`, `ec2:DescribeNetworkInterfaces`, `ec2:CreateNetworkInterface` and `ec2:DeleteNetworkInterface` actions
- Only extra assume roles and policies need to be configured other than the default

### Examples
1. Standard lambda function

Requirements:

The invokable lambda function files need to be placed on the top level of the project folder.

Put all your dependent packages in `requirements.txt`

Configuration:

```
Lambda:
- S3Bucket: lambda.bucket.name
FunctionName: your-function-name
Handler: handler.fly
```

2. Django wsgi lambda function

It wraps up Django and bridge between API gateway and your Django

Requirements:

Put all your dependent packages in `requirements.txt`

Optional configuration:

`CognitoUserPoolId`: It will set the API gateway authentication if provided. You can only have one per `ApiBasePath`

Configuration:

```
Lambda:
- S3Bucket: lambda.bucket.name
Type: wsgi
DisableApiGateway: False
ApiBasePath: /path
FunctionName: your-function-name
EnvironmentVariables:
DJANGO_SETTINGS_MODULE: mysite.settings
```

**Note**:

By default, `DisableApiGateway` is set to `False`. You must set your `ApiBasePath` when it's `False`

`DJANGO_SETTINGS_MODULE` environment variable is a must for it to work

3. Cron dispatcher

Cron dispatcher function allows you to create multiple event schedulers on different functions via a single dispatcher.
**Note**: lmdo will construct rule name based on `<user>-<stage>-<service-name>-<FunctionName>--<handler>`. CloudWatch events rule name can only be within 64 characters, so mind your names.

Configuration:

```
Lambda:
- S3Bucket: lambda.bucket.name
Type: cron_dispatcher
FunctionName: your-dispatcher-name
RuleHandlers:
- Handler: your.module.handler
Rate: your cron string e.g. Rate(1 minutes)
```

### Available reserved utility variables

They will be replaced with correct value during deployment

`$stack|stack-name::output-key`: The value of an existing stack's output based on key name. Can be used both in parameters and templates.

**Note**:

The stack referring to MUST exist before deployment.

For `CommaDelimitedList` type, you can do `"$stack|stack-name::key1, $stack|stack-name::key2"`.

### Commands

To create all functions, run (update/delete similar):

$ lmdo lm create

Options:
`--function-name`: Only action on a particular function

To package the function only:

$ lmdo package --function-name=your-function-name

API Gateway
---------------
Swagger template is used to create API Gateway

### Requirments

* A folder named 'swagger' under your project folder
* Name your swagger template as `apigateway.json`

### Configuration

ApiGatewayName: Your unique Apigateway name

Optionally, you can use `ApiVarMapToFile` to map your custom key to a file for replacement

ApiVarMapToFile:
$mappingKey: file/path/name

You can also use `ApiVarMapToVar` to map any string values to your defined key. For this mapping, you can also use `$stack` and `$lambda-arn` utitlity variable for the value.

ApiVarMapToVar:
$mappingKey1: value
$mappingKey2: $stack|stack-name::key1
$mappingKey3: $lambda-arn|lambda-name


**NOTE:** Please name your version as `$version` and your title as `$title` so that Lmdo can update it during creation using the value of `ApiGatewayName` in your lmdo.yaml

### Commands

To manage your APIGateway resource, run(update/delete similar):

$ lmdo api create

You can create or delete a stage by running:

$ lmdo api create-stage <from_stage> <to_stage>

or

$ lmdo api delete-stage <from_stage>

CloudWatch events
-----------------

### Configuration

CloudWatchEvent:
- Name: rule_name
ScheduleExpression: [schedule-expression]
EventPatternFile: [path/to/pattern/file]
Targets:
- Type: default
Arn: aws-resource-arn
- Type: local
FunctionName: local-function-name

`[schedule-expression]`: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
`[path/to/pattern/file]`: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatchEventsandEventPatterns.html

Options:

`Description`: description of your rule

`DisablePrefix`: default to `False`. If `True`, lmdo will use your rule name instead of using `<user>-<stage>-<service-name>-<rule_name>`

`RoleArn`: default lmdo will create lambda invokable role

**Note**

When target type is `local`, lmdo will replace it with your function ARN


CloudWatch Logs
-------------------

### Commands

You can tail any AWS cloudwatch group logs by running:

$ lmdo logs tail your_log_group_name

`--day`: defines how many days ago the logs need to be retrieved

`--start-date` or `--end-date`: specify a start date and/or end date for the log entries using format `YYYY-MM-DD`

You can also tail logs of your lambda function in your project by running:

$ lmdo logs tail function your-function-name

`your-function-name` is the function name you configure in your lmdo.yaml

S3 Upload
------

lmdo offers a simple command line to upload your local static asset into a S3 bucket.

### Configuration

AssetS3Bucket: your.bucket.url
AssetDirectory: directory/where/your/assets/in

### Commands

To upload (Note: it doesn't delete files):

$ lmdo s3 sync