Skip to main content

Simple solution to backup ec2 instances using snapshots

Project description

Simple EC2 Snapshot
===================

Simple solution to backup ec2 instances using snapshots

With Simple EC2 Snapshot supports:

* Hot snapshots (by default) and Cold snapshots
* Multiple instances snapshot in one line
* Detection of doubles
* Filters by tags (allowing wildcards) or by instance IDs
* Credentials file multiple with profiles
* Limit the number of snapshots
* Restrict snapshots to data disks only
* Snapshot retention over time or for a given number

## Installation

To install the tool, the simplest solution is to use pip:
```
pip install simplec2snap
```

## Filters

You can first decide to choose what to backup with Instance ID or Tags. You can set multiple tags and/or multiple instance ID at the same time.

For example, if I want to snapshot 2 instances in the same command line, it should looks like this:
```
> ./simplec2snap.py -i i-ad0fcc4b -i i-56489db2
```

To snapshot multiple instances by selecting multiple tags. Here is an example with 2 tagsi, so it should match both:
```
> ./simplec2snap.py -t Name 'instance*' -t env prod
```

If you want to add an instance in addition of the previous tags:
```
> ./simplec2snap.py -t Name 'instance*' -t env prod -i i-ad0fcc4b
```

## Credentials file

You can use a credentials file with several profiles. It should looks like this:
```ini
[default]
aws_region = <region>
aws_access_key_id = <key_id>
aws_secret_access_key = <access_key>

#[profile profile1]
#aws_region = <region>
#aws_access_key_id = <key_id>
#aws_secret_access_key = <access_key>
```
The default one should be located in '~/.aws_cred'. You can override this with '-c' argument and '-p' to specify the profile fulfill into brackets.

## Dry Run mode

Use the dry run mode (enabled by default) to see what actions will be performed when selecting a tag Name or an instance:
```
> ./simplec2snap.py -t Name "instance-name*"
2015-01-26 17:05:25,954 [INFO] == Launching dry run mode ==
2015-01-26 17:05:25,954 [INFO] Connecting to AWS
2015-01-26 17:05:25,955 [INFO] Getting instances information
2015-01-26 17:05:28,341 [INFO] Working on instance i-e16cc205 (instance-name2)
2015-01-26 17:05:28,341 [INFO] Hot snapshot made for vol-fa415bfd(/dev/sdb)
2015-01-26 17:05:28,341 [INFO] Hot snapshot made for vol-22465c25(/dev/sda)
2015-01-26 17:05:28,341 [INFO] Working on instance i-6f6ec08b (instance-name1)
2015-01-26 17:05:28,341 [INFO] Hot snapshot made for vol-9d465c9a(/dev/sda)
2015-01-26 17:05:28,341 [INFO] Hot snapshot made for vol-9c465c9b(/dev/sdb)
```

## Run mode

If you're ok with the previous dry run, then add '-u' for run mode:
```
> ./simplec2snap.py -t Name "instance-name*" -u
2015-01-26 17:06:19,163 [INFO] == Launching run mode ==
2015-01-26 17:06:19,163 [INFO] Connecting to AWS
2015-01-26 17:06:19,163 [INFO] Getting instances information
2015-01-26 17:06:21,083 [INFO] Working on instance i-e16cc205 (instance-name2)
2015-01-26 17:06:21,352 [INFO] Hot snapshot made for vol-fa415bfd(/dev/sdb) - snap-35adb8c4
2015-01-26 17:06:21,587 [INFO] Hot snapshot made for vol-22465c25(/dev/sda) - snap-36adb8c7
2015-01-26 17:06:21,587 [INFO] Working on instance i-6f6ec08b (instance-name1)
2015-01-26 17:06:21,832 [INFO] Hot snapshot made for vol-9d465c9a(/dev/sda) - snap-3cadb8cd
2015-01-26 17:06:22,087 [INFO] Hot snapshot made for vol-9c465c9b(/dev/sdb) - snap-21adb8d0
```

## Hot vs Cold snapshot

By default Hot mode is selected to perform snapshot without stopping instances. However, this may not be the best choice in some case, like for database purpose. To get a full consistent snapshot of your EC2 with attached EBS, you have to make a Cold snapshot which involves to shutdown, snapshot and start instance.

To do so, you have to add '-H' option:
```
> ./simplec2snap.py -t Name "instance-name*" -u -H
2015-01-26 17:07:10,281 [INFO] == Launching run mode ==
2015-01-26 17:07:10,281 [INFO] Connecting to AWS
2015-01-26 17:07:10,282 [INFO] Getting instances information
2015-01-26 17:07:12,490 [INFO] Working on instance i-e16cc205 (instance-name2)
2015-01-26 17:07:12,490 [INFO] Instance is going to be shutdown
2015-01-26 17:07:48,871 [INFO] Instance i-e16cc205 now stopped !
2015-01-26 17:07:49,134 [INFO] Cold snapshot made for vol-fa415bfd(/dev/sdb) - snap-a8afba59
2015-01-26 17:07:49,379 [INFO] Cold snapshot made for vol-22465c25(/dev/sda) - snap-adafba5c
2015-01-26 17:07:49,379 [INFO] Instance is going to be started
2015-01-26 17:08:20,565 [INFO] Instance i-e16cc205 now running !
2015-01-26 17:08:20,565 [INFO] Working on instance i-6f6ec08b (instance-name1)
2015-01-26 17:08:20,565 [INFO] Instance is going to be shutdown
2015-01-26 17:08:51,617 [INFO] Instance i-6f6ec08b now stopped !
2015-01-26 17:08:51,853 [INFO] Cold snapshot made for vol-9d465c9a(/dev/sda) - snap-b1aebb40
2015-01-26 17:08:52,098 [INFO] Cold snapshot made for vol-9c465c9b(/dev/sdb) - snap-b2aebb43
2015-01-26 17:08:52,098 [INFO] Instance is going to be started
2015-01-26 17:09:09,467 [INFO] Instance i-6f6ec08b now running !
```

## Limit snapshots for auto-scaling group

In auto-scaling groups, you normally have x time the same running intance. Snapshoting a huge number of time the same instance may not be very interesting. That's why you can limit the number of snapshot by using '-l' command followed by the number of desired snapshot. If I only want one:
```
> ./simplec2snap.py -t Name "instance-name*" -l 1
2015-01-26 17:11:27,561 [INFO] == Launching dry run mode ==
2015-01-26 17:11:27,561 [INFO] Connecting to AWS
2015-01-26 17:11:27,562 [INFO] Getting instances information
2015-01-26 17:11:29,659 [INFO] Working on instance i-e16cc205 (instance-name2)
2015-01-26 17:11:29,659 [INFO] Hot snapshot made for vol-fa415bfd(/dev/sdb)
2015-01-26 17:11:29,659 [INFO] Hot snapshot made for vol-22465c25(/dev/sda)
2015-01-26 17:11:29,659 [INFO] Working on instance i-6f6ec08b (instance-name1)
2015-01-26 17:11:29,660 [INFO] The requested limit of snapshots has been reached: 1
```

## Remove root device from snapshots

Still for auto-scaling groups, your root device may not be required to snapshot. Generally because it may be builded from a configuration manager and you just don't care of it. So the goal is to remove it from the snapshot list, you can so use '-o' option:

```
> ./simplec2snap.py -t Name "instance-name*" -o
2015-01-26 17:11:50,757 [INFO] == Launching dry run mode ==
2015-01-26 17:11:50,757 [INFO] Connecting to AWS
2015-01-26 17:11:50,758 [INFO] Getting instances information
2015-01-26 17:11:52,708 [INFO] Working on instance i-e16cc205 (instance-name2)
2015-01-26 17:11:52,708 [INFO] Hot snapshot made for vol-fa415bfd(/dev/sdb)
2015-01-26 17:11:52,708 [INFO] Working on instance i-6f6ec08b (instance-name1)
2015-01-26 17:11:52,708 [INFO] Hot snapshot made for vol-9c465c9b(/dev/sdb)
```

## Snapshot retention

There are 2 methods for snapshot retention. Choose the best one for your needs.

### Keep snapshots over time

You can define the retention of your backups. You need to specify 2 args:

* Number: sepcify a number for day, week... which is defined in the second arg
* Time element: specify s(second), m(min), h(hour), d(day), w(week), M(month), y(year)

So for example, if you want to keep snapshots for 3 weeks and delete the old ones, you have to set: 3 w.

Here is a basic example where I want to delete snapshots older than 10 days:
```
> ./simplec2snap.py -t Name "instance-name*" -n -g 10 d
2015-01-28 10:12:11,216 [INFO] == Launching dry run mode ==
2015-01-28 10:12:11,217 [INFO] Connecting to AWS
2015-01-28 10:12:11,217 [INFO] Getting instances information
2015-01-28 10:12:13,080 [INFO] Working on instance i-e16cc205 (pmavro-test2)
2015-01-28 10:12:14,109 [INFO] Deleting snapshot snap-b427c144 (vol-fa415bfd|/dev/sdb)
2015-01-28 10:12:14,205 [INFO] Deleting snapshot snap-a327c153 (vol-22465c25|/dev/sda)
2015-01-28 10:12:14,205 [INFO] Working on instance i-6f6ec08b (pmavro-test1)
2015-01-28 10:12:14,321 [INFO] Deleting snapshot snap-a627c156 (vol-9d465c9a|/dev/sda)
2015-01-28 10:12:14,440 [INFO] Deleting snapshot snap-a927c159 (vol-9c465c9b|/dev/sdb)
```

Here -n is used to not make snapshots, only delete olds. But you can ask on the same line to make snapshots AND remove old ones:
```
> ./simplec2snap.py -t Name "instance-name*" -g 10 m
2015-01-26 17:22:43,263 [INFO] == Launching dry run mode ==
2015-01-26 17:22:43,263 [INFO] Connecting to AWS
2015-01-26 17:22:43,264 [INFO] Getting instances information
2015-01-26 17:22:46,217 [INFO] Working on instance i-e16cc205 (instance-name2)
2015-01-26 17:22:46,218 [INFO] Hot snapshot made for vol-fa415bfd(/dev/sdb)
2015-01-26 17:22:46,218 [INFO] Hot snapshot made for vol-22465c25(/dev/sda)
2015-01-26 17:22:47,328 [INFO] Deleting snapshot snap-a8afba59 (vol-fa415bfd|/dev/sdb)
2015-01-26 17:22:47,491 [INFO] Deleting snapshot snap-adafba5c (vol-22465c25|/dev/sda)
2015-01-26 17:22:47,491 [INFO] Working on instance i-6f6ec08b (instance-name1)
2015-01-26 17:22:47,491 [INFO] Hot snapshot made for vol-9d465c9a(/dev/sda)
2015-01-26 17:22:47,492 [INFO] Hot snapshot made for vol-9c465c9b(/dev/sdb)
2015-01-26 17:22:47,669 [INFO] Deleting snapshot snap-b1aebb40 (vol-9d465c9a|/dev/sda)
2015-01-26 17:22:47,842 [INFO] Deleting snapshot snap-b2aebb43 (vol-9c465c9b|/dev/sdb)
```

### Keep at least a given number of snapshots

Another solution to manage the retention of your snapshots is to specify how many snapshots you want to keep. For example, if I have 5 snapshots per device of an instance and want to keep the last 4 ones:
```
> ./simplec2snap.py -t Name "instance-name*" -n -d 4
2015-01-28 10:14:02,713 [INFO] == Launching dry run mode ==
2015-01-28 10:14:02,713 [INFO] Connecting to AWS
2015-01-28 10:14:02,713 [INFO] Getting instances information
2015-01-28 10:14:04,328 [INFO] Working on instance i-e16cc205 (pmavro-test2)
2015-01-28 10:14:05,316 [INFO] Deleting snapshot snap-b427c144 (vol-fa415bfd|/dev/sdb)
2015-01-28 10:14:05,444 [INFO] Deleting snapshot snap-a327c153 (vol-22465c25|/dev/sda)
2015-01-28 10:14:05,444 [INFO] Working on instance i-6f6ec08b (pmavro-test1)
2015-01-28 10:14:05,540 [INFO] Deleting snapshot snap-a627c156 (vol-9d465c9a|/dev/sda)
2015-01-28 10:14:05,654 [INFO] Deleting snapshot snap-a927c159 (vol-9c465c9b|/dev/sdb)
```

## Help

Here is the help with the complete list of options:
```
> ./simplec2snap.py
usage: simplec2snap.py [-h] [-r REGION] [-k KEY_ID] [-a ACCESS_KEY]
[-c CREDENTIALS] [-p CRED_PROFILE] [-i INSTANCE_ID]
[-t ARG ARG] [-u] [-l LIMIT] [-H] [-m COLDSNAP_TIMEOUT]
[-o] [-g ARG ARG] [-d KEEP_LAST_SNAPSHOTS] [-n]
[-f FILE] [-s] [-v LEVEL] [-V]

Simple EC2 Snapshot utility

optional arguments:
-h, --help show this help message and exit
-r REGION, --region REGION
Set AWS region (ex: eu-west-1) (default: None)
-k KEY_ID, --key_id KEY_ID
Set AWS Key ID (default: None)
-a ACCESS_KEY, --access_key ACCESS_KEY
Set AWS Access Key (default: None)
-c CREDENTIALS, --credentials CREDENTIALS
Credentials file path (default:
/home/pmavro/.aws_cred)
-p CRED_PROFILE, --profile CRED_PROFILE
Credentials profile file defined in credentials file
(default: default)
-i INSTANCE_ID, --instance INSTANCE_ID
Instance ID (ex: i-00000000 or all) (default: [])
-t ARG ARG, --tags ARG ARG
Select tags with values (ex: tagname value) (default:
[])
-u, --dry_run Define if it should make snapshot or just dry run
(default: True)
-l LIMIT, --limit LIMIT
Limit the number of snapshot (can be usefull with
auto-scaling groups) (default: -1)
-H, --cold_snap Make cold snapshot for a better consistency
(Recommended) (default: False)
-m COLDSNAP_TIMEOUT, --timeout COLDSNAP_TIMEOUT
Instance timeout (in seconds) for stop and start
during a cold snapshot (default: 600)
-o, --no_root_device Do not snapshot root device (default: False)
-g ARG ARG, --max_age ARG ARG
Maximum snapshot age to keep (<int> <s/m/h/d/w/M/y>)
(ex: 1 h for one hour) (default: [])
-d KEEP_LAST_SNAPSHOTS, --keep_last_snapshots KEEP_LAST_SNAPSHOTS
Keep the x last snapshots (default: 0)
-n, --no_snap Do not make snapshot (useful when combien to -g
option) (default: False)
-f FILE, --file_output FILE
Set an output file (default: None)
-s, --stdout Log output to console (stdout) (default: True)
-v LEVEL, --verbosity LEVEL
Verbosity level: DEBUG/INFO/ERROR/CRITICAL (default:
INFO)
-V, --version Print version number
```

Project details


Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page