skip to navigation
skip to content

Not Logged In

snaptastic 0.2.12

Snaptastic is a python tool to enable easy snapshotting and mounting of snapshots on AWS/EC2 EBS volumes.

Snaptastic by Mike Ryan & Thierry Schellenbach ([mellowmorning.com](http://www.mellowmorning.com/))
-------------------------------------------------------------------------------------------------

Snaptastic allows you to automate the snapshotting (backup) and mounting of volumes
on your EC2 instances.

It uses tagging of snapshots to figure out which snapshot should be used to populate a volume upon boot.

About the Author
----------------

 - Thierry Schellenbach, Founder & CTO at Fashiolista
 - Mike Ryan, Syadmin at Fashiolista
 - Peter van Kampen, Developer at Fashiolista

![Travis CI](https://secure.travis-ci.org/tschellenbach/Snaptastic.png?branch=master "Travis CI")

Install
-------

**pip**

```python
pip install snaptastic
```

**settings file**

Create a settings file
**/etc/snaptastic/snaptastic_settings.py**
```python
AWS_ACCESS_KEY_ID = 'key'
AWS_SECRET_ACCESS_KEY = 'secret'
REGION = 'eu-west-1'

from snaptastic import Snapshotter, EBSVolume


class SolrSnapshotter(Snapshotter):
    name = 'solr'

    def get_volumes(self):
        volume = EBSVolume('/dev/sdf', '/mnt/solr', size=20)
        volumes = [volume]
        return volumes
```

**verifying**

```python
sudo snaptastic test
sudo snaptastic list-volumes solr
```

That's it, you're now ready to use snaptastic.

Tutorial
--------

**mount**

```python
snaptastic mount-snapshots solr
```
you should now have a mounted volume on /mnt/solr/

```bash
lets add a file to it
sudo touch /mnt/solr/helloworld
```

**make snapshots**

time to make a snapshot of this important change
```python
snaptastic make-snapshots solr
```

**destroying our work**
```python
snaptastic unmount-snapshots solr
```
check to see if /mnt/solr/ is actually gone

**restoring from backups**
```python
snaptastic mount-snapshots solr
```

**changing the loglevel for a cronjob**
```python
snaptastic mount-snapshots solr --loglevel=WARNING
```



customizing the tagging
-----------------------

When mounting volumes, snaptastic will search for snapshots with the correct tags.
By default it will look for:

 * role
 * cluster
 * environment
 * mount point (you're advised not to change this)

To uniquely identify snapshots to application logic.
Changing the tags which are used to fit with your application setup is trivial.
Simply subclass the snapshotter class and change the get_filter_tags function.
Have a look at the example below:

```python
class CustomFilterSnapshotter(Snapshotter):
    name = 'filter_example'

    def get_filter_tags(self):
        '''
        The tags which are used for finding the correct snapshot to load from.
        In addition to these tags, mount point is also always added.

        Use these to unique identify different parts of your infrastructure
        '''
        tags = {
            'group': self.userdata['group'],
        }
        return tags
```


settings file
-------------

For more examples see examples.py

Snaptastic searches for its setting file at:
* snaptastic_settings on sys.path
* /etc/snaptastic_settings.py
* /etc/snaptastic/snaptastic_settings.py

hooks
-----

Snaptastic defines several hooks to allow custom snapshotting behaviour.

The following hooks are available:

 * pre_mounts(self, volumes):
 * post_mounts(self, volumes):
 * pre_mount(self, vol):
 * post_mount(self, vol):
 * pre_unmounts(self, volumes):
 * post_unmounts(self, volumes):
 * pre_unmount(self, vol):
 * post_unmount(self, vol):
 * pre_snapshots(self, volumes):
 * post_snapshots(self, volumes):
 * pre_snapshot(self, vol):
 * post_snapshot(self, vol):

examples
--------

Basic volume customization

```python
class MySnapshotter(Snapshotter):
    name = 'simple_example'

    def get_volumes(self):
        volumes = [EBSVolume('/dev/sdf1', '/mnt/index', size=200)]
        return volumes
```

Customizing filter tags
```python
class CustomFilterSnapshotter(Snapshotter):
    name = 'filter_example'

    def get_filter_tags(self):
        '''
        The tags which are used for finding the correct snapshot to load from.
        In addition to these tags, mount point is also always added.

        Use these to unique identify different parts of your infrastructure
        '''
        tags = {
            'group': self.userdata['group'],
        }
        return tags
```

Reading volumes from the userdata
```python
class UserdataSnapshotter(Snapshotter):
    '''
    Looks for a list of volumes in the instance's userdata
    [{
       "device": "/dev/sdf1",
       "mount_point": "/var/lib/postgresql/9.1/main",
       "size": 200
    }]
    '''
    name = 'userdata_example'

    def get_volumes(self):
        volume_dicts = self.userdata['volumes']
        volumes = []
        for volume_dict in volume_dicts:
            volume = EBSVolume(**volume_dict)
            volumes.append(volume)
        return volumes
```

Using the hooks


```python
class PostgreSQLSnapshotter(Snapshotter):
    '''
    Customized mounting hooks for postgres
    '''
    name = 'postgres_example'

    def pre_mounts(self):
        import subprocess
        subprocess.check_output(['/etc/init.d/postgresql', 'stop'])

    def post_mounts(self):
        import subprocess
        # fix permissions on postgresql dirs
        subprocess.check_output(['chmod', '-R', '0700', '/var/lib/postgresql'])
        subprocess.check_output(
            ['chown', '-R', 'postgres:postgres', '/var/lib/postgresql'])

```




Features
--------

* Graceful failure handling
* Freezes for the absolute minimal required duration
* Batches boto API calls for faster batch volume mounting/snapshotting
* Tested codebase

Porting old systems
-------------------

You will often need to fake userdata for porting old systems.
Doing so is quite easy:

sudo snaptastic make-snapshots solr --userdata='{"role": "solr", "cluster": "solr", "environment": "aws"}'


Cronjobs and logging
--------------------

Changing the loglevel is really easy. Simply call any of the commands with the --loglevel option.
Internally this uses the python logging module. So valid options are:
DEBUG,INFO,WARNING,ERROR,CRITICAL
For cronjobs I recommend settings it to WARNING

**changing the loglevel**
```python
snaptastic mount-snapshots solr --loglevel=WARNING
```

If you need more detailed control over logging you can change the LOGGING_CONFIG
dictionary in snaptastic_settings.py used by the python logging module.

The volume object
-----------------

**basic 200GB volume**
```python
EBSVolume('/dev/sdf1', '/var/lib/postgresql/9.1/main', 200),
```

**now with IOPS**
```python
EBSVolume('/dev/sdf1', '/var/lib/postgresql/9.1/main', 200, iops=1600),
```

**now for EXT4**
```python
EBSVolume('/dev/sdf1', '/var/lib/postgresql/9.1/main', 200, file_system=FILESYSTEMS.EXT4),
```


Todo
----

* Integrated cleanup scripts
* Auto terminate volumes after instance termination
* Auto detect region for cross region usage

Contributing & Project Workflow
-------------------------------

Contributions are more than welcome. Have a look at the Todo to get an idea of what's still missing. Please always add unittests (where possible) for your feature/bug.

 * fab validate (checks pep8 and unittests)
 * fab publish (if tests are ok, publishes the new version, tag, pypi)
 * fab clean (attempt to auto cleanup pep8 mistakes)

Running tests
-------------

* python -m unittest snaptastic.tests


Django Jobs
-----------
Do you also see the beauty in clean code? Are you experienced with high scalability web apps?
Currently we're looking for additional talent over at our Amsterdam office.
Feel free to drop me a line at my personal email for more information: thierryschellenbach[at]gmail.com
 
File Type Py Version Uploaded on Size
snaptastic-0.2.12.zip (md5) Source 2013-08-26 38KB
  • Downloads (All Versions):
  • 38 downloads in the last day
  • 315 downloads in the last week
  • 1886 downloads in the last month
  • Author: Thierry Schellenbach
  • Home Page: http://github.com/tschellenbach/Snaptastic
  • License:
    Copyright (c) Thierry Schellenbach
    
     (http://www.mellowmorning.com)
    All rights reserved.
    
    Redistribution and use of this software in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
    - Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
    - Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
    - Neither the name of Thierry Schellenbach. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission of Thierry Schellenbach.
    
    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
  • Categories
  • Package Index Owner: tschellenbach
  • DOAP record: snaptastic-0.2.12.xml