Skip to main content

Turn your Raspberry Pi into an independent backup module for your network.

Project description

Very Hungry Pi
==============

News
----

Version 2 (beta) released.
~~~~~~~~~~~~~~~~~~~~~~~~~~

I'm happy to announce Version 2 (beta) of *vhpi*. It's an entire
rewrite. There is a ``vhpi`` package now on
`pypi <https://pypi.python.org/pypi/vhpi>`__ and a simple command line
interface to run *vhpi* more conveniently. There are some minor breaking
Changes in the config. The most important thing to notice, if you
upgrade from v1 to v2, is that the snapshot directories have a new
naming convention. ``monthly.1`` would now be
``2017-10-11__02:07:03__monthly.1``. The timestamp tells you when the
backup was finished. If you want to use your current snapshots with v2,
you should adjust their names accordingly. See
`CHANGELOG.md <https://github.com/feluxe/very_hungry_pi/blob/2.0/CHANGELOG.md>`__
for more changes.

Contents
--------

- `Description <#description>`__
- `Features <#features>`__
- `Requirements <#requirements>`__
- `Example Config <#example_config>`__
- `Installation & Configuration <#install>`__

Description
------------

With **vhpi** you can turn your Raspberry Pi into a silent backup module
for your Network. *Vhpi* creates
`incremental <https://en.wikipedia.org/wiki/Incremental_backup>`__
`snapshot <https://github.com/feluxe/very_hungry_pi/wiki/Snapshots-explanation>`__
backups of available network shares (e.g.
`NFS <https://en.wikipedia.org/wiki/Network_File_System>`__,
`Samba <https://en.wikipedia.org/wiki/Samba_(software)>`__) silently and
automated with a minimum of disk space required. *Vhpi* runs entirely on
'server-side'; clients only need to share/export backup sources with the
Pi and let the Pi run the backups in the background. *Vhpi* uses battle
proven tools like `rsync <https://en.wikipedia.org/wiki/Rsync>`__ to
create the backups and `cp <https://en.wikipedia.org/wiki/Cp_(Unix)>`__
to create hardlinks for the snapshots. To get the most control over the
backups *vhpi* takes raw `rsync
options <http://linux.die.net/man/1/rsync>`__ for configuration. *Vhpi*
writes two log files: one for a short overview of the entire process
(`info.log exmpl. <vhpi/examples/info.log>`__) and one for debugging
(`debug.log exmpl. <vhpi/examples/debug.log>`__).

**TL;DR:** Just setup *vhpi*, run your Pi 24/7 and don't care about
backups no more.

Features
---------

- *Vhpi* works with any rsync command you like. This gives you a wide
and well documented variety of configuration options for your backup.
- You can create multiple *exclude-lists* to exclude files/dirs from
the backup. (See 'exclude\_lib' in `Example
Config <#example_config>`__)
- *Vhpi* creates
`snapshots <https://github.com/feluxe/very_hungry_pi/wiki/Snapshots-explanation>`__
for any time-interval you like. (e.g. 'hourly', 'daily', 'weekly',
'monthly', 'each-4-hours', 'half-yearly', etc...) Just add the
interval name and its duration in seconds to the config. (See
'intervals' in `Example Config <#example_config>`__).
- You can set the amount of snapshots that you want keep for each used
interval. E.g. if you want to keep 3 snapshots for the 'hourly'
interval you get three snapshot dirs: ``hourly.0``, ``hourly.1``,
``hourly.2``. Each snapshot reaches an hour further into the past.
- Snapshots require a minimum of disk space:

- because the backups are created incrementally.
- because *vhpi* creates new snapshots as 'hard links' for all files
that haven't changed. (No duplicate files.. just links)

- The process is nicely logged ('info.log', 'debug.log').
- If a backup process takes long, *vhpi* blocks any attempt to start a
new backup process until the first one has finished to prevent the Pi
from overloading.
- More features are planned (See: `Version
Overview <https://github.com/feluxe/very_hungry_pi/wiki/Version-Overview-(TODOs)>`__)

Requirements:
--------------

- You need Python >= 3.6 on your Pi for *vhpi* to run. (`How to install
Python3.x on your
Pi <https://github.com/feluxe/very_hungry_pi/wiki/Install-Python3.X-from-source-on-a-Raspberry-Pi-(Raspbian)>`__)
- The file system of your Backup destination has to support hard links.
(most common fs like NTFS and ext do...)

Installation & Configuration
-----------------------------

Sharing sources with the Pi:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your Pi needs access to the directories of each client that you want to
backup. Just share/export them with ``NFS`` or ``Samba``. Perhaps *vhpi*
can create local backups as well.

You should use ``autofs`` or similar to automatically mount the shared
directories with your Pi whenever they are available. This way your Pi
will automatically mount the directories whenever a machine enters the
network.

There is a tutorial on this in the wiki: `How to share sources with your
Raspberry Pi using
NFS. <https://github.com/feluxe/very_hungry_pi/wiki/How-to-share-sources-with-your-Raspberry-Pi-using-NFS.>`__

Download and Install:
~~~~~~~~~~~~~~~~~~~~~

Simplest way to install *vhpi* is by useing pip. You need Python3.6 for
*vhpi* to run. (`How to install Python3.x on your
Pi <https://github.com/feluxe/very_hungry_pi/wiki/Install-Python3.X-from-source-on-a-Raspberry-Pi-(Raspbian)>`__)
After you installed Python3.6 you can run pip to install *vhpi* like
this:

::

$ pip3.6 install vhpi

Run this command to check if *vhpi* was isntalled successfully:

::

$ vhpi --help

It should print the help text to the terminal.

Configure vhpi:
~~~~~~~~~~~~~~~

When you run *vhpi* for the first time, it creates a config dir at
``~/.config/vhpi/``, you'll find a file called ``vhpi_cfg.yaml`` there.
This is where you configure your backups. The config file is pretty self
explanatory, just have a look at the `Example
Config <#example_config>`__

Test the configuration
~~~~~~~~~~~~~~~~~~~~~~

In order to test *vhpi* I suggest setting up some dummy backup sources
that point to some safe destinations. Maybe in the ``/tmp`` dir or so.
Then run the following command a couple of times and see if the
destination gets filled with backups/snapshots:

``$ vhpi run``

If you get an error try to adjust the config. If you think there is a
bug feel free to use the `github issue
tracker <https://github.com/feluxe/very_hungry_pi/issues>`__! The
results of each run is written to the log-files as well
(``~/.config/vhpi/debug.log`` and ``~/.config/vhpi/info.log``)

Create a Cronjob
~~~~~~~~~~~~~~~~~

You can run *vhpi* manually with this command ``vhpi run``, but I
suggest creating a cronjob that runs *vhpi* automatically every hour. To
do so you can add the following line to ``/etc/crontab``. (Replace
``username`` with the username that is supposed to run *vhpi*.)

::

@hourly username vhpi run

If want to set the cronjob to something else than *hourly*, you might
want to read this: `Choosing an interval for the
cornjob. <https://github.com/feluxe/very_hungry_pi/wiki/Choosing-an-interval-for-the-cronjob.>`__

You can also add multiple cronjobs that execute *vhpi* in different
intervals for different users.

After you added the cronjob, you should restart your Pi or restart the
crontab like this:

::

$ /etc/init.d/cron restart

If this is all done, your Pi should run *vhpi* every hour and you should
see some activity in the log files and of cause on your hard drive. Yay!
:D

Example Config
---------------

``~/.config/vhpi/vhpi_cfg.yaml``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

| \`\`\`yaml
| # IMPORTANT: If you use paths that contain spaces, make sure to escape
them # with  (backslash). The same counts for escape items.

Basic App Settings:
===================

app\_cfg: # Create different list of files/dirs that you want to exclude
from your # backups. exclude\_lib: standard\_list: [ lost+found/\ *,
.cache/chromium/*, .mozilla/firefox/\ */Cache, .cache/thumbnails/*,
.local/share/Trash/\* ] another\_list: [ some\_dir ] # Define time
intervals, which you may use for your snapshots. # Feel free to use your
own definitions like 'every\_four\_hours: 14400' etc. # Values must be
in Seconds. intervals: { hourly: 3600, six-hourly: 21600, daily: 86400,
weekly: 604800, monthly: 2592000, yearly: 31536000 }

Backup Jobs Config.
===================

Configure each backup source here:
==================================

jobs: # Source 1: - name: 'Dummy Source' source\_ip: '192.168.178.20' #
The ip of the computer to which the mounted src dir belongs to. If it's
a local source use: "127.0.0.1" or "localhost". rsync\_src:
'/tmp/tests/dummy\_src/src1/' # The path to the mounted or local dir.
rsync\_dst: '/tmp/tests/dummy\_dest/dest1/' # The path to the
destination dir in which each snapshot is created. rsync\_options:
'-aAHSvX --delete' # The options that you want to use for your rsync
backup. Default is "-av". More info on rsync:
http://linux.die.net/man/1/rsync exclude\_lists: [ # Add exclude lists
to exclude a list of file/folders. See above: app\_cfg -> exclude\_lib
standard\_list, another\_list ] excludes: [ # Add additional source
specific exclude files/dirs that are not covered by the exclude lists.
downloads, tmp ] snapshots: # Define how many snapshots you want to keep
for each interval. Older snapshots are deleted automatically. hourly: 6
six-hourly: 4 daily: 7 weekly: 4 monthly: 6 yearly: 6

# Source 2: # - name: 'Another Dummy Source' # source\_ip:
192.168.178.36 # etc...' \`\`\`

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

vhpi-2.0.0b12-py3-none-any.whl (29.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page