buttervolume 3.10

Introduction

BTRFS is a next-generation copy-on-write filesystem with subvolume and snapshot support. A BTRFS subvolume can be seen as an independant file namespace that can live in a directory and can be mounted as a filesystem and snapshotted individually.

On the other hand, Docker volumes are commonly used to store persistent data of stateful containers, such as a MySQL/PostgreSQL database or an upload directory of a CMS. By default, Docker volumes are just local directories in the host filesystem. A number of Volume plugins already exist for various storage backends, including distributed filesystems, but small clusters often can’t afford to deploy a distributed filesystem.

We believe BTRFS subvolumes are a powerful and lightweight storage solution for Docker volumes, allowing fast and easy replication (and backup) across several nodes of a small cluster.

Prerequisites

Make sure the directory /var/lib/buttervolume/ is living in a BTRFS filesystem. It can be a BTRFS mountpoint or a BTRFS subvolume or both.

You should also create the directories for the config and ssh on the host:

sudo mkdir /var/lib/buttervolume
sudo mkdir /var/lib/buttervolume/config
sudo mkdir /var/lib/buttervolume/ssh

Build and run as a contributor

If you want to be a contributor, read this chapter. Otherwise jump to the next section.

You first need to create a root filesystem for the plugin, using the provided Dockerfile:

git clone https://github.com/anybox/buttervolume
./build.sh

By default the plugin is built for the latest commit (HEAD). You can build another version by specifying it like this:

./build.sh 3.7

At this point, you can set the SSH_PORT option for the plugin by running:

docker plugin set anybox/buttervolume SSH_PORT=1122

Note that this option is only relevant if you use the replication feature between two nodes.

Now you can enable the plugin, which should start buttervolume in the plugin container:

docker plugin enable anybox/buttervolume:HEAD

You can check it is responding by running a buttervolume command:

export RUNCROOT=/run/docker/runtime-runc/plugins.moby/ # or /run/docker/plugins/runtime-root/plugins.moby/
alias drunc="sudo runc --root $RUNCROOT"
alias buttervolume="drunc exec -t $(drunc list|tail -n+2|awk '{print $1}') buttervolume"
sudo buttervolume scheduled

Increase the log level by writing a /var/lib/buttervolume/config/config.ini file with:

[DEFAULT]
TIMER = 120

Then check the logs with:

sudo journalctl -f -u docker.service

You can also locally install and run the plugin in the foreground with:

python3 -m venv venv
./venv/bin/python setup.py develop
sudo ./venv/bin/buttervolume run

Then you can use the buttervolume CLI that was installed in developer mode in the venv:

./venv/bin/buttervolume --version

Install and run as a user

If the plugin is already pushed to the image repository, you can install it with:

docker plugin install anybox/buttervolume

Check it is running:

docker plugin ls

Find your runc root, then define useful aliases:

export RUNCROOT=/run/docker/runtime-runc/plugins.moby/ # or /run/docker/plugins/runtime-root/plugins.moby/
alias drunc="sudo runc --root $RUNCROOT"
alias buttervolume="drunc exec -t $(drunc list|tail -n+2|awk '{print $1}') buttervolume"

And try a buttervolume command:

buttervolume scheduled

Or create a volume with the driver. Note that the name of the driver is the name of the plugin:

docker volume create -d anybox/buttervolume:latest myvolume

Note that instead of using aliases, you can also define functions that you can put in your .bash_profile or .bash_aliases:

function drunc () {
  RUNCROOT=/run/docker/runtime-runc/plugins.moby/ # or /run/docker/plugins/runtime-root/plugins.moby/
  sudo runc --root $RUNCROOT $@
}
function buttervolume () {
  drunc exec -t $(docker plugin ls --no-trunc  | grep 'anybox/buttervolume:latest' |  awk '{print $1}') buttervolume $@
}

Upgrade

You must force disable it before reinstalling it (as explained in the docker documentation):

docker plugin disable -f anybox/buttervolume
docker plugin rm -f anybox/buttervolume
docker plugin install anybox/buttervolume

Configure

You can configure the following variables:

• DRIVERNAME: the full name of the driver (with the tag)

• VOLUMES_PATH: the path where the BTRFS volumes are located

• SNAPSHOTS_PATH: the path where the BTRFS snapshots are located

• TEST_REMOTE_PATH: the path during unit tests where the remote BTRFS snapshots are located

• SCHEDULE: the path of the scheduler configuration

• RUNPATH: the path of the docker run directory (/run/docker)

• SOCKET: the path of the unix socket where buttervolume listens

• TIMER: the number of seconds between two runs of the scheduler

• DTFORMAT: the format of the datetime in the logs

• LOGLEVEL: the Python log level (INFO, DEBUG, etc.)

The configuration can be done in this order of priority:

1. from an environment variable prefixed with BUTTERVOLUME_ (ex: BUTTERVOLUME_TIMER=120)

2. from the [DEFAULT] section of the /etc/buttervolume/config.ini file inside the container or /var/lib/buttervolume/config/config.ini on the host

Example of config.ini file:

[DEFAULT]
TIMER = 120

If none of this is configured, the following default values are used:

• DRIVERNAME = anybox/buttervolume:latest

• VOLUMES_PATH = /var/lib/buttervolume/volumes/

• SNAPSHOTS_PATH = /var/lib/buttervolume/snapshots/

• TEST_REMOTE_PATH = /var/lib/buttervolume/received/

• SCHEDULE = /etc/buttervolume/schedule.csv

• RUNPATH = /run/docker

• SOCKET = $RUNPATH/plugins/btrfs.sock # only if run manually

• TIMER = 60

• DTFORMAT = %Y-%m-%dT%H:%M:%S.%f

• LOGLEVEL = INFO

Usage

docker volume create -d anybox/buttervolume:latest myvolume

docker volume create --volume-driver=anybox/buttervolume:latest

docker volume create -d anybox/buttervolume -o copyonwrite=false myvolume

sudo buttervolume run

docker volume create -d btrfs myvolume

docker create --volume-driver=btrfs

run                 Run the plugin in foreground
snapshot            Snapshot a volume
snapshots           List snapshots
schedule            (un)Schedule a snapshot, replication or purge
scheduled           List scheduled actions
restore             Restore a snapshot (optionally to a different volume)
clone               Clone a volume as new volume
send                Send a snapshot to another host
sync                Synchronise a volume from a remote host volume
rm                  Delete a snapshot
purge               Purge old snapshot using a purge pattern

buttervolume snapshot <volume>

buttervolume snapshots

buttervolume snapshots <volume>

buttervolume restore <snapshot>

buttervolume restore <snapshot> <volume>

buttervolume clone <volume> <new_volume>

buttervolume rm <snapshot>

buttervolume send <host> <snapshot>

buttervolume sync <volume> <host1> [<host2>][...]

buttervolume purge <pattern> <volume>

buttervolume purge --dryrun <pattern> <volume>

buttervolume schedule snapshot 60 <volume>

buttervolume schedule snapshot 0 <volume>

buttervolume schedule replicate:remote_host 3600 foovolume

buttervolume schedule replicate:remote_host 0 foovolume

buttervolume schedule purge:4h:1w:1y 60 foovolume

buttervolume schedule purge:4h:1w:1y 0 foovolume

buttervolume schedule snapshot 1440 <volume>
buttervolume schedule purge:1d:4w:1y 1440 <volume>

buttervolume schedule synchronize:remote_host1,remote_host2 60 foovolume

buttervolume schedule synchronize:remote_host1,remote_host2 0 foovolume

buttervolume scheduled

Testing

If your volumes directory is a BTRFS partition or volume, tests can be run with:

./test.sh

Working without a BTRFS partition

If you have no BTRFS partitions or volumes you can setup a virtual partition in a file as follows (tested on Debian 8):

Setup BTRFS virtual partition:

sudo qemu-img create /var/lib/docker/btrfs.img 10G
sudo mkfs.btrfs /var/lib/docker/btrfs.img

Mount the partition somewhere temporarily to create 3 new BTRFS subvolumes:

sudo -s
mkdir /tmp/btrfs_mount_point
mount -o loop /var/lib/docker/btrfs.img /tmp/btrfs_mount_point/
btrfs subvolume create /tmp/btrfs_mount_point/snapshots
btrfs subvolume create /tmp/btrfs_mount_point/volumes
btrfs subvolume create /tmp/btrfs_mount_point/received
umount /tmp/btrfs_mount_point/
rm -r /tmp/btrfs_mount_point/

Stop docker, create required mount point and restart docker:

systemctl stop docker
mkdir -p /var/lib/buttervolume/volumes
mkdir -p /var/lib/buttervolume/snapshots
mkdir -p /var/lib/buttervolume/received
mount -o loop,subvol=volumes /var/lib/docker/btrfs.img /var/lib/buttervolume/volumes
mount -o loop,subvol=snapshots /var/lib/docker/btrfs.img /var/lib/buttervolume/snapshots
mount -o loop,subvol=received /var/lib/docker/btrfs.img /var/lib/buttervolume/received
systemctl start docker

Once you are done with your test, you can unmount those volumes and you will find back your previous docker volumes:

systemctl stop docker
umount /var/lib/buttervolume/volumes
umount /var/lib/buttervolume/snapshots
umount /var/lib/buttervolume/received
systemctl start docker
rm /var/lib/docker/btrfs.img

Migrate to version 3

If you’re currently using Buttervolume 1.x or 2.0 in production, you must carefully follow the guidelines below to migrate to version 3.

First copy the ssh and config files and disable the scheduler:

sudo -s
docker cp buttervolume_plugin_1:/etc/buttervolume /var/lib/buttervolume/config
docker cp buttervolume_plugin_1:/root/.ssh /var/lib/buttervolume/ssh
mv /var/lib/buttervolume/config/schedule.csv /var/lib/buttervolume/config/schedule.csv.disabled

Then stop all your containers, excepted buttervolume

Now snapshot and delete all your volumes:

volumes=$(docker volume ls -f driver=btrfs --format "{{.Name}}")
# or: # volumes=$(docker volume ls -f driver=btrfs|tail -n+2|awk '{print $2}')
echo $volumes
for v in $volumes; do docker exec buttervolume_plugin_1 buttervolume snapshot $v; done
for v in $volumes; do docker volume rm $v; done

Then stop the buttervolume container, remove the old btrfs.sock file, and restart docker:

docker stop buttervolume_plugin_1
docker rm -v buttervolume_plugin_1
rm /run/docker/plugins/btrfs.sock
systemctl stop docker

If you were using Buttervolume 1.x, you must move your snapshots to the new location:

mkdir /var/lib/buttervolume/snapshots
cd /var/lib/docker/snapshots
for i in *; do btrfs subvolume snapshot -r $i /var/lib/buttervolume/snapshots/$i; done

Restore /var/lib/docker/volumes as the original folder:

cd /var/lib/docker
mkdir volumes.new
mv volumes/* volumes.new/
umount volumes  # if this was a mounted btrfs subvolume
mv volumes.new/* volumes/
rmdir volumes.new
systemctl start docker

Change your volume configurations (in your compose files) to use the new anybox/buttervolume:latest driver name instead of btrfs

Then start the new buttervolume 3.x as a managed plugin and check it is started:

docker plugin install anybox/buttervolume:latest
docker plugin ls

Then recreate all your volumes with the new driver and restore them from the snapshots:

for v in $volumes; do docker volume create -d anybox/buttervolume:latest $v; done
export RUNCROOT=/run/docker/runtime-runc/plugins.moby/ # or /run/docker/plugins/runtime-root/plugins.moby/
alias drunc="sudo runc --root $RUNCROOT"
alias buttervolume="drunc exec -t $(drunc list|tail -n+2|awk '{print $1}') buttervolume"
# WARNING : check the the volume you will restore are the correct ones
for v in $volumes; do buttervolume restore $v; done

Then restart your containers, check they are ok with the correct data.

Reenable the schedule:

mv /var/lib/buttervolume/config/schedule.csv.disabled /var/lib/buttervolume/config/schedule.csv

Credits

Thanks to:

• Christophe Combelles

• Pierre Verkest

• Marcelo Ochoa

• Christoph Rist

• Philip Nagler-Frank

• Yoann MOUGNIBAS

3.10 (2023-04-09)

• updated dependencies

• fixed bug #45

3.9 (2022-10-11)

• Fixed a crash when option_copyonwrite is not set

• Restored the test suite

• Improved and simplified the build script and test script

3.8 (2022-10-07)

• Switched to copy-on-write by default

• Allow to choose to enable/disable copy-on-write for each volume

• Allow to change the default SSH_PORT in the plugin config

• Updated the base docker image and dependencies

• Added an option to show the version number

• Improved documentation

3.7 (2018-12-13)

• Unpinned urllib3

3.6 (2018-12-11)

• Fixed zombie sshd processes inside the plugin

• Minor documentation change

3.5 (2018-06-07)

• Improved documentation

3.4 (2018-04-27)

• Fix rights at startup so that ssh works

3.3 (2018-04-27)

• Fixed a bug preventing a start in certain conditions

3.2 (2018-04-27)

• Fixed the socket path for startup

3.1 (2018-04-27)

• Fixed a declaration in Python 3.6

• Automatically detects the btrfs.sock path

• Made the runpath and drivername configurable

3.0 (2018-04-24)

• Now use the docker managed plugin system

• Stop the scheduler before shutdown to avoid a 5s timeout

• Improved logging

• Improved the migration doc from version 1 or 2

2.0 (2018-03-24)

• BREAKING CHANGE: Please read the migration path from version 1 to version 2:

  BTRFS volumes and snapshots are now stored by default in different directories under /var/lib/buttervolume

• Configuration possible through environment variables or a config.ini file

• implemented VolumeDriver.Capabilities and just return 'local'

• other minor fixes and improvements

1.4 (2018-02-01)

• Add clone command

• replace sync by btrfs filesystem sync

1.3.1 (2017-10-22)

• fixed packaging (missing README)

1.3 (2017-07-30)

• fixed the cli for the restore command

1.2 (2017-07-16)

• fixed the purge algorithm

1.1 (2017-07-13)

• allow to restore a snapshot to a different volume name

1.0 (2017-05-24)

• initial release, used in production