anycast-healthchecker 0.7.3

Introduction

anycast-healthchecker monitors a service by doing periodic health checks and based on the result instructs Bird daemon to either advertise or withdraw the route to reach the monitored service. As a result Bird will only advertise routes for healthy services. Routes for IPv4 and IPv6 addresses are supported.

Bird must be configured in a certain way to interface properly with anycast-healthchecker. The configuration is detailed later in this document.

anycast-healthchecker is a Python program, which uses the daemon library to implement a well-behaved Unix daemon process and threading to run multiple service checks in parallel.

What is Anycast

Anycast is a network address scheme where traffic from a sender has more than one potential receivers, but only one of them receives it. Routing protocols, decide which one of the potential receivers will actually receive traffic based on the topology of the network. The main attribute which contributes to the decision is the cost of the network path between a sender and a receiver.

Cost is a protocol specific (usually integer) value that only has meaning within a protocol that is used as a metric of distance. Routing protocols provide default values for common topologies (BGP associates the cost of a path with the number of autonomous systems between the sender and the receiver, OSPF calculates the default cost based on the bandwidth of links), but its main use is to allow administrative control over traffic flow by specifying cost according to business needs.

The nearest receiver to a sender always receives the traffic and this only changes if something changes on the network, another receiver with best path to the sender appears or current receiver disappears. If multiple receivers have the same distance from the sender more than one of the receivers might receive traffic, based on further details of the network configuration.

The three drawings below exhibit how traffic is routed between a sender and multiple potential receivers when something changes on network. In this example BGP routing protocol is used:

These potential receivers use BGP or OSPF by running an Internet Routing daemon, Bird or Quagga, to simultaneously announce the same destination IP address from different places on the network. Due to the nature of Anycast receivers can be located on any network across a global network infrastructure.

Anycast doesn’t balance traffic as only one receiver attracts traffic from senders. For instance, if there are two receivers which announce the same destination IP address in certain location, traffic will be distributed across those two receivers unevenly as senders can be spread across the network in an uneven way.

Anycast is being used as a mechanism to switch traffic between and within data-centers for the following main reasons:

• the switch of traffic occurs without the need to enforce a change to clients

In case of loss of a service in one location, traffic to that location will be switched to another data-center without any manual intervention and most importantly without pushing a change to clients which you don’t always control.

• the switch happens within few milliseconds

The same technology can be used for balancing traffic using Equal-Cost Multi-Path.

ECMP routing is a network technology where traffic can be routed over multiple paths. In the context of routing protocols, path is the route a packet has to take in order to be delivered to a destination. Because these multiple paths have the same cost, traffic is balanced across them.

This provides the possibility to perform load-balancing of traffic across multiple servers. Routers are the devices which perform load-balancing of traffic and most of them use a deterministic way to select the server based on the following four properties of IP packets:

• source IP

• source PORT

• destination IP

• destination PORT

Each unique combination of values for those four properties is called network flow. For each different network flow a different destination server is selected so traffic is evenly balanced across all servers. These servers run an Internet Routing daemon in the same way as with Anycast case but with the major difference that all servers receive traffic.

The main characteristic of this type of load-balancing is that is stateless. Router balances traffic to a destination IP address based on the quadruple network flow without the need to understand and inspect protocols above Layer 3. As a result it is very cheap in terms of resources and very fast at the same time. This is commonly advertised as traffic balancing at wire-speed.

anycast-healthchecker can be utilized in Anycast and ECMP environments.

How anycast-healthchecker works

The current release of anycast-healthchecker supports only the Bird daemon, which you have to configure in a specific way. Thus, it is mandatory to explain very briefly how Bird handles advertisements for routes.

Bird maintains a routing information base (RIB) and various protocols import/export routes to/from it. The diagram below illustrates how Bird advertises routes for IPs assigned to the loopback interface to the rest of the network using BGP protocol. Bird can also import routes learned via BGP/OSPF protocols, but this part of the routing process is irrelevant to the functionality of anycast-healthchecker.

A route is always associated with a service which runs locally on the box. The Anycasted service is a daemon (HAProxy, Nginx, Bind etc) which processes incoming traffic and listens to an IP (Anycast Service Address) for which a route exists in the RIB and advertised by Bird.

As it is exhibited in the above diagram a route is advertised only when:

1. The IP is assigned to the loopback interface.

2. direct protocol from Bird imports a route for that IP in the RIB.

3. BGP/OSPF protocols export that route from the RIB to a network peer.

The route associated with the Anycasted service must be either advertised or withdrawn based on the health of the service, otherwise traffic will always be routed to the local node regardless of the status of the service.

Bird provides filtering capabilities with the help of a simple programming language. A filter can be used to either accept or reject routes before they are exported from the RIB to the network.

A list of IP prefixes (<IP>/<prefix length>) is stored in a text file. IP prefixes that are not included in the list are filtered-out and they do not get exported from the RIB to the network. The white-list text file is sourced by Bird upon startup, reload and reconfiguration. The following diagram illustrates how this technique works:

This configuration logic allows a separate process to update the list by adding or removing IP prefixes and trigger a reconfiguration of Bird in order to advertise or withdraw routes. anycast-healthchecker is that separate process. It monitors Anycasted services and based on the status of the health checks updates the list of IP prefixes.

Bird does not allow the definition of a list with no elements and when that happens Bird will emit an error and refuses to start. Because of this anycast-healthchecker makes sure that there is always an IP prefix in the list, see dummy_ip_prefix and dummy_ip6_prefix settings in Daemon section.

Configuring anycast-healthchecker

Because anycast-healthchecker is very much tied in with Bird daemon, the configuration of Bird is explained first. Next, the configuration of anycast-healthchecker (including the configuration for the health checks) is covered and finally the options for invoking the program from the command line will be described.

include "/etc/bird.d/*.conf";
protocol device {
    scan time 10;
}
protocol direct direct1 {
    interface "lo";
        export none;
        import all;
}
template bgp bgp_peers {
    import none;
    export where match_route();
    local as 64815;
}
protocol bgp BGP1 from bgp_peers {
    disabled no;
    neighbor 10.248.7.254 as 64814;
}

function match_route()
{
    return net ~ ACAST_PS_ADVERTISE;
}

function match_route6()
{
    return net ~ ACAST6_PS_ADVERTISE;
}

define ACAST_PS_ADVERTISE =
    [
        10.189.200.255/32
    ];

define ACAST6_PS_ADVERTISE =
    [
        2001:db8::1/128
    ];

% ls -ls /etc/bird.d/anycast-prefixes.conf
4 lrwxrwxrwx 1 root root 105 Dec  2 16:08 /etc/bird.d/anycast-prefixes.conf ->
/var/lib/anycast-healthchecker/anycast-prefixes.conf

% ls -ls /etc/bird.d/6/anycast-prefixes.conf
4 lrwxrwxrwx 1 root root 107 Jan 10 10:33 /etc/bird.d/6/anycast-prefixes.conf
-> /var/lib/anycast-healthchecker/6/anycast-prefixes.conf

[DEFAULT]
interface            = lo

[daemon]
pidfile              = /var/run/anycast-healthchecker/anycast-healthchecker.pid
ipv4                 = true
ipv6                 = false
bird_conf            = /var/lib/anycast-healthchecker/anycast-prefixes.conf
bird6_conf           = /var/lib/anycast-healthchecker/6/anycast-prefixes.conf
bird_variable        = ACAST_PS_ADVERTISE
bird6_variable       = ACAST6_PS_ADVERTISE
bird_reconfigure_cmd = sudo /usr/sbin/birdc configure
bird6_reconfigure_cmd = sudo /usr/sbin/birdc6 configure
dummy_ip_prefix      = 10.189.200.255/32
dummy_ip6_prefix     = 2001:db8::1/128
bird_keep_changes    = false
bird6_keep_changes   = false
bird_changes_counter = 128
bird6_changes_counter = 128
purge_ip_prefixes    = false
loglevel             = debug
log_maxbytes         = 104857600
log_backups          = 8
log_file             = /var/log/anycast-healthchecker/anycast-healthchecker.log
stderr_file          = /var/log/anycast-healthchecker/stderr.log
stdout_file          = /var/log/anycast-healthchecker/stdout.log

[foo.bar.com]
check_cmd         = /usr/bin/curl --fail --silent http://10.52.12.1/
check_interval    = 10
check_timeout     = 2
check_fail        = 2
check_rise        = 2
check_disabled    = false
on_disabled       = withdraw
ip_prefix         = 10.52.12.1/32

[foo6.bar.com]
check_cmd         = /usr/bin/curl --fail 'http://[fd12:aba6:57db:ffff::1]:8888'
check_timeout     = 5
check_rise        = 2
check_fail        = 2
check_disabled    = false
on_disabled       = withdraw
ip_prefix         = fd12:aba6:57db:ffff::1/128
ip_check_disabled = false

anycast-healthchecker --help
A simple healthchecker for Anycasted services.

Usage:
    anycast-healthchecker [ -f <file> -c -p -P ] [ -d <directory> | -F <file> ]

Options:
    -f, --file=<file>          read settings for the daemon from <file>
                               [default: /etc/anycast-healthchecker.conf]
    -d, --dir=<dir>            read settings for service checks from files
                               under <dir> directory
                               [default: /etc/anycast-healthchecker.d]
    -F, --service-file=<file>  read <file> for settings of a single service
                               check
    -c, --check                perform a sanity check on configuration
    -p, --print                show default settings for daemon and service
                               checks
    -P, --print-conf           show running configuration with default settings
                               applied
    -v, --version              show version
    -h, --help                 show this screen

anycast-healthchecker -f ./anycast-healthchecker.conf -d ./anycast-healthchecker.d

Installation

Use pip:

pip install anycast-healthchecker

From Source:

sudo python setup.py install

Build (source) RPMs:

python setup.py clean --all; python setup.py bdist_rpm

Build a source archive for manual installation:

python setup.py sdist

Release

1. Bump version in anycast_healthchecker/__init__.py

2. Commit above change with:

   git commit -av -m'RELEASE 0.1.3 version'

3. Create a signed tag, pbr will use this for the version number:

   git tag -s 0.1.3 -m 'bump release'

4. Create the source distribution archive (the archive will be placed in the dist directory):

   python setup.py sdist

5. pbr will update ChangeLog file and we want to squeeze them to the previous commit thus we run:

   git commit -av --amend

6. Move current tag to the last commit:

   git tag -fs 0.1.3 -m 'bump release'

7. Push changes:

   git push;git push --tags

Development

I would love to hear what other people think about anycast_healthchecker and provide feedback. Please post your comments, bug reports and wishes on my issues page.

Licensing

Apache 2.0

Acknowledgement

This program was originally developed for Booking.com. With approval from Booking.com, the code was generalised and published as Open Source on github, for which the author would like to express his gratitude.

Contacts

Project website: https://github.com/unixsurfer/anycast_healthchecker

Author: Pavlos Parissis <pavlos.parissis@gmail.com>