Keepalived Check and Notify Scripts

Keepalived is a Linux implementation of the VRRP (Virtual Router Redundancy Protocol) protocol to make IPs highly available - a so called VIP (Virtual IP).

The daemon is furthermore able to provide load balancing mechanisms using the "Linux Virtual Server" (IPVS). In this blog post I'll write about an almost undocumented feature: check scripts and notify scripts. These scripts can be used to regularly check anything you want to ensure the VRRP master is on the correct node and take action if there is a state change.

How VRRP works

Usually the VRRP protocol ensures that one of participating nodes is master. The backup node(s) listens for multicast packets from a node with a higher priority. If the backup node fails to receive VRRP advertisements for a period longer than three times of the advertisement timer, the backup node takes the master state and assigns the configured IP(s) to itself. In case there are more than one backup nodes with the same priority, the one with the highest IP wins the election.

There is no fencing mechanism available. If f.e. two participating nodes don't see each other, both will have the master state and both will carry the same IP(s). When I was looking for a way to detect which one should stay the master or give up his master state, I discovered the "Check Script" mechanism.

Check Script

A check script is a script written in the language of your choice which is executed regularly. This script needs to have a return value: 0 for "everything is fine", 1 (or other than 0) for "something went wrong".
This value is used by Keepalived to take action. Scripts are defined like this:

vrrp_script chk_myscript {
  script       "/usr/local/bin/mycheckscript.sh"
  interval 2   # check every 2 seconds
  fall 2       # require 2 failures for KO
  rise 2       # require 2 successes for OK
}

As you can see in the example it's possible to specify the interval in seconds and also how many times the script needs to succeed or fail until any action is taken.

The script can check anything you want. Here are some ideas:

• Is the daemon X running?
• Is the interface X on the remote switch Y up?
• Is the IP 8.8.8.8 pingable?
• Is there enough disk space available to run my application?
• $MYIDEA

This script definition can now be used in a vrrp_instance:

vrrp_instance MyVRRPInstance {
  state MASTER
  interface eth0
  virtual_router_id 5
  priority 200
  advert_int 1
  virtual_ipaddress {
    192.168.1.1/32 dev eth0
  }
  track_script {
    chk_myscript
  }
}

As soon as the track_script returns another code than 0 two times, the VRRP instance will change the state to FAULT, removes the IP 192.168.1.1 from eth0 and stops sending multicast VRRP packets.

Notify Script

A notify script can be used to take other actions, not only removing or adding an IP to an interface.
It can f.e. start or stop a daemon, depending on the VRRP state. And this is how it's defined in the Keepalived configuration:

vrrp_instance MyVRRPInstance {
 [...]
 notify /usr/local/bin/keepalivednotify.sh
}

The script is called after any state change with the following parameters:

• $1 = "GROUP" or "INSTANCE"
• $2 = name of group or instance
• $3 = target state of transition ("MASTER", "BACKUP", "FAULT")

Here is a sample script:

#!/bin/bash

TYPE=$1
NAME=$2
STATE=$3

case $STATE in
        "MASTER") /etc/init.d/apache2 start
                  exit 0
                  ;;
        "BACKUP") /etc/init.d/apache2 stop
                  exit 0
                  ;;
        "FAULT")  /etc/init.d/apache2 stop
                  exit 0
                  ;;
        *)        echo "unknown state"
                  exit 1
                  ;;
esac

One example of using these notify scripts is to have a highly available IPsec gateway (start and stop the IPsec process). We are using it successfully at nine.ch for customer IPSec endpoints.

• 1: http://www.keepalived.org/ "Keepalived website"
• 2: https://en.wikipedia.org/wiki/Virtual_Router_Redundancy_Protocol