Redundant HAProxy – Active/Backup
This mini how-to shall cover HAProxy with high-availability configuration using keepalived.
Keepalived is a software that provides load-balancing and high-availability feature.
Load balancing feature based on linux kernel based module IPVS (IP Virtual Server) that provides Layer-4 level load-balancing.
High-Availability is achieved by VRRP protocol (Virtual Router Redundancy Protocol), describes a method of implementing a redundant IP interface shared between two or more routers on a common LAN segment, allowing a group of routers to function as one virtual router. When this IP interface is specified as a default gateway on hosts directly attached to this LAN, the routers sharing the IP interface prevent a single point of failure by limiting access to this gateway address. VRRP can be implemented on IES service interfaces and on core network IP interfaces.
High-Availability is achieved by VRRP protocol (Virtual Router Redundancy Protocol), and in common routing world, it is a fundamental element for router fail-over. It provides a method to assign automatic IP to router in pair acting as master and backup.
With above little overview, let’s continue over configuration part.
Note: You are configuring this HA – HAProxy, that includes VRRP, in your existing network topology, your TOR may it-self has configured as active/standby, so you need to be careful while choosing "virtual_router_id" parameter.
Basic HAPROXY Config:
global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon defaults mode http log global option tcplog option dontlognull option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 listen web_api 0.0.0.0:443 mode tcp server web2 karblrnvsd02.live.domain.in:443 maxconn 1024 check check-ssl inter 30s fall 2 verify none server web1 karblrnvsd01.live.domain.in:443 maxconn 1024 backup check check-ssl inter 30s fall 2 verify none server web3 karblrnvsd03.live.domain.in:443 maxconn 1024 backup check check-ssl inter 30s fall 2 verify none option log-health-checks option httpchk GET /nuage/health http-check expect status 200
Keepalived config:
vrrp_script chk_haproxy {
script "killall -0 haproxy" ### Install psmisc, you do not have this command
interval 2 # every 2 seconds
weight 2 # add 2 points if OK
}
vrrp_instance VI_1 {
interface eth0
state MASTER ### use BACKUP on stand-by Server
virtual_router_id 199 ### This is you should be alert for, this may be using on your TOR, so first check with Network Team.
priority 50
virtual_ipaddress {
172.21.166.248
}
track_script {
chk_haproxy
}
}
Firewalld config:
firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface eth0 --destination 224.0.0.18 --protocol vrrp -j ACCEPT firewall-cmd --reload
Start your Haproxy and keepalived services and check Master node virtual IP
Please patiently go through the logs:
################################# HA PROXY-1 ################################# Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: Registering Kernel netlink reflector Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: Registering Kernel netlink command channel Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: Registering gratuitous ARP shared channel Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: Opening file ‘/etc/keepalived/keepalived.conf’. Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: VRRP_Instance(VI_1) removing protocol VIPs. Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: WARNING – script `killall` resolved by path search to `/usr/bin/killall`. Please specify full path. Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: SECURITY VIOLATION – scripts are being executed but script_security not enabled. Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: Using LinkWatch kernel netlink reflector… Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Oct 12 13:52:03 karblrnhap01 Keepalived_vrrp[14072]: VRRP_Script(chk_haproxy) succeeded Oct 12 13:52:04 karblrnhap01 Keepalived_vrrp[14072]: VRRP_Instance(VI_1) Transition to MASTER STATE Oct 12 13:52:04 karblrnhap01 Keepalived_vrrp[14072]: VRRP_Instance(VI_1) Changing effective priority from 101 to 103 Oct 12 13:52:05 karblrnhap01 Keepalived_vrrp[14072]: VRRP_Instance(VI_1) Entering MASTER STATE Oct 12 13:52:05 karblrnhap01 Keepalived_vrrp[14072]: VRRP_Instance(VI_1) setting protocol VIPs. Oct 12 13:52:05 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:05 karblrnhap01 Keepalived_vrrp[14072]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 172.21.166.248 Oct 12 13:52:05 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:05 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:05 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:05 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01 Keepalived_vrrp[14072]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01 Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 [root@karblrnhap01 log]# ip a eth0: <broadcast,multicast,up,lower_up>mtu 1500 qdisc pfifo_fast state UP qlen 1000</broadcast,multicast,up,lower_up> link/ether 52:54:00:a5:3f:d7 brd ff:ff:ff:ff:ff:ff inet 172.21.166.250/28 brd 10.166.39.159 scope global eth0 valid_lft forever preferred_lft forever inet 172.21.166.248/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fea5:3fd7/64 scope link valid_lft forever preferred_lft forever [root@karblrnhap01 log]# systemctl status keepalived keepalived.service – LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2018-10-12 13:52:03 IST; 25min ago Process: 14069 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 14070 (keepalived) CGroup: /system.slice/keepalived.service ├─14070 /usr/sbin/keepalived -D ├─14071 /usr/sbin/keepalived -D └─14072 /usr/sbin/keepalived -D Oct 12 13:52:05 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:05 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:05 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:05 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on….148 Oct 12 13:52:10 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:52:10 karblrnhap01.live.domain.in Keepalived_vrrp[14072]: Sending gratuitous ARP on eth0 for 172.21.166.248 Hint: Some lines were ellipsized, use -l to show in full.
[root@karblrnhap01 log]#
########################################### HA-PROXY-2 ########################################### Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: Registering Kernel netlink reflector Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: Registering Kernel netlink command channel Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: Registering gratuitous ARP shared channel Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: Opening file ‘/etc/keepalived/keepalived.conf’. Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) removing protocol VIPs. Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: WARNING – script `killall` resolved by path search to `/usr/bin/killall`. Please specify full path. Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: SECURITY VIOLATION – scripts are being executed but script_security not enabled. Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: Using LinkWatch kernel netlink reflector… Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Entering BACKUP STATE Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)] Oct 12 08:21:50 localhost Keepalived_vrrp[31016]: VRRP_Script(chk_haproxy) succeeded Oct 12 08:21:51 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Changing effective priority from 100 to 102 Oct 12 08:21:53 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Transition to MASTER STATE Oct 12 08:21:54 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Entering MASTER STATE Oct 12 08:21:54 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) setting protocol VIPs. Oct 12 08:21:54 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:54 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 172.21.166.248 Oct 12 08:21:54 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:54 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:54 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:54 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:59 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:59 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 172.21.166.248 Oct 12 08:21:59 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:59 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:59 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:21:59 localhost Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 08:41:27 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Received advert with higher priority 103, ours 102 Oct 12 08:41:27 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Entering BACKUP STATE Oct 12 08:41:27 localhost Keepalived_vrrp[31016]: VRRP_Instance(VI_1) removing protocol VIPs. [root@karblrnhap02 ~]# ip a eth0: <broadcast,multicast,up,lower_up> mtu 1500 qdisc pfifo_fast state UP qlen 1000</broadcast,multicast,up,lower_up> link/ether 52:54:00:a3:3f:8b brd ff:ff:ff:ff:ff:ff inet 172.21.166.249/28 brd 10.166.39.159 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fea3:3f8b/64 scope link valid_lft forever preferred_lft forever [root@karblrnhap02 ~]# systemctl status keepalived keepalived.service – LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2018-10-12 13:51:50 IST; 27min ago Process: 31013 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 31014 (keepalived) CGroup: /system.slice/keepalived.service ├─31014 /usr/sbin/keepalived -D ├─31015 /usr/sbin/keepalived -D └─31016 /usr/sbin/keepalived -D Oct 12 13:51:54 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:51:59 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:51:59 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on….148 Oct 12 13:51:59 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:51:59 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:51:59 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 13:51:59 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: Sending gratuitous ARP on eth0 for 172.21.166.248 Oct 12 14:11:27 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Received advert with higher priorit… 102 Oct 12 14:11:27 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: VRRP_Instance(VI_1) Entering BACKUP STATE Oct 12 14:11:27 karblrnhap02.live.domain.in Keepalived_vrrp[31016]: VRRP_Instance(VI_1) removing protocol VIPs.
Hint: Some lines were ellipsized, use -l to show in full.