A Generalized Reference Architecture For DNS-based Global Load-Balancing

Introduction

I recently found myself needing to establish deployment architecture guidelines for DNS-based global load-balancing services here at Blue Cross NC , and I wanted to do so in the context of a cogent reference architecture. I found an abundance of vendor-specific documentation, but wanted a platform-agnostic framework for us to work from, so I put this together.

My hope is that this reference architecture will provide us with a useful perspective as we dig into just what role dGLB will play in our future-state hybrid/multi-cloud environment as well as providing us with a explicit and direct reference to the specific patterns we seek to implement.

Overview

DNS-based global load-balancing is a service that directs inbound connections to a nominal application or service to instances of the actual application/service, or (more often) to application load-balancers ("ALBs") "in front" of clusters of the application/service. GLB is distinguished from ALB in that it deals effectively with distributing traffic across multiple physical locations, rather than just across application-instances within a single location. It is further distinguished by the fact that a DNS-based GLB service is only "in path" for the DNS resolution process. It is not in path for the actual application traffic between client and server, as an ALB is.

In this architecture, a DNS-based global-load-balancing (dGLB) service provides a mechanism for distributing incoming connections across a given application/service to two or more different locations where clusters of the application/service are being front-ended by application load balancers (ALBs.)  The GLB devices implement three core functions:

• Responding to DNS queries for the FQDNs of globally load-balanced services
• Performing health-checks of the downstream load-balancing targets for each GLB-hosted FQDN
• Executing a configured policy to determine which  downstream load-balancing target's information to include in DNS responses for the load-balanced FQDNs

By executing these functions, the GLB devices are able to distribute "connections" to a given FQDN across multiple backend targets.  dGLB is only in-path for the DNS phase of client-to-server connectivity, which makes for extremely low capacity requirements on the GLB devices, but significantly limits to granularity and accuracy with which load-balancing can be implemented.

Conceptual Diagram

Elements of the Architecture

The suffix "(a-n)" is used to indicate that multiple instances of this element may be instantiated, with a parenthesized alphabetic character index to differentiate between instances. For example, an element definition with a label of "cl(a-n)" could may be instantiated multiple times. In such a case, individual instances would be named: "cl(a)", "cl(b)", cl(c)", etc..

"cl(a-n)"

These are the client devices accessing the globally load-balanced services. There may be 0, 1, or many (a-n)instances of this element present in any locale.

"rr(a-n)"

Recursive DNS resolvers serving the client devices.  There may be 1 or many (a-n) instances of this element.  It is assumed that access to the DNS resolvers by the client devices is mediated by either IP anycast on the transport network or round-robin rotation on the part of the client devices (each configured with the IP addresses of multiple "rr" instances.)

"ans1(a-n)"

The authoritative nameservers for the enterprise-domain (e.g. "example.com").  There 1 or many instances (a-n) of this element.   It is assumed that access to the authoritative nameservers by recursive resolvers is mediated by either IP anycast at the network layer, or the use of multiple NS records in whichever external nameservers have delegated "ed1" to the enterprise. 

"glb(a-n)"

Instances of DNS-based global load-balancers.  There must be a minimum of two instances ("glb(a)", and glb(b)") in a minimum of two "locales", but there may be more instances ("glb(c-n)").

"alb(a-n)"

Instances of application load-balancers.  There must be a minimum of two instances in separate locales, but there may be more (c-n) instances.

"app(a-n)"

The application that the global load-balancer is serving.  There may be zero, one, or many load-balanced applications.  (Though there is not much use for a global load-balancer if there are no applications to load-balance.)

"app(a-n).ai(a-n)"

Instances of the application/service that is being load-balanced by the GLB and ALB services.  There must be a minimum of one instance at a minimum two locales for each load-balanced application, but there may be more instances (a-n).

"ed1"

A DNS domain which has been delegated to the enterprise authoritative nameservers.  "example.com" is used as an example throughout this architecture specification.

"glbd"

A DNS sub-domain of the enterprise domain (ed1) which has been delegated by the enterprise authoritative nameservers (ans1) to the global load-balancers (glb).  "glb.example.com" is used as an example throughout this architecture specification.  There must be a minimum of one instance of this element, but additional instances (b-n) are permitted.

"rrset1"

A set of NS records maintained (below the zone apex) in the enterprise domain (ed1) zonefile on the enterprise authoritative nameservers ("ans1") for the "glbd1" domain.   The "name" value of these records is the global load-balancing domain ("glbd1") and the DATA value is the FQDN of a "glb" instance.  There is a separate record for each GLB instance. 

"rrset2"

A set of A records maintained in the enterprise domain ("ed1") zonefile on the enterprise authoritative nameservers ("ans1") with "name" values of the individual GLB instances (glbd1).  The "name" field in each of these resource-records must correspond directly to the RDATA field from one of the rrset1 records. The data portion of each record is the IP address of the corresponding instance of "glb". These are the "glue" records for the delegation of the "glbd" domain.

"rrset3(a-n)"

A CNAME record maintained in the enterprise domain ("ed1") zone on the enterprise authoritative nameservers ("ans1") for each globally load-balanced service's FQDN ("fqdn1").  The "name" portion of each resource-record is the unique relative (aka host-only) value assigned to a single globally load-balanced service.   The RDATA field is the corresponding FQDN ("fqdn2") configured on the global load-balancers ("glb1") for the same globally load-balanced service.

"fqdn1(a-n)"

The fully-qualified domain name (FQDN) of a globally load-balanced service which is used by clients of the service.  Each globally-load-balanced service has its own dedicated FQDN, which is "located" in the "ed1" domain (as "rrset3").

"fqdn2(a-n)"

The fully-qualified domain name (FQDN) of a globally load-balanced service which is hosted authoritatively by the GLB. (E.g. "svc-x.glb.example.com) in "glbd1".  Each globally-load-balanced service has its own dedicated FQDN, in the "glbd1" domain.

"fqdn3(a-n)"

A fully-qualified domain name (FQDN) that appears in the list of load-balancing targets for a GLB-hosted service FQDN.

"fqdn4(a-n)"

The fully qualified domain (FQDN) used in the "delegation" (NS) records ("rrset2") on the enterprise authoritative nameserver's "ed1" zone

"glbpool(a-n)"

The list of load-balancing targets for a GLB-hosted service FQDN ("fqdn2").  The items in this list may be either IP addresses or FQDNs.  There must be a single instance of this element for each globally-load-balanced service.

"rrset4(a-n)"

A synthetic/dynamic A record resource record set maintained on the global load-balancers ("glb") for "fqdn2". 

"rrset5(a-n)"

A synthetic/dynamic CNAME record resource record set maintained on the global load-balancers ("glb") for "fqdn2". 

"rrset6(a-n)"

An A record resource set maintained on authoritative nameservers for any FQDNs that are included in the glbpool.  (E.g. "alb1.aws.example.com")

"glbp(a-n)"

A nominal policy executed on the GLB instances ("glb(a-n)") which determines which element of "glbpool(a-n)" the GLB will include in its response to a query of "fqdn2(a-n)".  This policy will typically use some combination of the following data elements: (1) the state of "alb(a-n)" instances appearing in the "glbpool(n)" of "fqdn2(n)"; (2) the proximity of "rr(n)" and "alb(a-n)" instances; the contents of the nominal session-persistence table for "fqdn(2)" in "glb"; and (3) explicitly configured preference for a given entry on "glbpool(n)".

The policy also specifies what TTL value should be used in the DNS responses generated by the GLB.

"locale(a-n)"

An environment in which IT workload can be hosted.  May denote regionally-based availability zones.  Must be sufficiently localized for an ALB within the locale to effectively serve backend load-balancing targets in the same locale.

Network Flows

1. DNS query to resolve the advertised FQDN (eg. "svc-x.example.com) of a globally load-balanced service.  From client ("c") to its preferred DNS resolver ("rr")
2. DNS query to resolve the same FQDN as in flow 1 (eg. "svc-x.example.com) of a globally load-balanced service.  From a recursive resolver ("rr") to the enterprise authoritative name-server ("ans")
3. DNS response to query from  flow 2. From "ans" to "rr".  Response includes:  rrset3, rrset2, and rrset1.
4. DNS query from "rr" to "glb".  Querying the RDATA portion of the CNAME record  (e.g. svc-x.example.com) from flow 3 (data from rrset3) and specifying "A" record type
5. DNS response from "glb" to "rr".  Response includes either an A record or CNAME record (depending on wether the glb targets were defined as IP addresses or FQDNs.)
6. DNS query from "rr" to "ans" to resolve CNAME record returned in flow 5
7. DNS response to query from flow 6.  Response includes one or more A records
8. DNS response from rr to client (response to query from flow 1)
9. Health-checks from GLB instances to members of their  target pools
10. Health-checks from ALBs to members of their target pools
11. Application traffic from clients to ALBs
12. Application traffic from ALBs to service instances

Functional Diagram

DNS Zone Structure

The structure of the DNS zones maintained by the DNS authoritative nameservers global load-balancers is illustrated in the following figures.

DNS Resolution Process

The process of DNS resolution from client-device, to recursive-resolver, to authoritative nameserver, back to recursive-resolver, to glb, back to recursive resolver, and back to client-device is depicted in the following flowchart:

Connection Persistence Mechanism

DNS-based global load-balancing can implement a "persistence" mechanism to help ensure that a given client is consistently sent to the same ALB instance.  This persistence mechanism is implemented on the GLB devices, and causes the GLB to continue to provide the same response to a DNS-client (typically the actual client's preferred recursive resolvers) when it queries a given load-balanced FQDN as long as the persistence mechanism is engaged.   

If persistence is enabled, the GLB will maintain a nominal persistence list/table with the following fields:

• DNS-client IP address
• FQDN queried by client
• Time of last query
• Persistence Timeout
• Persistent Response

The first two fields above act as a primary key for the table. That is, there can be no more than one record with the same values in the "Client-IP address" and "Queried FQDN" fields.

The following flow-chart illustrates the persistence-related logic that a GLB executes when it receives a query for an FQDN that it is servicing:

The net effect of this arrangement is that a client/consumer of the load-balanced service will end up getting "sent" to the same site's ALB consistently/persistently for a GLB-serviced FQDN until the persistence timer has elapsed.

The desired dynamics are that as long as a user is connected to a load-balanced service, they will periodically send DNS queries for the service's FQDN, which will reset the "time last-queried" value to "now", keeping the persistence record current/valid. Then, after the client stops using the connection, they will stop querying the FQDN, and after the persistence record will time-out. (After which the next query for the same FQDN from the same DNS client resolver will get a "balanced" response.)

Impact on Load-balancing

Each GLB service configuration should include a TTL value to be used in responses to DNS queries for the FQDN hosted on the GLB.

There is an inherent design incentive to keep TTLs low, so as to allow for quicker failure detection and redirection of traffic away from "dead" sites

The persistence mechanism relies on the TTL values advertised by the GLB, treating the periodic repeated DNS queries from the same "clients" (recursive resolvers) as functional "keep-alives" for session persistence

Ergo, the TTL advertised in the DNS responses from the GLB must not be longer than the persistence-timeout value for the same FQDN. This is particularly important to keep in mind when the GLB targets are service-abstractions provided by 3rd parties (such as AWS Application Load Balancers; which AWS DNS also resolves with a 60-second TTL).

Limitations

Coarseness of Load-balancing Granularity and Impact on Session Persistence

Because DNS-based GLBs are only in path for DNS, they are only able to act on data in the DNS queries it receives when categorizing the client-side of connections that it is load-balancing.  The source-IP address of the DNS queries that it receives for the FQDN's of GLB'ed services is the primary mechanism for that type of categorization.  In this architecture, the IP addresses of the recursive resolvers (not the clients) are exposed to the GLBs.  The upshot of this is that with session persistence enabled, the GLBs are required to make the same  load-balancing decision for each recursive resolver that it receives DNS queries from.

Presentation of Downstream ALBs as IP addresses or FQDNs

At least one DNS-based global-load-balancing platform (rhymes with "Jeff Drive") is not able to perform health-checks of downstream ALBs if the GLB pool of targets consists of FQDNs (rather than IP addresses.)  This is problematic if the downstream ALBs need to be accessed by FQDN (as is the case with AWS Application Load Balancers.)  This limitation can be circumvented by installing an additional software module (which is  able to perform health-checks on behalf of the GLB) that runs co-resident with the GLB instance , but at the cost of significantly increased configuration complexity (and licensing cost).

1:1 Relationship between FQDNs and  load-balanced services

ALBs may have a 1:many FQDN to load-balanced-service relationship, whereas that relationship is exclusivley 1:1 for GLB services. 

Application load-balancers (ALBs) typically present to the consumer with one or more IP address, which may be abstracted as FQDNs.  A single client-facing FQDN can be sufficient for an ALB to act as a load-balancer for multiple different applications.  Because it is inline for the actual application data plane, the ALB can perform application-layer inspection to classify requests for different URIs to different back-end resource pool.  It can likewise instantiate TCP listeners on multiple ports using the same IP address, and map each TCP port to a different backend resource pool.   In this scenario, an ALB with a single FQDN and single virtual-IP address can independently distribute traffic for multiple backend load-balanced services.  

While a single FQDN can suffice for ALBs to independently manage load-balancing for multiple services, the same is not true of GLBs.