Cross Zone Load Balancing always on, right?

Cross Zone Load Balancing is one of the less known and most confusing options of the different load balancers on AWS. Until 2013 the choice was simple, Amazon offered only one load balancer as a service – the Classic Load Balancer – and there was no option to perform Cross Zone Load Balancing. No feature, no doubts, no extra costs.

In 2019, with three different load balancers (Classic Load Balancer, Application Load Balancer and Network Load Balancer), there is much more flexibility but as well a higher risk of an incorrect configuration or unexpected charges.

Photo by Daniele Levis Pelusi

What is Cross Zone Load Balancing?

As for AWS documentation:

“With cross-zone load balancing, (…) each load balancer node distributes requests evenly across the registered instances in all enabled Availability Zones. If cross-zone load balancing is disabled, each load balancer node distributes requests evenly across the registered instances in its Availability Zone only.”

What is the default for the Cross Zone Load Balancing?

Unfortunately the default is different on every load balancer and not very intuitive. As for AWS documentation:

Classic Load Balancer: with the API or CLI, cross-zone load balancing is disabled by default. With the AWS Management Console, the option to enable cross-zone load balancing is selected by default.

Application Load Balancer: cross-zone load balancing is always enabled

Network Load Balancer: cross-zone load balancing is disabled by default. You can enable or disable cross-zone load balancing at any time.

Should I always enable it?

There are many documents and posts on the benefits of enabling cross-zone load balancing. And if you have only one target in every Availability Zone, it is usually an easy choice . But what are the main reasons to disable it or keep it disabled?

Maybe you want to minimise the latency between your load balancer and the application nodes and have all the traffic in the subnet. Or you take advantage of the SSL termination on the load balancer and you do not want to manage not encrypted traffic across data centres and different subnets. Or maybe you want simply to save a few dollars.

Do I pay extra for Cross Zone Load Balancing?

You do not pay for the the feature itself but you might pay for the generated regional data transfer. The voice that in your billing ends up under

$0.010 per GB - regional data transfer - in/out/between EC2 AZs or using elastic IPs or ELB

and that can end in significant charges if you manage large binaries on your load balancers. According to the AWS FAQ, the cost varies according to the specific service.

Q: Am I charged for regional AWS data-transfer for cross-zone load balancing in Application Load Balancer?
A: No. Since cross-zone load balancing is always on with Application Load Balancer, you are not charged for this type of regional data transfer.


Q: Am I charged for regional AWS data-transfer when I enable cross-zone load balancing in Network Load Balancer?
A: Yes, you will be charged for regional data transfer between Availability Zones with Network Load Balancer when cross-zone load balancing is enabled

Q: Am I charged for regional AWS data-transfer when I enable cross-zone load balancing in Classic Load Balancer?
A: No, you are not charged for regional data transfer between Availability Zones when you enable cross-zone load balancing for your Classic Load Balancer.

To summarize…

Cross Zone Load Balancing is a very useful feature and you likely end up enabling it in many common scenarios. But it is vital to understand the default values and the implications according to the specific AWS service you choose.

Using WAF with ELB: CloudFront and request timeouts

To minimize maintenance and increase device support for a web application I would like to have a AWS load balancer (either Classic Load Balancer or Application Load Balancer) supporting IPv6 and taking advantage of WAF as web application firewall. In short, I would like to leverage AWS services for the entire stack, firewall included.

Let’s see how to achieve that starting with the constraints we have on AWS:

  • ELB does not currently support IPv6 in VPC.  Something I discuss with more details in a separate post.
  • WAF currently supports only CloudFront and not directly the load balancer
  • As for last week AWS news, CloudFront and WAF now support IPv6

Given all the above, the most obvious approach to have a load balancer with a managed application firewall is to set up a CloudFront distribution in front of the existing ELB and serve all the requests using CloudFront, having the existing ELB as origin for dynamic content. This setup allows us to use WAF as application firewall and have IPv6 support for the service. And as far as I know it’s the only way to use WAF with ELB.

Any limitation?

All very good and the service works but there is a critical limitation for long HTTP requests. As for AWS documentation, ELB has a configurable request timeout that can be configured outside of CloudFront (for example, you can raise to 60 or 120 or 180 seconds to manage long processing requests according to your application specs) while CloudFront has a hard setting of 30 seconds that cannot be modified:

The request timeout for CloudFront depends on the HTTP method:

GET and HEAD requests – If the origin doesn’t respond within 30 seconds or stops responding for 30 seconds, CloudFront drops the connection and makes two additional attempts to contact the origin. If the origin doesn’t reply during the third attempt, CloudFront doesn’t try again until it receives another request for content on the same origin.

DELETE, OPTIONS, PATCH, POST, and POST requests – If the origin doesn’t respond within 30 seconds, CloudFront drops the connection and doesn’t try again to contact the origin. The client can resubmit the request if necessary.

The request timeout cannot be changed.

So any request that is taking longer than 30 seconds (and is still processed by the load balancer where the timeout is higher)  will timeout and generate a 504 error code to the end user.

<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: ********
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

To summarize, the only way to use WAF in front of a ELB endpoint introduces a fixed 30 seconds request timeout. According to your specific requirements, this might be or not an acceptable compromise for a production deployment.

Classic Load Balancer and Application Load Balancer support for IPv6

Going back a bit in time, in 2011 AWS announced support for  “Elastic Load Balancing – IPv6, Zone Apex Support, Additional Security”. As Jeff Barr wrote at that time: “All Elastic Load Balancers in the US East (Northern Virginia) and EU (Ireland) regions now have publicly routable IPv6 addresses in addition to their existing IPv4 addresses.” This was well over 5 years ago, when the ELB was only available in EC2 classic. And since then we lost support of IPv6 at Classic Load Balancer and there is apparently no support either in the new Application Load Balancer. And reading again the blog post about “some panic-inducing articles” and “it will soon become necessary” is quite funny.

Looking now at the official AWS documentation for Public DNS Names for Your Load Balancer the current status is:

EC2-VPC
Load balancers in a VPC support IPv4 addresses only.
EC2-Classic
Load balancers in EC2-Classic support both IPv4 and IPv6 addresses.

So effectively the support has been dropped given that more recent AWS accounts do not even have the option to run instances in Ec2-Classic, so the idea to rollback to classic or use a E2-Classic link are both not options.

Trying to create a new application load balancer I see that the dualstack CNAME is resolved

dig dualstack.test-1665400693.eu-west-1.elb.amazonaws.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.47.rc1.el6 <<>> dualstack.test-1665400693.eu-west-1.elb.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30456
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;dualstack.test-1665400693.eu-west-1.elb.amazonaws.com. IN A

;; ANSWER SECTION:
dualstack.test-1665400693.eu-west-1.elb.amazonaws.com. 23 IN A 54.229.122.86

Not having IPv6 support has already side effects. For example, given that iOS now requires supporting IPv6 during the Apple Submission Process there is no simple way to test it purely on AWS. I am aware of possible workarounds but we are  saying that I need to use a local laptop to perform a test of a mobile app I might have entirely developed on the cloud. And that’s far from ideal. Looking forward to proper IPv6 from AWS.