Using WAF with ELB: CloudFront and request timeouts

To minimize maintenance and increase device support for a web application I would like to have a AWS load balancer (either Classic Load Balancer or Application Load Balancer) supporting IPv6 and taking advantage of WAF as web application firewall. In short, I would like to leverage AWS services for the entire stack, firewall included.

Let’s see how to achieve that starting with the constraints we have on AWS:

  • ELB does not currently support IPv6 in VPC.  Something I discuss with more details in a separate post.
  • WAF currently supports only CloudFront and not directly the load balancer
  • As for last week AWS news, CloudFront and WAF now support IPv6

Given all the above, the most obvious approach to have a load balancer with a managed application firewall is to set up a CloudFront distribution in front of the existing ELB and serve all the requests using CloudFront, having the existing ELB as origin for dynamic content. This setup allows us to use WAF as application firewall and have IPv6 support for the service. And as far as I know it’s the only way to use WAF with ELB.

Any limitation?

All very good and the service works but there is a critical limitation for long HTTP requests. As for AWS documentation, ELB has a configurable request timeout that can be configured outside of CloudFront (for example, you can raise to 60 or 120 or 180 seconds to manage long processing requests according to your application specs) while CloudFront has a hard setting of 30 seconds that cannot be modified:

The request timeout for CloudFront depends on the HTTP method:

GET and HEAD requests – If the origin doesn’t respond within 30 seconds or stops responding for 30 seconds, CloudFront drops the connection and makes two additional attempts to contact the origin. If the origin doesn’t reply during the third attempt, CloudFront doesn’t try again until it receives another request for content on the same origin.

DELETE, OPTIONS, PATCH, POST, and POST requests – If the origin doesn’t respond within 30 seconds, CloudFront drops the connection and doesn’t try again to contact the origin. The client can resubmit the request if necessary.

The request timeout cannot be changed.

So any request that is taking longer than 30 seconds (and is still processed by the load balancer where the timeout is higher)  will timeout and generate a 504 error code to the end user.

<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: ********
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

To summarize, the only way to use WAF in front of a ELB endpoint introduces a fixed 30 seconds request timeout. According to your specific requirements, this might be or not an acceptable compromise for a production deployment.

Classic Load Balancer and Application Load Balancer support for IPv6

Going back a bit in time, in 2011 AWS announced support for  “Elastic Load Balancing – IPv6, Zone Apex Support, Additional Security”. As Jeff Barr wrote at that time: “All Elastic Load Balancers in the US East (Northern Virginia) and EU (Ireland) regions now have publicly routable IPv6 addresses in addition to their existing IPv4 addresses.” This was well over 5 years ago, when the ELB was only available in EC2 classic. And since then we lost support of IPv6 at Classic Load Balancer and there is apparently no support either in the new Application Load Balancer. And reading again the blog post about “some panic-inducing articles” and “it will soon become necessary” is quite funny.

Looking now at the official AWS documentation for Public DNS Names for Your Load Balancer the current status is:

EC2-VPC
Load balancers in a VPC support IPv4 addresses only.
EC2-Classic
Load balancers in EC2-Classic support both IPv4 and IPv6 addresses.

So effectively the support has been dropped given that more recent AWS accounts do not even have the option to run instances in Ec2-Classic, so the idea to rollback to classic or use a E2-Classic link are both not options.

Trying to create a new application load balancer I see that the dualstack CNAME is resolved

dig dualstack.test-1665400693.eu-west-1.elb.amazonaws.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.47.rc1.el6 <<>> dualstack.test-1665400693.eu-west-1.elb.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30456
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;dualstack.test-1665400693.eu-west-1.elb.amazonaws.com. IN A

;; ANSWER SECTION:
dualstack.test-1665400693.eu-west-1.elb.amazonaws.com. 23 IN A 54.229.122.86

Not having IPv6 support has already side effects. For example, given that iOS now requires supporting IPv6 during the Apple Submission Process there is no simple way to test it purely on AWS. I am aware of possible workarounds but we are  saying that I need to use a local laptop to perform a test of a mobile app I might have entirely developed on the cloud. And that’s far from ideal. Looking forward to proper IPv6 from AWS.