Using WAF with ELB: CloudFront and request timeouts

To minimize maintenance and increase device support for a web application I would like to have a AWS load balancer (either Classic Load Balancer or Application Load Balancer) supporting IPv6 and taking advantage of WAF as web application firewall. In short, I would like to leverage AWS services for the entire stack, firewall included.

Let’s see how to achieve that starting with the constraints we have on AWS:

  • ELB does not currently support IPv6 in VPC.  Something I discuss with more details in a separate post.
  • WAF currently supports only CloudFront and not directly the load balancer
  • As for last week AWS news, CloudFront and WAF now support IPv6

Given all the above, the most obvious approach to have a load balancer with a managed application firewall is to set up a CloudFront distribution in front of the existing ELB and serve all the requests using CloudFront, having the existing ELB as origin for dynamic content. This setup allows us to use WAF as application firewall and have IPv6 support for the service. And as far as I know it’s the only way to use WAF with ELB.

Any limitation?

All very good and the service works but there is a critical limitation for long HTTP requests. As for AWS documentation, ELB has a configurable request timeout that can be configured outside of CloudFront (for example, you can raise to 60 or 120 or 180 seconds to manage long processing requests according to your application specs) while CloudFront has a hard setting of 30 seconds that cannot be modified:

The request timeout for CloudFront depends on the HTTP method:

GET and HEAD requests – If the origin doesn’t respond within 30 seconds or stops responding for 30 seconds, CloudFront drops the connection and makes two additional attempts to contact the origin. If the origin doesn’t reply during the third attempt, CloudFront doesn’t try again until it receives another request for content on the same origin.

DELETE, OPTIONS, PATCH, POST, and POST requests – If the origin doesn’t respond within 30 seconds, CloudFront drops the connection and doesn’t try again to contact the origin. The client can resubmit the request if necessary.

The request timeout cannot be changed.

So any request that is taking longer than 30 seconds (and is still processed by the load balancer where the timeout is higher)  will timeout and generate a 504 error code to the end user.

<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: ********
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

To summarize, the only way to use WAF in front of a ELB endpoint introduces a fixed 30 seconds request timeout. According to your specific requirements, this might be or not an acceptable compromise for a production deployment.