Deployment models for AWS Network Firewall, and part 2 covering the latest VPC routing options, explains in details how AWS Network Firewall can be deployed in different routing configurations. While these are typically one time tasks, I thought it would make sense to create a Cloudformation template for this because the setup has many components that are easy to misconfigure but difficult to debug.

Everything went smoothly up to the point where I would have to create a route from subnet to Network Firewall endpoint within the same AZ. This is important as you want to avoid cross-AZ traffic for high-availability and lower latency. It also used to be a cost factor but not any longer as inter-AZ traffic is free.

Problem

Here is how to get firewall endpoints

Fn::GetAtt Firewall.EndpointIds

The unique IDs of the firewall endpoints for all of the subnets that you attached to the firewall. The subnets are not listed in any particular order. For example: ["us-west-2c:vpce-111122223333", "us-west-2a:vpce-987654321098", "us-west-2b:vpce-012345678901"]

Ok, this sounds like a challenge, or lazy design, not providing a way to get the endpoint for particular AZ. Note that simply ordering the list wouldn’t be a good solution as you may not have deployed the firewall for every AZ.

First attempt

It should be possible to extact the endpoint for given AZ from the list using Fn::Join, Fn::Split and Fn::Select -functions. Let’s get to work and find the endpoint for AZ-a.

First I must join the list into a string. I’m inserting “:” between list elements.

Fn::Join: [ ':', ["us-west-2c:vpce-111122223333", "us-west-2a:vpce-987654321098", "us-west-2b:vpce-012345678901" ] ]
-> "us-west-2c:vpce-111122223333:us-west-2a:vpce-987654321098:us-west-2b:vpce-012345678901"

As the format of list elements is, AZ:endpoint, I know the endpoint for AZ-a would start after “a:”, so let’s split the string at “a:” to find the begining of the endpoint. Looking at the sample from documentation, IDs are all numeric and therefore the can be only one “a:” in this string we are splitting.

Fn::Split: [ 'a:', "us-west-2c:vpce-111122223333:us-west-2a:vpce-987654321098:us-west-2b:vpce-012345678901" ]
-> [ "us-west-2c:vpce-111122223333:us-west-2", "vpce-987654321098:us-west-2b:vpce-012345678901" ]

Now I know the endpoint ID would be the 2nd element of the list (index starts from 0).

Fn::Select: [ 1, [ "us-west-2c:vpce-111122223333:us-west-2", "vpce-987654321098:us-west-2b:vpce-012345678901" ] ]
-> "vpce-987654321098:us-west-2b:vpce-012345678901"

Endpoint ID ends at “:” because that was the character inserted between list elements.

Fn::Split: [ ':', "vpce-987654321098:us-west-2b:vpce-012345678901" ]
-> [ "vpce-987654321098", "us-west-2b:vpce-012345678901" ]

And now the endpoint ID is the first element of the list (index start from 0).

Fn::Select [ 0, [ "vpce-987654321098", "us-west-2b:vpce-012345678901" ] ]
-> "vpce-987654321098"

Done! All looks good and I managed to deploy correct routes when testing the template. But after couple of iterations I started to see strange errors every now and then saying the endpoint didn’t exists !?! My first reaction was that maybe there is some issue with syncing the resource creation and one endpoint wasn’t yet ready when route was added.

Solution

Closer look to error revealed that there was nothing wrong with the endpoints being available, but the endpoint ID I was using was missing an “a” -character at the end of it. But the sample from documentation had only numeric IDs?

It turns out IDs are actually hexadecimals, and the error occurs when there is ‘a’, ‘b’ or ‘c’ at the end ID. The last letter of an ID is cut off at this point. I did modify AZ-a endpoint to end with “a” to illustrate the problem.

Fn::Split: [ 'a:', "us-west-2c:vpce-111122223333:us-west-2a:vpce-98765432109a:us-west-2b:vpce-012345678901" ]
-> [ "us-west-2c:vpce-111122223333:us-west-2", "vpce-98765432109", "us-west-2b:vpce-012345678901" ]

I don’t mind getting a list of 3 instead of 2, but problem is split string “a:” being deleted from the original string. In this case deleting the last “a” of the endpoint ID.

Once the bug was found, the fix was to add “,” instead of “:” between the list element when joining the original list of endpoints. Then it would be sure the above split would cut the string only where it should.

Final version with all functions nested looks this

  Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref InspectionTGWRouteTableA
      DestinationCidrBlock: '0.0.0.0/0'
      VpcEndpointId: !Select [ 0, !Split [ ',', !Select [ 1 , !Split [ 'a:', !Join [ ',', !GetAtt NetworkFW.EndpointIds ] ] ] ] ]

Huh, I wouldn’t call it trivial, but possible ;-)

If you would like to have GetAtt simply return the endpoint ID for given AZ, please add your comment, or just +1, to GitHub issue.