This text was originally published in French in 2020 on the Revolve Blog by Jérémie RODON. Translation to English by Nicole Hage Chahine.
In my previous article, we explored the main components and concepts of AWS Transit Gateway. In this article, I will present my method for easily configuring Transit Gateway to create complex networks. However, before diving into that, there is one final concept to explore that I didn’t cover in the last article to avoid overwhelming the readers. Let’s talk about black holes for a moment.
The Blackhole
Those who have experience with VPC route tables may have come across “blackhole” routes in their route tables. This occurs when the destination of a route ceases to exist (for example, a stopped or terminated NAT instance), and the API marks the route as a “blackhole,” indicating that it leads to nowhere.
Figure 1: Route to a Stopped NAT Instance
In VPC route tables, this phenomenon occurs in abnormal circumstances (i.e., when a target is unreachable), but in Transit Gateway Route Tables (TGWRT), it is an explicit option: the ability to create a “blackhole” route. One might wonder why anyone would intentionally create a route that leads nowhere.
To understand this, let’s consider an example configuration where the Transit Gateway is used to centralize internet access. In this setup, all VPCs should route their traffic through a dedicated VPC that serves as the egress point to the internet. However, we also want to isolate the VPCs from each other, preventing direct communication between them. To achieve this, we can implement the following configuration:
Figure 2: Configuration with Centralized Internet Egress
The “Apps” VPCs are associated with a TGWRT that has a default route pointing to the “Outbound” VPC. The “Outbound” VPC, in turn, is associated with a TGWRT where all the attachments of the “Apps” VPCs are propagated (this serves as a small revision of the vocabulary explained in the previous article). Additionally, each “Apps” VPC has a default route to the Transit Gateway. As a result, one of the objectives is achieved: all the VPCs can access the internet through the “Outbound” VPC.
Figure 3: Round-trip between App 2 and the Internet, the routes used are highlighted.
But are the “Apps” VPCs truly isolated? One might be tempted to answer yes, since the TGWRT they are associated with does not contain any routes to the other “Apps” VPCs, only to “Outbound.” However, let’s see what would happen if one of the VPCs tried to communicate with the other:
Figure 4: Traffic flow between App 1 and App 2, highlighted routes indicate the paths taken for communication. The return traffic follows the same principle.
And yes! The traffic actually ends up “bouncing” through the “Outbound” VPC, and communication becomes possible. And this is where explicit blackholes come into play:
Figure 5: This time the traffic is “matched” by the blackhole route and gets discarded.
Now all our objectives are achieved: the VPCs all exit to the internet through a centralized “Outbound” VPC, and they cannot communicate with each other.
In summary, blackholes serve to ensure that VPCs (and attachments in general) cannot communicate.
PS: Note that there may not always be possible “bounces” and therefore no need for blackholes. However, since it can potentially be difficult to ensure that bounces are impossible, while blackholes are easy to implement and free… why not take advantage of them?
A relatively complex use case
Let’s now dive into the heart of our subject and consider the following situation:
Figure 6: A relatively complex scenario
We want the following requirements to be met:
- All VPCs (Apps & Shared Services) should communicate: With the Internet through the VPC Outbound. With the Corp network through the VPN.
- All “Apps” VPCs should communicate with the Shared Service VPC, regardless of their environment.
- VPCs in the “Dev” environment should not communicate with VPCs in the “Prod” environment.
- VPCs of different applications should not communicate with each other.
- The VPN and VPC Outbound should not communicate.
The question is: How should we configure the Transit Gateway? How many Transit Gateway Route Tables do we need? What routes should be defined?
It’s not easy to determine at first glance, right? Luckily we have Bubbles !
A Tale of Bubbles
I call them “bubbles” because the first time I encountered these types of problems, I drew diagrams on a whiteboard and circled groups of VPCs while explaining, and it reminded me of bubbles. We narrowly avoided me writing an article about potatoes instead…
Photo 1: The birth of bubbles on a whiteboard.
Since then, I have come out of my cave several times, and it seems that the whole world calls them “routing domains”… It’s less funny. So I’ll stick with my bubbles.
Anyway, the important thing is not the name, but the properties. There are only two! A bubble is defined by:
- the list of “attachments” (VPC, VPN, …) it contains;
- the list of bubbles with which communication is allowed (including itself, if applicable).
We will see that the second property is the most important.
Let’s see how to define our bubbles.
Step 1 – Fill the bubbles.
To begin, we need to fill the bubbles, which means grouping together certain attachments.
In theory, we should group attachments that need to be routed in the same way and follow the same communication rules. After all, we should not lose sight of the main property of our bubbles: “who communicates with whom.”
In practice, it may not be easy to group them correctly at first glance. Therefore, we can start by making logical groupings based on environment or VPC type, for example. Often, this approach works well, especially considering that Steps 1 and 2 can be iterative. So, there’s no real problem if mistakes are made initially. Finally, it’s better to start with too many bubbles rather than too few (as it’s easier to correct quickly).
In our case, we will define the following bubbles:
Figure 7: Bubbling the different VPCs
A “Shared Services” bubble containing the Shared Service VPC alone because it seems to be a separate case, a “Networks” bubble with the Outbound VPC and the VPN, and then a bubble for each environment: “Apps Dev” and “Apps Prod.”
Let’s move on to the important step.
Step 2 – Define communications between bubbles.
Now that we have our bubbles, we need to determine for each one the list of bubbles it can communicate with. For this step, we can follow the instructions point by point!
All VPCs (Apps & Shared Services) should communicate with the Internet through the Outbound VPC and with the Corp network through the VPN.
Alright, let’s note that on our diagram.
All “Apps” VPCs should communicate with the Shared Service VPC, regardless of their environment.
The VPCs in the “Dev” environment should not communicate with those in the “Prod” environment.
The VPCs of different applications should not communicate with each other.
Please note that this instruction implies that the “Apps Dev” and “Apps Prod” bubbles should not be able to communicate with themselves.
The VPN and the Outbound VPC should not communicate with each other.
Similarly, there should be no communication within the “Networks” bubble itself.
Regarding the “Shared Services” bubble, since there is only one VPC within it, the question of communication with itself doesn’t make much sense. However, if we had multiple VPCs within the Shared Service bubble, it would be logical for them to be able to communicate with each other. Therefore, we can consider the following:
If multiple VPCs exist within the “Shared Services” bubble, they can communicate with each other.
Here, you can see that everything is going quite smoothly: each statement is easy to reflect in our diagram. However, it’s possible that one day you may encounter a situation where a contradiction arises. Don’t panic! It simply means that you haven’t created enough bubbles. In such cases, you can simply split the bubble or bubbles where the contradiction occurs.
Keep in mind that there is always a solution (unless the statement itself contradicts itself, such as “A and B communicate” and “B and A do not communicate”). In the most extreme case, each bubble would contain only one VPC.
There will also be cases where there are more bubbles than necessary, which can be easily realized by presenting the information in a tabular form:
It’s easy to see here that the “Apps Dev” and “Apps Prod” lines are identical, which means they can be merged into a single bubble. This is precisely how we can identify any unnecessary bubbles.
However, in our case, I will leave things as they are. Having extra bubbles is not a problem, and I like the idea of having separate bubbles for the development and production environments as it makes more sense. Additionally, future network developments may differentiate these bubbles, so it makes sense!
Now that all our bubbles are well defined, we know what each one contains and with which bubbles it communicates. What should we do with this information?
Step 3 – Translate our bubbles into Transit Gateway configuration
The beauty of this approach is that we can systematically translate our bubbles into Transit Gateway configuration using four simple rules:
- Each bubble corresponds to one (and only one) dedicated Transit Gateway Route Table (TGWRT).
- Saying a VPC (or VPN) is within a bubble means that the corresponding attachment is associated with the TGWRT of that bubble.
- If bubble “A” communicates with bubble “B,” it means that the attachments associated with TGWRT “A” are propagated into TGWRT “B” (and vice versa).
- If bubble “A” does not communicate with bubble “B,” it means that the attachments associated with TGWRT “A” (specifically, the CIDR blocks of those attachments) are blackholes in TGWRT “B” (and vice versa).
In our case, we need to create four TGWRTs with the corresponding associations, propagations, and blackholes. Taking into account the addition of CIDR blocks in our situation, the configuration will be as follows:
Figure 7: Transit Gateway configuration for our relatively complex scenario
You can verify that it perfectly matches the instructions!
Indeed, it results in a lot of routes… In practice, it often generates more routes than necessary. However, that’s not a problem as Transit Gateway supports 10,000 routes across 20 TGWRTs, and these are soft limits, so there’s no need to worry about that. What matters is that the method itself is simple!
However, you might say, “But Jérémie, doing this manually would be a nightmare!” And I would agree with you. That’s why, in the next article, we will explore how this method is not only simple but also highly automatable!