In our last blog, “AWS Hybrid Architecture and Edge 2.0,” we covered the workflow of registering an Edge 2.0 gateway, attaching it to Aviatrix Transit, and forming a BGP peering with on-premise devices. Now, let’s take a closer look at the features of the Edge 2.0 gateway. By leveraging Edge 2.0, enterprises gain high throughput and intelligent packet processing capabilities at the edge of their network. Edge 2.0 provides a robust set of features, including intelligent packet routing to streamline network traffic and advanced security features, such as network segmentation, to provide an added layer of protection to your network.

Quick indexes

• Customize Spoke Advertised VPC CIDRs
• Gateway Manual BGP Advertised Network List
• Connection Manual BGP Advertised Network List
• Preserve AS Path
• Gateway AS Path Prepend
• Connection AS Path Prepend
• Transitive Routing
• Network Segmentation

Current setup

Aviatrix Transit will learn all the specific routes from TGW: 10.32.1.0/24, 1.32.3.0/24 and 10.64.254.0/23. Since the security VPC is also used for Egress inspection, the default route 0.0.0.0/0 is also been advertised. Aviatrix Transit will used the Edge 2.0 gateway to advertise these ranges towards on-premise router via overlay High Performance Encryption BGP over IPSec tunnels.

Aviatrix Transit VPC CIDR: 10.64.0.0/23 will be advertised by VGW towards on-premise router via underlay Direct Connect connection.

On-premise router have following BGP configuration:

• Peering with AWS DXGW(Direct Connect Gateway) via VIF (Direct Connect Virtual Interface) 169.254.96.17
• Peering with Edge 2.0 LAN interface 10.1.12.2
• Advertises: 10.1.13.0/24 and 10.1.88.88

show running-config | sec router
router bgp 65300
 bgp log-neighbor-changes
 network 10.1.13.0 mask 255.255.255.0
 network 10.1.88.88 mask 255.255.255.255
 neighbor 10.1.12.2 remote-as 65010
 neighbor 10.1.12.2 description Edge LAN
 neighbor 169.254.96.17 remote-as 65020
 neighbor 169.254.96.17 description VIF
 neighbor 169.254.96.17 password 7 055B1E0...

From the on-premise router, we can see four prefixes of as-path of 65010 (Edge) 65001 (Aviatrix Transit). These came from overlay. We can also see 1 prefixes of as-path of 65020, this came from underlay.

ISR-2#show ip bgp 
BGP table version is 36, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r> 0.0.0.0          10.1.12.2                0             0 65010 65001 i
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.32.1.0/24     10.1.12.2                0             0 65010 65001 i
*> 10.32.3.0/24     10.1.12.2                0             0 65010 65001 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i
*> 10.64.254.0/23   10.1.12.2                0             0 65010 65001 i

On Aviatrix Controller, Multi-Cloud Transit

Aviatrix Controller, Multi-Cloud Transit -> Transit -> Select Transit gateway -> Details/ Diag, under Route Info DB Details,

We can see that 0/0, 10.32.3.0/24, 10.64.254.0/23 and 10.32.1.0/24 came from TGW VPC attachments.

10.1.13.0/24 and 10.1.88.88/32 came from Edge peering, with as-path of 65010 (Edge), 65300 (ISR)

Aviatrix Controller, Multi-Cloud Transit -> Transit -> Select Transit gateway -> Details/ Diag, under Gateway Routing Table click on the two circling arrows. This is the Linux routing table on the Transit Gateway. Aviatrix coverts the routes learned via Aviatrix Spoke/Edge attachments, as well as from external BGP devices to Linux routing table. In the end, it doesn’t really matter what’s in the BGP, Linux will use it’s own routing table to determine where to send the traffic.

Since the setup is using Aviatrix Orchestrated TGW, it uses eth1 to communicate to TGW side of VPCs, such as 10.32.1.0/24, 10.32.3.0/24…

10.1.12.2/32 is Edge LAN interface

10.1.13.0/24 and 10.1.88.88/32 is been advertised via ISR towards Edge LAN interface.

10.1.13.0/24 and 10.1.88.88/32 also have a backup route via the HA transit gateway.

Customize Spoke Advertised VPC CIDRs

Multi-cloud transit -> List -> Spoke -> Select Edge Gateway -> Actions -> Customize Spoke Advertised VPC CIDRs

Let’s enter something here, such as 77.77.77.77/32, someone’s lucky number maybe?

In Transit Route Info DB, we can see this gets inserted. Remember that it’s a BGP like behavior.

In Transit Gateway Routing Table, we can also see this gets inserted.

In VPC that are attached to Aviatrix Orchestrated TGW routing table, we can see this entry gets inserted pointing towards TGW

If the firewall is also orchestrated by Aviatrix, this will also be programed. I’m using a GWLB enabled Firewall, which is a Firewall on a stick, where outbound traffic go back via the same inbound interface, hence it cannot have routing table management via Aviatrix. (Sorry cannot provide a screenshot of Firewall route table here!)

TGW routing table:

ISR routing table, nothing added

ISR-2#show ip bgp 
BGP table version is 36, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r> 0.0.0.0          10.1.12.2                0             0 65010 65001 i
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.32.1.0/24     10.1.12.2                0             0 65010 65001 i
*> 10.32.3.0/24     10.1.12.2                0             0 65010 65001 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i
*> 10.64.254.0/23   10.1.12.2                0             0 65010 65001 i

Summary: Customize Spoke Advertised VPC CIDRs tells Aviatrix fabric that specified prefix will be coming from this Spoke, which will be propagated throughout Aviatrix fabric. But it will not send over to it’s external BGP peers. Aviatrix is intelligence to handle Software Defined Networking, whether it’s on VPC routing table, orchestrated TGW routing table, orchestrated Firewall routing table, or Aviatrix Gateway routing table.

Gateway Manual BGP Advertised Network List

Multi-cloud transit -> Advanced Config -> Edit Spoke -> Select the Edge gateway -> Gateway Manual BGP Advertised Network List

In Transit Route Info DB, not observed

In Transit Gateway Routing Table, not observed

On ISR, we can see all previously advertised routes via Aviatrix Transit disappeared, now we only have 10.64.0.0/23 from 65020 (DXGW underlay). Also the 77.77.77.77/32 came from 65010 (Edge), Aviatrix Transit AS number is not in the as-path.

ISR-2#show ip bgp 
BGP table version is 41, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i
*> 77.77.77.77/32   10.1.12.2                0             0 65010 i

Summary: Gateway Manual BGP Advertised Network List will make Edge gateway advertise towards all it’s BGP peers of selected CIDR, and it will be shown as coming from the Edge gateway. It is a manual entry that will completely override all the prefixes coming from Aviatrix fabric. Use case would be provide a summarized route to on-prem. Let’s say that we want to advertise 10.32.0.0/16 to on-prem instead of more specific routes:

ISR-2#show ip bgp 
BGP table version is 43, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.32.0.0/16     10.1.12.2                0             0 65010 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i

Terraform: In aviatrix_edge_spoke resource or mc-edge module, this option maps to: spoke_bgp_manual_advertise_cidrs

Connection Manual BGP Advertised Network List

Multi-cloud transit -> Advanced Config -> Edit Spoke -> Select the Edge gateway -> Connection Manual BGP Advertised Network List

Connection Manual BGP Advertised Network List is similar to Gateway Manual BGP Advertised Network List mentioned about. The Edge gateway may have multiple external BGP peers, which may creates multiple connections. Gateway Manual BGP Advertised Network List will advertise to *ALL* it’s BGP peers of the specified prefixes. With Connection Manual BGP Advertised Network List you can specify which BGP peer you want to perform the manual advertisement.

Preserve AS Path

Multi-cloud transit -> Advanced Config -> Edit Spoke -> Select the Edge gateway -> Preserve AS Path

According to Aviatrix Document

Preserve As Path option is applicable to both Gateway Manual BGP Advertised Network List and Connection Manual BGP Advertised Network List. When disabled, behavior defaults to the AS path being stripped during BGP route advertisements from transit or spoke gateways to neighbors. When enabled, AS Path is preserved. Gateways will not advertise manual BGP advertised CIDRs if the CIDRs are no longer in the best route DB.

What exactly does this mean? Let’s say that we are adding a spoke VPC on the right, and attach it to Aviatrix Transit, then we use the forementioned: Customize Spoke Advertised VPC CIDRs on this spoke towards Aviatrix transit. Now that the 77.77.77.77/32 is within Best Route table, as Aviatrix fabric consider this prefix came from the spoke.

On the Edge, when we specify Manual BGP Advertised Network List using 77.77.77.77/32 + Preserve AS Path, on-premise ISR will learn following. Unlike previously the prefix as-path was just from the Edge, now you can see the full as-path.

ISR-2#show ip bgp 
BGP table version is 56, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i
*> 77.77.77.77/32   10.1.12.2                0             0 65010 65001 i

Gateway AS Path Prepend

Multi-cloud transit -> Advanced Config -> Edit Spoke -> Select the Edge gateway -> Gateway AS Path Prepend

You can prepend Edge gateway ASN here, the net effect is Edge gateway ASN will be appended to the prefixes advertise to on-premise. The use case would be wanting to use this Edge as secondary route.

In Transit Route Info DB, the edge as-prepend has been inserted for the prefixes received from on-premise

In Transit Gateway Routing Table, no change observed

ISR side, we can see that prefixes from Edge have been prepended.

ISR-2#show ip bgp 
BGP table version is 68, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r> 0.0.0.0          10.1.12.2                0             0 65010 65010 65010 65010 65001 i
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.32.1.0/24     10.1.12.2                0             0 65010 65010 65010 65010 65001 i
*> 10.32.3.0/24     10.1.12.2                0             0 65010 65010 65010 65010 65001 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i
*> 10.64.254.0/23   10.1.12.2                0             0 65010 65010 65010 65010 65001 i

Terraform: In aviatrix_edge_spoke resource or mc-edge module, this option maps to: prepend_as_path

Connection AS Path Prepend

Connection AS Path Prepend is similar to Gateway AS Path Prepend mentioned before. The Edge gateway may have multiple external BGP peers, which may creates multiple connections. Gateway AS Path Prepend will prepend to *ALL* it’s BGP peers, while Connection AS Path Prepend you can specify which BGP peer you want to perform the as-prepend

Connection AS Path Prepend towards Connection to onprem device

Following example shows AS Path Prepend on Edge gateway, on the connection from Edge to the on-premise router. Since the setting is on Edge gateway, you need to provide Edge AS number.

Remember Gateway AS Path Prepend, the prepends were towards both Aviatrix Transit attachment, as well as Edge BGP peers. However, for this example, the as-prepend is only applied on the connection from Edge to OnPrem router, hence Aviatrix Transit won’t observe this prepend.

In Transit Route Info DB, we indeed don’t observe this prepend

In Transit Gateway Routing Table, no change observed

On ISR side, we do observe Edge AS prepend inserted

ISR-2#show ip bgp
BGP table version is 211, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r> 0.0.0.0          10.1.12.2                0             0 65010 65010 65010 65010 65001 i
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.32.1.0/24     10.1.12.2                0             0 65010 65010 65010 65010 65001 i
*> 10.32.3.0/24     10.1.12.2                0             0 65010 65010 65010 65010 65001 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i
*> 10.64.254.0/23   10.1.12.2                0             0 65010 65010 65010 65010 65001 i

Connection AS Path Prepend towards Connection to Transit Gateway

Now we remove the prepend from Edge to OnPrem router connection. Then add prepend from Edge to Aviatrix Transit connection

In Transit Route Info DB, we observed prepend on the prefixes received from onprem.

In Transit Gateway Routing Table, no change observed

On ISR side, we won’t observe the as-prepend

ISR-2#show ip bgp 
BGP table version is 96, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r> 0.0.0.0          10.1.12.2                0             0 65010 65001 i
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.32.1.0/24     10.1.12.2                0             0 65010 65001 i
*> 10.32.3.0/24     10.1.12.2                0             0 65010 65001 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i
*> 10.64.254.0/23   10.1.12.2                0             0 65010 65001 i

Terraform

In aviatrix_edge_spoke_transit_attachment resource or mc-edge module, you may use spoke_prepend_as_path. This is applied to Edge gateway, on the connection from Edge towards Aviatrix transit. (The connection is always named as <transit_gw_name>-peering). The setting is applied on Edge, so you need to provide Edge AS number.

ISR side: Since this prepend is from Edge towards Transit Gateway, ISR won’t get AS Prepend.

In aviatrix_edge_spoke_transit_attachment resource or mc-edge module, there is another setting: transit_prepend_as_path. The setting is applied to the Transit, on the connection between Transit to Edge. Since this is applied on Transit, you will need AS number of the Transit.

Once applied, you can observe the result by going to: Multi-Cloud Transit -> Advanced Config -> Edit Transit -> Select the Transit where the Edge was attached to

Scroll down and in the Connection AS Path Prepend section, select the Edge to OnPrem router connection (which is always named as <edge_gw>-peering. Note the Transit AS prepend here.

Since the prepend happen from Transit to Edge, it’s expected that Transit Route Info DB has no change.

Of course Gateway route table has no change

ISR side we can noticed the prepend happened on the Transit ASN

SR-2#show ip bgp 
BGP table version is 124, local router ID is 169.254.253.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, x best-external, f RT-Filter
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r> 0.0.0.0          10.1.12.2                0             0 65010 65001 65001 65001 65001 i
*> 10.1.13.0/24     0.0.0.0                  0         32768 i
*> 10.1.88.88/32    0.0.0.0                  0         32768 i
*> 10.32.1.0/24     10.1.12.2                0             0 65010 65001 65001 65001 65001 i
*> 10.32.3.0/24     10.1.12.2                0             0 65010 65001 65001 65001 65001 i
*> 10.64.0.0/23     169.254.96.17                          0 65020 i
*> 10.64.254.0/23   10.1.12.2                0             0 65010 65001 65001 65001 65001 i

Transitive Routing

If you have a edge gateway connected to multiple Aviatrix Transits, enable Edge Transit Routing would allow traffic between Aviatrix Transits flow via Edge

I’ll give credit to following image created by Bayu Wibowo, he did an incredible job on Edge documents, this image is no exception. I cannot do a better job than this.

As show above, once Edge Transit Routing is enabled, it allows AWS Aviatrix Transit and Azure Aviatrix Transit to use Edge gateway as a transit router. Another use case would be use as-prepend on the Aviatrix Transit Peering via Internet, so that the Transit peering via Internet would be treated as backup path, while all traffic will flow through encrypted High Performance Encryption tunnels via the Edge gateway on top of private connections, such as Direct Connect or Express Route.

Network Segmentation

In previous topology, traffic are been fully routed from anywhere to anywhere, while VPCs attached to Aviatrix Orchestrated TGW can be segmented by the Firewalls in Security VPC. What if these VPCs need to talk to Shared VPC that’s attached to Aviatrix Transit, these VPCs also need to talk to on-prem via Edge, but Shared VPC cannot be accessed by on-premise?

Aviatrix made this very simple by introducing Network Domain concepts similar to VRF (Virtual Routing and Forwarding), but without the complexity of VRF. Let’s take a look at how this is done.

Following examples shows everything can ping everything for now:

Left: Instance in Dev VPC 10.32.1.121 successful ping Shared VPC instance 10.16.0.59
Instance in Dev VPC 10.32.1.121 successful ping on-prem instance 10.1.88.88

Right: Instance in Shared VPC 10.16.0.59 successful ping Dev VPC instance 10.32.1.121
Instance in Shared VPC 10.16.0.59 successful ping on-prem instance 10.1.88.88

In Aviatrix Controller -> Multi-Cloud Transit -> List -> Transit Gateways -> Select the Transit Gateway -> Details/ Diag -> Gateway Routing Table -> Click on the drop down, notice there is nothing in there at the moment.

CoPilot -> Programmable Intent -> Network Segmentations -> Transit Gateways

Enable Network Segmentation on specific transit gateway

Create new network domain, one for Shared_VPC, one for On_Prem, one for AWS_TGW_Attached_Dev

Notice, that you can associate either an Aviatrix Spoke, or an Aviatrix Edge Gateway or an Aviatrix Orchestrated TGW Domain

In my lab setup, I have this Aviatrix Spoke aws-ue1-spoke, let’s assume this is the Shared VPC and add it to Shared_VPC network domain. This interface actually allow you to perform three actions in one step:

1. Creation of Network Domain
2. Association of Aviatrix Spoke, or an Aviatrix Edge Gateway or an Aviatrix Orchestrated TGW Domain to this Network Domain
3. Create a Connection Policy between Network Domains (Similar to VRF Leak)

Since we don’t have any Network Domain yet, let’s just use first two steps

In Aviatrix Controller -> Multi-Cloud Transit -> List -> Transit Gateways -> Select the Transit Gateway -> Details/ Diag -> Gateway Routing Table -> Click on the drop down, notice there is a new Route Table called Shared_VPC. This matches to the Network Domain we just created, this is a good indication that we are creating a new Routing Table for this network domain, similar to what VRF does.

Now let’s add Edge to On_Prem Network domain and associate Edge gateway branch1 to it. Since Shared_VPC is the only other Network domain, we don’t want On_Prem to talk to Shared_VPC, so leave Connect to Network Domain empty

In Aviatrix Controller -> Multi-Cloud Transit -> List -> Transit Gateways -> Select the Transit Gateway -> Details/ Diag -> Gateway Routing Table -> Click on the drop down, notice now we have two network domains. (We can have up to 250 Network Domains per Transit, which is the limit of Linux routing table)

Now try to add Dev VPC to AWS_TGW_Attached_Dev Network domain, and try to connect with Shared_VPC network domain:

Since I’m using Security VPC inspecting the Dev VPC, error would occur. In the past, Aviatrix Orchestrated TGW Network Domain cannot have connection policy with Aviatrix Spoke Network Domain. Newer release added this feature, but it’s limited to only Aviatrix Orchestrated TGW Network Domains that are not been inspected.

Check current status, network domain AWS_TGW_Attached_Dev created, and have a connection policy to Shared_VPC, but not associated with Dev VPC yet.

After disabled E/W and Egress inspection of the Dev VPC, now I’m able to add the Dev VPC to AWS_TGW_Attached_Dev

In Aviatrix Controller -> Multi-Cloud Transit -> List -> Transit Gateways -> Select the Transit Gateway -> Details/ Diag -> Gateway Routing Table -> Click on the drop down, notice now we have three network domains.

Note OnPrem isn’t connected to anything yet. Let’s try to connect On-Prem with Dev but not Shared

Logic view clearly shows that dev are connected to both onprem and shared network domains. But there’s no connection between onprem and shared.

Now check connectivity

Left: Instance in Dev VPC 10.32.1.121 successful ping Shared VPC instance 10.16.0.59
Instance in Dev VPC 10.32.1.121 FAILED to ping on-prem instance 10.1.88.88

Right: Instance in Shared VPC 10.16.0.59 successful ping Dev VPC instance 10.32.1.121
Instance in Shared VPC 10.16.0.59 successful ping on-prem instance 10.1.88.88

So far I have not touched any routing table, and Aviatrix Controller did all the heavy lifting. This is truly indent based routing. Instead of wrangling with routing tables, Let’s leave the computer do what it can do best 🙂

In case you want to see what’s been done, here’s the routing table for AWS_TGW_Attached_Dev

Shared VPC

On-Prem

Route Info BD also shows the separate routing tables.

Conclusion

• Aviatrix Edge 2.0 is a feature-rich solution for connecting multiple clouds and regions with a focus on high performance, security, and availability.
• It offers advanced features for influencing routing decisions and is fully supported by Terraform providers and modules, making it easy for enterprises to rapidly deploy connectivity from multiple data centers to the cloud.
• With Aviatrix Edge 2.0, creating network segmentation is a breeze. You can define a network domain, associate it with a connection policy. CoPilot further simplifies this workflow by performing both steps simultaneously. This intent-based network segmentation can greatly simplify and clarify your IT operations.