- \n
- Each IPSec tunnel have 1.25G throughput limit<\/mark><\/li>\n\n\n\n
- Azure only support IPSec, not GRE as tunneling protocol<\/mark><\/li>\n\n\n\n
- On-premise device must be able to support BGP over IPSec, also it is manual process to build\/maintain IPSec tunnels from on-premise device.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Express Route to Aviatrix Transit \u2013 Option 2<\/a>, where we utilize Azure Route Server and some smart design to bridge the BGP between Aviatrix Transit, Azure Route Server and ExpressRoute Gateway, then towards on-premise device. This solution have fpllowing constrains:\n
- \n
- ARS can only exchange up to 200 routes with ERGW<\/mark><\/li>\n\n\n\n
- No end to end encryption between on-premsie towards Aviatrix Transit, only MACSec can be used between on-premise devices towards Microsoft Enterprise Edge router.<\/mark>\n
- \n
- Brad Hedlund<\/a> have an excellent blog about the difference: Securing your network connection to the cloud: MACSec vs. IPSec<\/a><\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Additional architecture complexity\/cost and lose operational visibility, also this solution is in Azure only, means you will end up with different architecture in different clouds.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
For enterprises moving business critical applications to multi-cloud, needing point to point encryption without sacrificing the throughput, looking for unified solution that can provides enterprise level visibility, control, audibility, standardization and troubleshooting toolsets. Neither above two solution would be ideal. IPsec is industry standard utilized by all Cloud Service Providers, but how are we able to overcome it’s limitation of 1.25Gbps per tunnel?<\/p>\n\n\n\n
Aviatrix’s winning formular solves these challenges with it’s patented technology called High Performance Encryption (HPE). It automatically builds multiple IPSec tunnels over either private connectivity such as express route, or over Internet. Aviatrix then combine these tunnels into a logical pipe, to achieve line rate of encryption up to 25Gbps per appliance. <\/p>\n\n\n\n
Aviatrix have several products supports HPE from edge locations: CloudN (Physical form factor), Edge 1.0 and Edge 2.0 (Virtual and physical form factor). They can be deployed on-premise data center, co-location, branch offices or retail locations. These edge devices enable customer enterprise grade visibility and control, monitoring and auditing and troubleshooting capability, as well as providing unified architecture for all major Cloud Service Providers. These solutions enable us easily push all the goodies Aviatrix Transit and Spoke architecture from the clouds towards on-premise.<\/p>\n\n\n\n
In this blog, we will focus on how CloudN is deployed and connect to Aviatrix Transit. Here below is the architecture diagram:<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/ER-to-Aviatrix-Transit-Options-Option2-641x1024.png\")
<\/figure>\n\n\n\n\n\n\n\n- \n
- Aviatrix Transit Gateway have to be deployed with Insane mode (HPE) enabled. <\/li>\n\n\n\n
- The underlay ExpressRoute connectivity has to be build from on-premise device towards Aviatrix Transit Gateway vNet. Follow steps in previous blog post: Express Route to Aviatrix Transit \u2013 Option 1<\/a><\/li>\n\n\n\n
- We don’t need loopback setup, as the HPE tunnels will be established between CloudN WAN interface and Aviatrix Transit Gateway eth0. As such the CIDR range of CloudN WAN interface need to be reachable from Aviatrix Transit Gateway eth0, via it’s subnet route table going towards ExpressRoute Gateway (ERGW) through Microsoft Enterprise Edge Router (MSEE)<\/li>\n\n\n\n
- As illustrated in the diagram, the CloudN device have three interfaces:\n
- \n
- eth0 : WAN interface, this is where IPSec tunnels will be built towards Aviatrix Transit Gateways. Then BGP session will be established between CloudN to Aviatrix Transit Gateways.<\/li>\n\n\n\n
- eth1: LAN interface, this is where BGP is established between CloudN with on-premise router<\/li>\n\n\n\n
- eth2: MGMT interface, this is where you connect to CloudN for management, as well as where CloudN connects to internet for software updates.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
A review and validation of underlay setup<\/h2>\n\n\n\n
My lab cloudN interface IP address setup<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-41-1024x180.png\")
<\/figure>\n\n\n\nMy lab router already have BGP session towards Azure Private Peering primary connection<\/p>\n\n\n\n
Azure Private Peering Primary subnet range: 169.254.80.80\/30. On-premise router will use first IP of 169.254.80.81, and MSEE will use second IP of: 169.254.80.82<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-42-1024x611.png\")
<\/figure>\n\n\n\nOn-premise router config.<\/p>\n\n\n\n
- \n
- GigabitEthernet0\/0\/0.803 on VLAN 803, assigned IP: 169.254.80.81\/30<\/li>\n\n\n\n
- On-premise router asn: 65000, neighbor with MSEE 169.254.80.82, ER circuit asn is static: 12076 <\/li>\n\n\n\n
- On-premise router advertise multiple subnets including CloudN WAN range of 10.1.32.0\/24<\/li>\n\n\n\n
- On-premise router limit advertisement towards ER peering to only 10.1.32.0\/24<\/li>\n<\/ul>\n\n\n\n
interface GigabitEthernet0\/0\/0.803\n description to be connected to an Azure ER circuit\n encapsulation dot1Q 803\n ip address 169.254.80.81 255.255.255.252\n\nrouter bgp 65000\n bgp log-neighbor-changes\n neighbor 169.254.80.82 remote-as 12076\n neighbor 169.254.80.82 description Express Route\n !\n address-family ipv4\n network 10.1.30.0 mask 255.255.255.0\n network 10.1.30.10 mask 255.255.255.255\n network 10.1.31.0 mask 255.255.255.0\n network 10.1.32.0 mask 255.255.255.0\n neighbor 169.254.80.82 activate\n neighbor 169.254.80.82 soft-reconfiguration inbound\n neighbor 169.254.80.82 prefix-list router-to-er out\n maximum-paths 8\n exit-address-family\n\nip prefix-list router-to-er description Advertise Loopback only\nip prefix-list router-to-er seq 10 permit 10.1.32.0\/24\n<\/code><\/pre>\n\n\n\nVerify on ExpressRoute Circuit -> Peerings -> Azure Private -> View route table<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-44-1024x466.png\")
<\/figure>\n\n\n\nIt received Aviatrix Transit vNet CIDR 10.0.16.0\/23 from ERGW of static asn: 65515. It also received CloudN WAN CIDR range of 10.1.32.0\/24 from on-premise router asn: 65000 <\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-43-1024x429.png\")
<\/figure>\n\n\n\n10.0.17.236 and 10.0.17.237 is ERGW, as seen below ERGW is associated with GatewaySubnet of 10.0.17.224\/27<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-45.png\")
<\/figure>\n\n\n\nWe can also validate on the Aviatrix Transit Gateway eth0 effective route table<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-46-1024x322.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-47-1024x733.png\")
<\/figure>\n\n\n\nAlso remember, the Route table must have Propagate gateway route checked to allow ERGW to program routes received from ER private peering<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-48.png\")
<\/figure>\n\n\n\nFrom Aviatrix Transit Gateway ping the default gateway of CloudN WAN interface, this interface is on the on-premise router.<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-49-983x1024.png\")
<\/figure>\n\n\n\nYou will not be able to ping CloudN’s WAN interface 10.1.32.100 yet, even though earlier we saw the CloudN WAN interface have a gateway point to on-premise router 10.1.32.1. To explain this, logon to CloudN device itself, when CloudN is not registered with Aviatrix Controller, it’s considered as Stand Alone CloudN, and diagnostics can be performed on the device itself. The interface looks like an older version Aviatrix Controller interface. Troubleshoot -> Diagnostics -> Gateway -> Make sure Gateway is set to none, and Controller checked -> Run<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-50-893x1024.png\")
<\/figure>\n\n\n\nLook at ip rule section<\/p>\n\n\n\n
\"ip rule\": [\n \"0:\\tfrom all lookup local \",\n \"5:\\tfrom all fwmark 0xf4240 lookup mgmt \",\n \"10:\\tfrom all iif lo lookup exclude_gateway \",\n \"32766:\\tfrom all lookup main \",\n \"32767:\\tfrom all lookup default\"\n ],<\/code><\/pre>\n\n\n\n- \n
- 0: First it will look at ip route table local<\/strong>, where it won\u2019t find a match for Aviatrix Transit vNet CIDR 10.0.16.0\/23<\/li>\n\n\n\n
- 5: Then it will look at packet that\u2019s marked 0xf4240, which are packets came from mgmt interface eth0, this won\u2019t be a match either, as the response packet would originate from CloudN itself<\/li>\n\n\n\n
- 10: Packet came from loopback <\/strong>or local <\/strong>will use exclude_gateway route table, where it will use 10.1.0.254 as default gateway via mgmt interface<\/li>\n\n\n\n
- It will not process further rules as it found a match.<\/li>\n<\/ul>\n\n\n\n
exclude_gateway route table<\/p>\n\n\n\n
\"ip route table exclude_gateway\": [\n \"default via 10.1.0.254 dev eth2 metric 100 \",\n \"10.1.0.0\/24 dev eth2 proto kernel scope link src 10.1.0.3 \",\n \"10.1.31.0\/24 dev eth1 scope link src 10.1.31.100 \",\n \"10.1.32.0\/24 dev eth0 scope link src 10.1.32.100 \",\n \"throw 169.254.0.0\/16\"\n ],<\/code><\/pre>\n\n\n\nBut how Aviatrix will then build IPSec tunnel with Aviatrix Transit Gateway? Read on please<\/p>\n\n\n\n
CloudN registration, attachment and validation<\/h2>\n\n\n\n
As mentioned earlier, CloudN that’s not registered with Aviatrix Controller is considered as Stand Alone CloudN, and you will need to manage everything directly from the CloudN appliance itself, including building Site2Cloud IPSec tunnels.<\/p>\n\n\n\n
Once you have registered CloudN to Aviatrix Controller, it became Managed CloudN appliance, all operations should be performed on the Aviatrix Controller. It makes building HPE tunnels as easy as attach Spoke Gateway to Transit Gateway.<\/p>\n\n\n\n
To register CloudN, login to CloudN, click on UseCases -> Managed CloudN<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-51.png\")
<\/figure>\n\n\n\nThis brings Settings -> Advanced -> Registration page, where you will need to enter Aviatrix Controller IP or DNS name, username and password, as well as pick a name for the CloudN appliance, note:<\/p>\n\n\n\n
- \n
- Network Security Group on the controller must allow public IP of the mgmt interface access on TCP 443<\/li>\n\n\n\n
- The user account need to have CloudN permission<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-52-1024x646.png\")
<\/figure>\n\n\n\nExample of adding CloudN read\/write permission to “test” permission group in Controller -> Accounts -> Permission Groups -> Select permission group -> Manage permissions<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-53.png\")
<\/figure>\n\n\n\nWait for the registration to complete<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-54.png\")
<\/figure>\n\n\n\nThe registration process require outbound Internet connectivity. If run into issue, reference to Required Access for External Sites<\/a><\/p>\n\n\n\n
Once registered, the CloudN appliance console top right would indicate the appliance is now managed by Aviatrix controller. All operations going forward need to be performed from Aviatrix Controller.<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-55-1024x85.png\")
<\/figure>\n\n\n\nIn Aviatrix Controller -> CloudN -> List, you should see the device is registered<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-56-1024x346.png\")
<\/figure>\n\n\n\nController -> CloudN -> Attach -> Prepare to Attach, this step is only necessary for building tunnels over Internet connectivity. We are building the tunnels over ExpressRoute, so this step can be skipped. However if you are building tunnels on Internet and detection failed, you can manually specify the public IP of the WAN port, then click on Apply<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-57-1024x430.png\")
<\/figure>\n\n\n\nOn-premise router LAN interface and BGP configuration<\/p>\n\n\n\n
- \n
- Create loopback interface 88 is for testing purpose<\/li>\n\n\n\n
- Note the on-premise router LAN interface is 10.1.31.1 peering with CloudN LAN interface 10.1.31.100 <\/li>\n\n\n\n
- ip prefix-list limit only loopback 88 gets propagated to CloudN<\/li>\n<\/ul>\n\n\n\n
interface Loopback88\n ip address 192.168.88.88 255.255.255.255\n\ninterface GigabitEthernet0\/0\/1.31\n description \"cloudN-3 LAN\"\n encapsulation dot1Q 31\n ip address 10.1.31.1 255.255.255.0\n\nrouter bgp 65000\n bgp log-neighbor-changes\n neighbor 10.1.31.100 remote-as 65003\n neighbor 10.1.31.100 description cloudN-3\n !\n address-family ipv4\n network 10.1.30.0 mask 255.255.255.0\n network 10.1.30.10 mask 255.255.255.255\n network 10.1.31.0 mask 255.255.255.0\n network 10.1.32.0 mask 255.255.255.0\n neighbor 10.1.31.100 activate\n neighbor 10.1.31.100 soft-reconfiguration inbound\n neighbor 10.1.31.100 prefix-list to-cloudN out\n maximum-paths 8\n exit-address-family\n\nip prefix-list to-cloudN seq 10 permit 192.168.88.88\/32\n<\/code><\/pre>\n\n\n\nAttach CloudN to Aviatrix Transit, the same workflow also establish BGP with on-premise router. Over Private Network is checked for using ExpressRoute<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-58-1024x739.png\")
<\/figure>\n\n\n\nCloudN is building IPSec tunnels towards both Aviatrix Transit Gateways<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-60.png\")
<\/figure>\n\n\n\nOnce the attach is completed, CloudN -> List -> you can see the CloudN is now attached<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-61-1024x220.png\")
<\/figure>\n\n\n\nIn Site2Cloud -> Setup, we should see the connection from CloudN to Aviatrix Transit Gateways are Up<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-62-1024x219.png\")
<\/figure>\n\n\n\nTo confirm BGP, Multi-Cloud Transit -> List -> Transit Gateways -> Select a Transit Gateway, then click on Details\/Diag<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-63-1024x303.png\")
<\/figure>\n\n\n\nScroll to Gateway Route Table, then click on the Refresh button.<\/p>\n\n\n\n
Search for the 192.168.88.88 loopback we advertised, you can see 8 tunnels been between CloudN towards Aviatrix Transit. Also a tunnel going towards HA gateway for redundant route.<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-65-1024x789.png\")
<\/figure>\n\n\n\nWe can observe in Azure Portal, the Transit Gateway have 8 IPs assigned to eth0<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-67-1024x661.png\")
<\/figure>\n\n\n\nSince we have not attached any spoke to the Transit, on-premise router won’t receive any route from Aviatrix Transit yet<\/p>\n\n\n\n
show ip bgp neighbors 10.1.31.100 received-routes \n\nTotal number of prefixes 0 <\/code><\/pre>\n\n\n\nAs described in previous blog, we have multiple way to test this<\/p>\n\n\n\n
- \n
- Attach an Aviatrix Spoke <\/li>\n\n\n\n
- Advertise Transit CIDR<\/li>\n\n\n\n
- Gateway Manual BGP Advertised Network List<\/li>\n\n\n\n
- Connection Manual BGP Advertised Network List<\/li>\n<\/ul>\n\n\n\n
Following example using the last method:<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-68-1024x459.png\")
<\/figure>\n\n\n\n#show ip bgp neighbors 10.1.31.100 received-routes \nBGP table version is 10, local router ID is 192.168.77.1\nStatus codes: s suppressed, d damped, h history, * valid, > best, i - internal, \n r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, \n x best-external, a additional-path, c RIB-compressed, \n t secondary path, L long-lived-stale,\nOrigin codes: i - IGP, e - EGP, ? - incomplete\nRPKI validation codes: V valid, I invalid, N Not found\n\n Network Next Hop Metric LocPrf Weight Path\n *> 80.80.80.80\/32 10.1.31.100 0 65003 65001 i\n\nTotal number of prefixes 1 \n\n<\/code><\/pre>\n\n\n\nCloudN advanced features<\/h2>\n\n\n\n
Since now the CloudN is managed by Aviatrix Controller, following operations are done in Aviatrix Controller. CloudN -> Advanced -> BGP -> Pick a command to run, example showing ip bpg on the CloudN itself<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-69-1024x393.png\")
<\/figure>\n\n\n\nYou can also specify AS Path Prepend against a particular CloudN device, this would help with multiple CloudN setup and you want one CloudN to be on the preferred active path<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-70-1024x377.png\")
<\/figure>\n\n\n\nRevisit the priory routing issue between CloudN and Aviatrix Transit Gateway<\/p>\n\n\n\n
CloudN -> List -> Select the CloudN device -> Diag -> Run<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-71-1024x448.png\")
<\/figure>\n\n\n\nAviatrix Controller programmed all Aviatrix Transit Gateways’ eth0 IP address in the exclude_gateway route table pointing to on-premise router, this resolved the connectivity issue between CloudN WAN interface to Aviatrix Transit Gateways eth0 mentioned before.<\/p>\n\n\n\n
![\"\"](\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/11\/image-72.png\")
<\/figure>\n\n\n\nConclusion<\/h2>\n\n\n\n
By utilizing Aviatrix CloudN to connect on-premise to the clouds<\/p>\n\n\n\n
- \n
- We are able to solve the challenges enterprises are facing when moving mission critical applications to the clouds, by providing unified architecture, enterprise grade visibility and control, monitoring and auditing and advanced troubleshooting capability<\/li>\n\n\n\n
- High performance encryption solves the dilemma of choosing amongst security or performance or complexity<\/li>\n\n\n\n
- Unified architecture requires less training and will enable workforce quickly onboard secure connectivity to the clouds, and focus what’s really matters to the business.<\/li>\n<\/ul>\n\n\n\n
Additional readings<\/h2>\n\n\n\n
- \n
- Direct Connect to Aviatrix Transit \u2013 Option 1<\/a><\/li>\n\n\n\n
- Direct Connect to Aviatrix Transit \u2013 Option 2<\/a><\/li>\n\n\n\n
- Direct Connect to Aviatrix Transit \u2013 Option 3<\/a><\/li>\n<\/ul>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
In the past blogs, we have reviewed two options to connect from on-premise to Aviatrix Transits: For enterprises moving business critical applications to multi-cloud, needing point to point encryption without sacrificing the throughput, looking for unified solution that can provides … Continue reading
- Direct Connect to Aviatrix Transit \u2013 Option 2<\/a><\/li>\n\n\n\n
- Direct Connect to Aviatrix Transit \u2013 Option 1<\/a><\/li>\n\n\n\n
- 0: First it will look at ip route table local<\/strong>, where it won\u2019t find a match for Aviatrix Transit vNet CIDR 10.0.16.0\/23<\/li>\n\n\n\n
- Additional architecture complexity\/cost and lose operational visibility, also this solution is in Azure only, means you will end up with different architecture in different clouds.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
- No end to end encryption between on-premsie towards Aviatrix Transit, only MACSec can be used between on-premise devices towards Microsoft Enterprise Edge router.<\/mark>\n
- Azure only support IPSec, not GRE as tunneling protocol<\/mark><\/li>\n\n\n\n