{"id":862,"date":"2022-10-28T11:09:28","date_gmt":"2022-10-28T16:09:28","guid":{"rendered":"https:\/\/cloudlearning365.com\/?p=862"},"modified":"2022-10-28T12:57:49","modified_gmt":"2022-10-28T17:57:49","slug":"learning-of-trace-route-icmp-and-ip-route-table","status":"publish","type":"post","link":"https:\/\/cloudlearning365.com\/?p=862","title":{"rendered":"Learning of Trace Route, ICMP and IP route table"},"content":{"rendered":"\n<p>We are using traceroute very often and sometimes take it for granted, until an very interesting question hit me and we have do dive a little deeper to get the answer. Here&#8217;s the full story:<\/p>\n\n\n\n<p>Aviatrix CloudN is an appliance that helps to deliver line rate of encryption from on-premises towards the Aviatrix Transit Gateways, it is shipped with three interfaces:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>eth0 : WAN interface, this is where IPSec tunnels will be built towards Aviatrix Transit Gateways. Then BGP session will be established between CloudN to Aviatrix Transit Gateways.<\/li><li>eth1: LAN interface, this is where BGP is established between CloudN with on-premise router<\/li><li>eth2: MGMT interface, this is where you connect to CloudN for management, as well as where CloudN connects to internet for software updates.<\/li><\/ul>\n\n\n\n<p>It&#8217;s very common practice to have all three interfaces connected to the same router, have VRF configured on router to segment the three interfaces. As you may recall in my previous blog: <a href=\"https:\/\/cloudlearning365.com\/?p=608\">Direct Connect to Aviatrix Transit \u2013 Option 3<\/a>. The WAN\/LAN\/MGMT(not in the diagram) can connect to the same router as show below.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"595\" height=\"1024\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-138-595x1024.png\" alt=\"\" class=\"wp-image-864\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-138-595x1024.png 595w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-138-174x300.png 174w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-138-768x1323.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-138.png 839w\" sizes=\"auto, (max-width: 595px) 100vw, 595px\" \/><\/figure>\n\n\n\n<p>After we have CloudN inline with traffic, when customer tried to do a traceroute from on-premises towards cloud, they discovered that the CloudN hop was responded by the management interface IP, rather than LAN interface IP. <\/p>\n\n\n\n<p><span style=\"text-decoration: underline;\"><strong>Customer is rightfully concerning that if the data traffic is actually going through MGMT interface instead of from LAN interface.<\/strong><\/span><\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">Before understanding traceroute, we need to understand what is Time to Live or TTL<\/h2>\n\n\n\n<p>(Following test is done in Windows, Linux\/MAC command line may vary)<\/p>\n\n\n\n<p>When we issue an command to ping google DNS server 8.8.8.8, windows ping issues 4 ICMP ECHO Request towards 8.8.8.8<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ping 8.8.8.8\n\nPinging 8.8.8.8 with 32 bytes of data:\nReply from <mark style=\"background-color:#efee72\" class=\"has-inline-color\">8.8.8.8<\/mark>: bytes=32 time=27ms TTL=116\nReply from 8.8.8.8: bytes=32 time=23ms TTL=116\nReply from 8.8.8.8: bytes=32 time=26ms TTL=116\nReply from 8.8.8.8: bytes=32 time=23ms TTL=116\n\nPing statistics for 8.8.8.8:\n    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),\nApproximate round trip times in milli-seconds:\n    Minimum = 23ms, Maximum = 27ms, Average = 24ms<\/code><\/pre>\n\n\n\n<p>Each router would pass this ICMP to next hop until it reaches 8.8.8.8. 8.8.8.8 itself then send ICMP ECHO reply back directly to the client.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1010\" height=\"368\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp.png\" alt=\"\" class=\"wp-image-867\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp.png 1010w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-300x109.png 300w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-768x280.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-500x182.png 500w\" sizes=\"auto, (max-width: 1010px) 100vw, 1010px\" \/><\/figure>\n\n\n\n<p>Packet capture shows:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Reply source 8.8.8.8<\/li><li>ICMP reply type is 0<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"561\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-139-1024x561.png\" alt=\"\" class=\"wp-image-868\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-139-1024x561.png 1024w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-139-300x164.png 300w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-139-768x421.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-139-1536x842.png 1536w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-139-2048x1122.png 2048w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-139-500x274.png 500w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>When we tell ping to use Time to Live (TTL) switch, each router on it&#8217;s path will deduct TTL value by 1, and forward towards next hop if TTL-1 didn&#8217;t reach zero yet. On the first router where TTL-1 reaches 0, that specific router would return ICMP Time to Live Exceeded directly to client, and the packet is no longer forwarded to next hop.<\/p>\n\n\n\n<p>Following example using TTL = 1, when trying to ping 8.8.8.8. The first hop (default gateway) received the packet and deduct TTL by 1 and reached to 0, then the first hop router responded back TTL expired in transit message<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code> ping -i 1 8.8.8.8\n\nPinging 8.8.8.8 with 32 bytes of data:\nReply from <mark style=\"background-color:#efee72\" class=\"has-inline-color\">192.168.68.1<\/mark>: TTL expired in transit.\nReply from 192.168.68.1: TTL expired in transit.\nReply from 192.168.68.1: TTL expired in transit.\nReply from 192.168.68.1: TTL expired in transit.\n\nPing statistics for 8.8.8.8:\n    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),\n<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"273\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-1-1-1024x273.png\" alt=\"\" class=\"wp-image-871\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-1-1-1024x273.png 1024w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-1-1-300x80.png 300w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-1-1-768x205.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-1-1-500x133.png 500w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-1-1.png 1430w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Packet capture shows:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Reply source 192.168.68.1, not 8.8.8.8<\/li><li>ICMP reply type is 11<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"562\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-140-1024x562.png\" alt=\"\" class=\"wp-image-870\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-140-1024x562.png 1024w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-140-300x165.png 300w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-140-768x421.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-140-1536x843.png 1536w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-140-2048x1123.png 2048w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-140-500x274.png 500w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Now we try TTL = 2. First hop router deduct TTL by 1, it&#8217;s not zero yet. So it forward to second router, which also deduct TTL by 1 reached 0. The second router responded TTL expired in transit with it&#8217;s own IP.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ping -i 2 8.8.8.8\n\nPinging 8.8.8.8 with 32 bytes of data:\nReply from <mark style=\"background-color:#efee72\" class=\"has-inline-color\">192.168.1.1<\/mark>: TTL expired in transit.\nReply from 192.168.1.1: TTL expired in transit.\nReply from 192.168.1.1: TTL expired in transit.\nReply from 192.168.1.1: TTL expired in transit.\n\nPing statistics for 8.8.8.8:\n    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"309\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-2-1024x309.png\" alt=\"\" class=\"wp-image-872\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-2-1024x309.png 1024w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-2-300x90.png 300w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-2-768x231.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-2-500x151.png 500w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/traceroute-icmp-TTL-2.png 1430w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Packet capture shows:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Reply source 192.168.1.1, not 8.8.8.8<\/li><li>ICMP reply type is 11<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"563\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-141-1024x563.png\" alt=\"\" class=\"wp-image-873\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-141-1024x563.png 1024w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-141-300x165.png 300w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-141-768x422.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-141-1536x844.png 1536w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-141-2048x1125.png 2048w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-141-500x275.png 500w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Traceroute does 3 ICMP request with TTL start from 1, next 3 ICMP request TTL set to 2, then it repeat this process until TTL reaches default 30<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>tracert 8.8.8.8\n\nTracing route to dns.google &#91;8.8.8.8]\nover a maximum of 30 hops:\n\n  1     6 ms     4 ms     4 ms  192.168.68.1\n  2     8 ms     6 ms     6 ms  192.168.1.1\n  3    22 ms    22 ms    21 ms  69.194.50.2\n  4    24 ms    24 ms    23 ms  69.194.50.1\n  5    25 ms    22 ms    24 ms\n\nctrl + c canceled at this point<\/code><\/pre>\n\n\n\n<p>In the capture, you can see TTL start from 1 and increases by 1 every 3 requests. Also notice the responding IP matches what you see in tracert output.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"467\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-144-1024x467.png\" alt=\"\" class=\"wp-image-877\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-144-1024x467.png 1024w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-144-300x137.png 300w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-144-768x350.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-144-1536x701.png 1536w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-144-2048x934.png 2048w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-144-500x228.png 500w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Ok, we established trace route response came from the router, where TTL-1 gets to 0. Then why it get out from MGMT interface instead of LAN interface?<\/h2>\n\n\n\n<p>According to <a href=\"https:\/\/www.rfc-editor.org\/rfc\/rfc1812#section-4.3.2.4\">RFC1812<\/a><\/p>\n\n\n\n<p>Except where this document specifies otherwise, the IP source address in an ICMP message originated by the router MUST be one of the IP addresses associated with the physical interface over which the ICMP message is transmitted. If the interface has no IP addresses associated with it, the router&#8217;s router-id (see Section [5.2.5]) is used instead.<\/p>\n\n\n\n<p>For folks with Windows background, there is only one route table in OS. When I was at Microsoft Support, dual default gateways always was a No No. This article is still in effect since Windows NT: <a href=\"https:\/\/learn.microsoft.com\/en-us\/troubleshoot\/windows-server\/networking\/connectivity-issues-multiple-default-gateways\">Multiple default gateways can cause connectivity problems<\/a><\/p>\n\n\n\n<p>In Linux, multiple route tables co-exists. This made VRF (Virtual Routing and Forwarding) possible.<\/p>\n\n\n\n<p>If you obtain diagnostics logs from CloudN, you can find following IP rules:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code> \"ip rule\": &#91;\n                \"0:\\tfrom all lookup local \",\n                \"5:\\tfrom all fwmark 0xf4240 lookup mgmt \",\n                \"10:\\tfrom all iif lo lookup exclude_gateway \",\n                \"32766:\\tfrom all lookup main \",\n                \"32767:\\tfrom all lookup default\"\n            ],\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\"><li>0: First it will look at ip route table <strong>local<\/strong>, where it won\u2019t find a match<\/li><li>5: Then it will look at packet that\u2019s marked 0xf4240, which are packets came from mgmt interface eth0, this won\u2019t be a match either, as the packet originated from CloudN itself<\/li><li>10: Packet came from <strong>loopback <\/strong>or <strong>local <\/strong>will use exclude_gateway route table, this is a match as the ICMP TTL exceeded originated from CloudN itself<\/li><li>It will not process further rules as it found a match.<\/li><\/ul>\n\n\n\n<p>In the exclude_gateway route table, the only match is the default route, which will be sent via eth2, MTMT interface.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"772\" height=\"59\" src=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-146.png\" alt=\"\" class=\"wp-image-879\" srcset=\"https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-146.png 772w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-146-300x23.png 300w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-146-768x59.png 768w, https:\/\/cloudlearning365.com\/wp-content\/uploads\/2022\/10\/image-146-500x38.png 500w\" sizes=\"auto, (max-width: 772px) 100vw, 772px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion: Trace Route responses is originated directly from the responding router itself, it doesn&#8217;t represent actual path data will be going through, as the actual data will be subjecting to different routing tables.<\/h2>\n\n\n\n<p>Another mystery solved \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>We are using traceroute very often and sometimes take it for granted, until an very interesting question hit me and we have do dive a little deeper to get the answer. Here&#8217;s the full story: Aviatrix CloudN is an appliance &hellip; <a href=\"https:\/\/cloudlearning365.com\/?p=862\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-862","post","type-post","status-publish","format-standard","hentry","category-aviatrix"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=\/wp\/v2\/posts\/862","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=862"}],"version-history":[{"count":8,"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=\/wp\/v2\/posts\/862\/revisions"}],"predecessor-version":[{"id":885,"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=\/wp\/v2\/posts\/862\/revisions\/885"}],"wp:attachment":[{"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=862"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=862"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cloudlearning365.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=862"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}